AI Research
TRACE
Solver-guided preference optimization for strategic SLM training.
TRACE: Teacher-separated Reasoning and Actions with Counterfactual Evaluation
Frontier LLMs can explain how an agent should respond to another player's behavior, yet often fail to execute that strategy. Against tit-for-tat (an opponent that copies the model's previous move), three frontier models — Haiku 4.5, Gemma 3-27B, and Llama 3.3-70B — earn a cumulative reward of only 1.50, exactly matching an untrained Qwen 2.5-3B. That leaves standard trajectory distillation with no better strategic actions to teach a small model, since the "teacher" plays no better than the "student."
Key idea: split the action teacher from the reasoning teacher
TRACE constructs DPO preference pairs from two different sources instead of one:
- Action label: an exact game solver computes the ex-post best response — the action that would have earned the highest reward against the opponent's recorded move (e.g., optimal bid-shading in a first-price auction, or the take-the-rest proposal in bargaining).
- Reasoning label: a frontier model (Haiku 4.5) paraphrases supporting reasoning for the solver-selected action, explicitly forbidden from referencing the opponent's simultaneous move — it must reason only from history through the prior round, so the trajectory stays decision-time faithful.
- Counterfactual horizon filter: an exact rollout rejects locally attractive actions (e.g., defecting against a cooperator) whose return over the remaining horizon is actually worse than the recorded trajectory — preventing "myopically correct but strategically harmful" labels from entering the training set.
A leakage audit (regex + manual inspection across 688 final oracle rounds) confirmed the reasoning paraphraser does not leak the opponent's hidden current-round move.
Results
| Model | In-domain reward | Held-out reward | Tit-for-tat reward |
|---|---|---|---|
| Qwen 2.5-3B base | 14.04 | −1.45 | 1.50 |
| Qwen 2.5-3B + frontier-action DPO | 13.51 | −0.23 | 1.50 |
| Qwen 2.5-3B + TRACE-RW | 13.93 | +1.91 | 14.06 |
| Haiku 4.5 (frontier reference) | 14.60 | +2.05 | 1.50 |
Across twelve two-player game environments, TRACE raises the 3B student's tit-for-tat reward from 1.50 to 14.06 — well above every frontier reference — and turns held-out reward positive (−1.45 → +1.91), with the held-out gain concentrated in the first-price auction, where solver-labelled data is available (base auction reward −8.67 → +4.97, near Haiku's +5.82).
Why filtering frontier actions alone doesn't work
An earlier "Hypothesis B" pipeline pinned chosen-side actions to frontier play and filtered pairs by reasoning-action coupling (does the chosen action match the argmax of the model's own stated expected values?). This reduced both coupling and reward, because the training distribution was too narrow (5 of 8 in-domain games sit on a single cooperate-or-defect axis) and filtering only shrank the chosen-side distribution further rather than correcting a wrong prior. TRACE resolves this by replacing the action label itself — with the game solver's best response — rather than filtering an already-flawed label source.
Adapter merging
Two specialist LoRA adapters — an auction/negotiation-heavy one (AUX) and a repeated-game-heavy one (ALL) — were simply averaged (W_merge = ½W_AUX + ½W_ALL) without further training, reaching 13.92 ID / +1.75 HO reward, within 0.16 HO reward of the purpose-trained reweighted variant (13.93 / +1.91) — showing specialist capabilities can be combined without a new optimization run.
Limitations
Each game cell uses only n=12 episodes, so differences under ~0.2 cumulative reward should be read as ties; results cover one 3B base model, one solver family, and deterministic training opponents; and TRACE does not surpass Haiku in aggregate in-domain reward or reasoning-action coupling. Stochastic opponents would require estimated (rather than exact) counterfactuals, and games like tic-tac-toe need a sequential solver such as minimax.