WUWeifeng710 Trace as State — Put the State Before the Material
0. In one sentence
Spend one pass over the material, write the resulting task state onto a card, put that card before the material, and re-read with it in hand before answering.
Order is the only variable. The same trace, placed before vs. after, differs by an order of magnitude.
Read §7 before you start. The essence of this method is accuracy, not cost: it trades roughly 2× input tokens for a jump in accuracy, leaves total tokens on par with the control, and adds latency. Cost reduction comes only from the discipline in §7.3 — the method itself does not save tokens. Use it in the wrong setting and you lose.
1. When to enable
Hit any item on the left → enable. Hit any item on the right → do not.
| Enable signals | Do not enable |
|---|---|
| Material exceeds ~1/3 of the context (long docs, large codebases, multiple reports) | Single-hop fact lookup (a value, a definition) |
| The task needs "you only know what to look for after reading" (multi-hop retrieval, "find the clause that contradicts X", "who is most suspicious") | Short input (a few K tokens) |
| State must be maintained across the material (excluded hypotheses, resolved entities, open sub-questions) | Pure generation (writing / translation) with no state to maintain |
| A single long-context call (no history, no cache to sacrifice) | Results insensitive to information order |
| Sub-agent delegation / multi-stage workflow | Multi-turn long session that is prefix-cache sensitive (§7.2) — use intra-turn prefixing instead |
| A hard task a single pass will clearly fail (long-graph / long edge-list state maintenance, cross-file tracking) | A first pass that already meets the bar — gating, §7.3.0 |
Cost starts at roughly doubled input tokens. With no state to maintain, or when a single pass already suffices, using it is a pure loss.
2. Four steps
Step 1 — Get the trace (reuse first, don't re-run)
Look for traces that already exist: searches, greps, full reads, and last turn's conclusions that happened before this turn — those are the trace. Excerpt them straight onto the card. Do not re-run work already done just to "produce a probing pass" — that is pure waste.
Only when there is genuinely no reasoning trace yet, do one probing pass:
- Process the material normally, but don't rush to answer — the goal is a trace, not a finished product
- It may be incomplete and may be wrong — the trace is only a flawed textual proxy for task state
- Write down just three things: what I'm looking for / what I ruled out / what I found
- Record anchors along the way (file:line, section number, URL)
Step 2 — Serialize (write the state card)
Use templates/state-card.md. Four hard requirements:
- Section 0 must be the task condition (goal / acceptance criteria / constraints / sub-goal of this pass)
- Every fact carries an anchor; anything without one is marked "unlocated" and never mixed into established facts
- It must contain an "Excluded hypotheses" section — this is the part of the state that the append order cannot hold, and it is the highest-value part
- The card ends with a disclaimer ("this card may be incomplete or wrong; verify against the source material")
Keep the card short. It is compressed state, not another copy of the material. Past ~800 tokens you are copying the source — which is worse than not compressing. The paper truncated traces to 50,000 characters; we deliberately push to ≤800 tokens to contain cost.
Step 3 — Placement (read §7.2 first to choose)
Single call (no history) → global prefix:
[state card S] → [source material x] → [current question q]
Multi-turn session (history, cache) → intra-turn prefix (default):
[history, unchanged] → [state card S] → [this turn's new material x] → [this turn's task q]
Do not use the paper's control (trace append), and do not reorder the whole history every turn:
[source material x] → [state card S] → [current question q] ← append, the control, worst
[state card S] → [entire history] → [this turn's task] ← global reorder, cache dies, see §7.2
Step 4 — Verify
The state card is a proxy, not the truth. Check it line by line against the source material; where they conflict, the source material wins, and the correction goes into the next card. Skipping this freezes pass-one errors into the answer.
3. Why this ordering — not folklore
A causal-attention processor recurses as s_i = U(s_{i-1}, c_i), and the condition z (= task state) sets the initial state:
| Ordering | Memory cost | Intuition |
|---|---|---|
[z, C] condition first |
Only the current state must be stored, ⌈b⌉ bits | Reading with a purpose, pruning as you go |
[C, z] condition last |
Must retain a full response profile for every possible z, worst case ⌈b·2^b⌉ bits | Reading a 500k-word novel before being told "find the villain" — the whole read was wasted |
Ordering principle: the task condition should be available before the information it is meant to guide.
Note this is not the "put important info first" rule of thumb (lost-in-the-middle); it is a structural consequence of causal state updates. The same content placed before beats placed after — strongly enough to beat "pick the best answer after the fact" (Oracle@5).
4. Anti-patterns (guardrails the paper's ablations hand you directly)
| Anti-pattern | Result | Do instead |
|---|---|---|
| Prefix only the question (Question First) | Helps only some tasks (Parents EM 53.0 vs. 81.8 prefixed) | Prefix the trace/state, not just the question |
| Only a plain re-read (Re2) | Helps (50.0) but far short (81.8) | Re-read with the state card |
| Put only the answer (Answer Feedback) | Worse than the trace (45.4 vs. 81.8) | The process of the state is worth more than the conclusion |
| Put a different question's trace (Random Trace) | 14.2 — worse than no trace (first pass 29.2) | The trace must belong to the same task; cross-task reuse is contamination |
| Trace only, no material (Trace Only) | 43.8 — far below prefixing (81.8) | The source material is indispensable |
| Ship the state card as the answer | Errors are inherited | Step 4 verification is not optional |
| Re-do existing work "to run a probing pass" | Burns a full-extra pass of tokens | Step 1 reuse-first |
| Insert the card ahead of the whole history every turn | Prefix cache dies every turn; cost far exceeds the gain | Use intra-turn prefixing (§7.2) |
5. Agent scenario mapping
The paper is an academic setting (single long-context QA). On an agent it has five landing spots:
| Scenario | How to order |
|---|---|
| A. Long document / large codebase analysis | Done within a single call → global prefix: [state card] → [full text / repo] → [analysis goal] |
| B. Multi-turn agent loop | Intra-turn prefix: [history] → [state card] → [this turn's new material] → [this turn's task]. Leave history untouched so the cache survives |
| C. Sub-agent delegation | Prompt skeleton: [task condition + goal + constraints + acceptance criteria] → [reference material] → [specific question]. Most delegation failures come from writing acceptance criteria after the material |
| D. Multi-stage workflow (research / review / experiment design) | The previous stage's output is not saved for a final summary — it is prefixed ahead of the next stage's material. If stages share a long prefix, insert the output only within the stage boundary |
| E. System prompt / long session | Keep stable constraints at the very front; don't move them to the end just because "they were emphasized recently" — the tail slot belongs to the current question. This one adds no cost (the system prompt is already first and stable) |
6. The trace-count knob (not cheap)
Probing passes n_tr from 1 to 5 give generally rising performance (the paper's word — not strictly monotonic), and the prefix advantage holds throughout (the paper's second pass merges all 5 traces, truncated to 50k characters).
But note: n_tr = 5 means paying for 5 traces of input. Raise it only when the task value is high enough and multi-path exploration is genuinely needed. Default n_tr = 1.
7. The cost ledger (read before you start)
7.1 An honest cost model (from the paper's Appendix F, correcting an earlier misread)
| Item | Change | Note |
|---|---|---|
| Prefix-cache invalidation | May force a full recompute every turn | ⚠️ See §7.2. The largest hidden cost |
| Input tokens | ≈ doubled and up | Two passes + the card; more if n_tr is high |
| Output / reasoning tokens | Varies by task — not universally down | Down on GraphWalks (DeepSeek first pass 62.5M → tas 28.0M, append 35.2M); but up on MRCRv2 / NUB-1M (DeepSeek MRCRv2 256K: append 0.69M → tas 1.36M) |
| Total tokens | On par with trace append, sometimes higher | Appendix F, same order of magnitude; GLM-5.2 GraphWalks total output 35.95M vs. 28.59M |
| Latency | Increased | One extra generation + a longer input |
Net effect: trace as state is an "accuracy" method, not a "cost" method. Input roughly doubles, total tokens are on par with the control, and latency goes up. The paper's own Limitations state that it "increases inference latency and token cost."
Where the gain is largest (consistent with the paper): hard tasks where the first pass is weak and the state only surfaces late — multi-hop retrieval, "find the clause that contradicts X", long-graph / long edge-list state maintenance, cross-file tracking in a large codebase. There the Parents-EM-style gain can exceed +50 points (DeepSeek GraphWalks Parents EM 29.2 → 81.8). Spend the accuracy where it pays.
7.2 The key conflict: prefixing vs. cache
The paper's global prefix puts the state card at the very front of the whole context, which changes the prefix. In a multi-turn conversation the history prefix's KV cache is the main source of cost savings — a cache hit is discounted, a miss is full price.
Reordering history every turn = the cache dies every turn. Over a long session this item alone can exceed everything the "output tokens go down" story buys. The paper does note that state prefixing reduces KV-cache reuse in multi-turn settings, but as an aside in the academic setting; on a real agent it is the primary tension.
The three placements and their trade-offs:
| Placement | Positional advantage | Cache | Use for |
|---|---|---|---|
Global prefix [S, x, q] |
Strongest (the paper's original setting) | Lossless when there is no history to sacrifice | Single call, sub-agent delegation, one-shot long-document analysis |
Intra-turn prefix [history] → [S, x, q] |
Locally preserved (the card still precedes this turn's material) | Preserved (the history prefix is unchanged) | Default for multi-turn agent loops |
Append at the end [history] → [x, S, q] |
Weakest (= the paper's control) | Best | Tolerable when the card is tiny (<200 tokens) and the task simple |
Honest note: intra-turn prefixing is not the paper's global prefix. The paper's mechanism is about state initialization across the entire forward pass; intra-turn prefixing only reproduces the same ordering within this turn's local causal chain — directionally consistent, but the gain may be weaker than global prefixing and has no experimental support. It is an engineering compromise for cache cost, not a paper result.
7.3 Four "don't waste it" hard rules (the only source of cost reduction)
The method does not save cost; cost reduction is discipline: run it only when you can collect accuracy and afford the input, and keep the input minimal.
- Gating (most important): first judge whether a single pass suffices. If the task is "you only know what to look for after reading" and the first pass already shows gaps / contradictions / open hypotheses → go. If the first pass already meets the bar → don't — there is no accuracy to collect, only 2× input to spend.
- Reuse first: excerpt existing traces, don't re-run a probing pass (§2 Step 1)
- Keep the card short: target 200–800 tokens. The paper truncates traces at 50k characters; we go far tighter to contain cost. Past 800, check whether you are copying the source
- Intra-turn prefix: in multi-turn settings, don't touch history (§7.2)
- n_tr = 1 by default: each extra trace is one more input pass; raise it only when the task value is high and the first pass is clearly weak
7.4 Where to spend the accuracy (targeting)
Reserve state prefixing for hard tasks where a single pass clearly fails and the state only emerges late:
- Multi-hop retrieval / "find the clause that contradicts X" / "who is most suspicious"
- Long-graph / long edge-list state maintenance (the GraphWalks class)
- Cross-file tracking in a large codebase (entities / call chains only assemble after several files)
- Sub-agent delegation where acceptance criteria are written ahead of the material
Soft tasks, tasks a single pass already handles, and short inputs — let gating reject them.
8. Pre-delivery checklist
- Is Section 0 of the state card the task condition, not a list of facts?
- Was the trace excerpted from existing reasoning, not re-run for its own sake?
- Is the state card within 800 tokens?
- In multi-turn settings, did you use intra-turn prefixing with the history prefix untouched?
- Does every fact carry an anchor?
- Is the "Excluded hypotheses" section non-empty?
- Does the card carry a disclaimer, and has Step 4 run?
- Is the source material still in context (not degraded to trace-only)?
- Did gating pass? Is this a task the first pass will fail, and were single-pass-sufficient tasks rejected?
- Does this task really need two passes? (Otherwise the cost is wasted)
Basis
Zou X, Tang J. Trace as State: Reasoning Traces as Conditional States for Long-Context Transformers. arXiv:2609.02702 [cs.CL], 2026-09-02.
Trace as state outperforms trace append in 26 of 27 reported combinations of model, task, and metric. GraphWalks Parents EM: DeepSeek V4 Pro Preview 29.2% → 43.0% (append) → 81.8% (prefix); GLM-5.2 66.4% → 83.2% → 100.0%.
Cost (Appendix F, the key correction): this is an accuracy, not cost method — input ≈ doubled, total tokens on par with the control, output tokens task-dependent (down on GraphWalks, up on MRCRv2 / NUB-1M), latency increased. The paper's Limitations explicitly acknowledge that it "increases inference latency and token cost."
Full experimental data, exact ablation figures, confidence intervals, and the token table are in references/evidence.md (load only when checking numbers or when challenged; not needed for daily use).