State Tracking Eval

state-tracking-eval

Exploring State Tracking Capabilities of Large Language Models — Rezaee et al. (2025) (arXiv:2511.10457, 2025)

What this evaluates

Evaluates whether LLMs can track dynamic states over sequential update instructions. It probes the model's ability to maintain and update internal representations of an environment's state across multiple steps, testing sequential reasoning and input-window memory limits.

Datasets

• State-Tracking-Tasks (LinearWorld, HandSwap, Lights) — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly answered queries across all depth levels and tasks. Calculated as the number of correct predictions divided by the total number of test instances.

Input / output format

Input: Natural language prompt containing an initial state description, a sequence of update instructions (state changes), and a final query about the current state.

Output: Direct answer to the query (e.g., a value, boolean, or short phrase), optionally preceded by a Chain-of-Thought reasoning trace if CoT prompting is used.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_label in zip(predictions, gold):
        if str(pred).strip().lower() == str(gold_label).strip().lower():
            correct += 1
    return correct / len(gold)

Common pitfalls

• Models may default to answering based solely on the initial state rather than tracking updates, which the authors explicitly control for with a 'Stateless' baseline.
• Performance degradation at depth ≥ 2 for smaller models is gradual, not binary, so averaging across all depths can mask early-stage failures.
• Chain-of-Thought (CoT) prompting alters the input format by appending reasoning traces, which increases context length and may conflate state-tracking ability with input-window memory limits.

Evidence (verbatim from paper)

Table 1: The main evaluation results of systems on different tasks reported in terms of accuracy at various depths.

Citation

@misc{rezaee2025exploring,
  title={Exploring State Tracking Capabilities of Large Language Models},
  author={Rezaee et al. (2025)},
  year={2025},
  note={arXiv:2511.10457}
}

• arXiv: 2511.10457