Landscape Of Thoughts Eval

landscape-of-thoughts-eval

Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models — Zhou et al. (2025) (arXiv:2503.22165, 2025)

What this evaluates

Evaluates the internal reasoning dynamics of LLMs on multi-choice tasks by tracking intermediate thought states and measuring convergence behavior. It quantifies consistency, uncertainty, and perplexity across different model scales, reasoning tasks, and decoding methods to visualize how reasoning trajectories evolve toward correct or incorrect answers.

Datasets

• AQuA — total ?; splits: eval (50)
• MMLU — total ?; splits: eval (50)
• StrategyQA — total ?; splits: eval (50)
• CommonSenseQA — total ?; splits: eval (50)

Metrics

• reasoning accuracy (primary) — range: percent
  • Standard exact-match accuracy calculated as the fraction of problems where the model's final selected answer matches the ground truth.
• consistency — range: [0, 1]
  • Measures the alignment between intermediate reasoning states and the final state within trajectory bins (e.g., 0-20%, 20-40%).
• perplexity — range: other
  • Model's estimated perplexity over generated reasoning steps, used as a feature vector for state distance calculation.
• uncertainty — range: other
  • Derived from the variance or spread of perplexity-based feature vectors across reasoning steps.

Input / output format

Input: Multi-choice reasoning problem (question and options) paired with a reasoning method prompt (e.g., CoT, LtM, MCTS, ToT).

Output: A sequence of intermediate reasoning states/thoughts followed by a final answer choice.

Scoring recipe

# Calculate reasoning accuracy
accuracy = sum(1 for pred, gold in zip(predictions, golds) if pred == gold) / len(golds)

# Calculate consistency per trajectory bin (e.g., 0-20%, 20-40%)
consistency = []
for start_pct, end_pct in bins:
    mid_states = get_states(start_pct, end_pct)
    final_state = get_final_state()
    consistency.append(measure_alignment(mid_states, final_state))

Common pitfalls

• Consistency is measured against the model's own final state, not the ground truth, so high consistency does not guarantee correctness.
• Perplexity values are used for cross-model comparison but require careful validation due to architectural differences, as noted in the appendix.
• Trajectory length varies across methods, but the authors explicitly verify that consistency metrics are independent of trajectory length.

Evidence (verbatim from paper)

The reasoning accuracy of the four subfigures is: (a) 15.8%, (b) 42.0%, (c) 53.2%, and (d) 84.4%. By comparing the consistency plots in Fig.[5], we found that the model generally has low consistency between the intermediate states and the final state.

Citation

@misc{zhou2025landscapeofthoughts,
  title={Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2503.22165}
}

• arXiv: 2503.22165