Sketch Of Thought Eval

sketch-of-thought-eval

Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching — Simon A. Aytes, Jinheon Baek, and Sung Ju Hwang (2025) (arXiv:2503.05179, 2025)

What this evaluates

Evaluates the reasoning efficiency and accuracy of LLMs under cognitive-inspired prompting constraints. It probes the model's ability to produce structured, concise reasoning chains while maintaining correctness across mathematical, commonsense, logical, multi-hop, scientific, medical, multilingual, and multimodal tasks.

Datasets

• GSM8K — total ?; splits: test (-1)
• SVAMP — total ?; splits: test (-1)
• AQUA-RAT — total ?; splits: test (-1)
• DROP — total ?; splits: test (-1)
• CommonsenseQA — total ?; splits: test (-1)
• OpenbookQA — total ?; splits: test (-1)
• StrategyQA — total ?; splits: test (-1)
• LogiQA — total ?; splits: test (-1)
• ReClor — total ?; splits: test (-1)
• HotPotQA — total ?; splits: test (-1)
• MuSiQue-Ans — total ?; splits: test (-1)
• QASC — total ?; splits: test (-1)
• Worldtree — total ?; splits: test (-1)
• PubMedQA — total ?; splits: test (-1)
• MedQA — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• MMMLU — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• ScienceQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Computed via exact match with ground truth for multiple-choice, yes/no, or numeric tasks. For open-ended generation, correctness is assessed using GPT-4o as an LLM-as-a-judge.
• output_token_count — range: other
  • Total number of generated tokens in the intermediate reasoning steps.
• token_reduction_percentage — range: percent
  • Percentage reduction in output tokens compared to the Chain-of-Thought baseline.
• accuracy_delta — range: other
  • Change in accuracy between the baseline method (CoT) and the evaluated method.

Input / output format

Input: Text-based queries (and images for multimodal datasets), processed by a lightweight router model that dynamically selects a reasoning paradigm. Few-shot exemplars are appended via prompting to illustrate the required reasoning style.

Output: Intermediate reasoning steps followed by a final answer. Answers are extracted according to a predefined format (detailed in Appendix B.2) for exact matching or LLM-judge evaluation.

Scoring recipe

def compute_metrics(predictions, golds, generated_tokens):
    exact_matches = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = exact_matches / len(golds)
    avg_tokens = sum(generated_tokens) / len(generated_tokens)
    token_reduction = 1 - (avg_tokens / baseline_tokens)
    accuracy_delta = accuracy - baseline_accuracy
    return {'accuracy': accuracy, 'avg_tokens': avg_tokens, 'token_reduction': token_reduction, 'accuracy_delta': accuracy_delta}

Common pitfalls

• Only 150 questions are sampled from each dataset instead of using the full test set, which may not represent overall performance.
• Performance is averaged over three independent runs per question, introducing sampling variance that can obscure small differences.
• Open-ended accuracy relies on GPT-4o LLM-as-a-judge, which may differ from human evaluation or standard exact-match baselines.
• Token count measures total intermediate reasoning tokens, not just final answer length, making efficiency comparisons sensitive to reasoning verbosity.

Evidence (verbatim from paper)

We evaluate using two primary metrics: accuracy and output token count. For multiple-choice, yes/no, or numeric tasks, accuracy is computed via exact match with the ground truth. For open-ended generation, we follow the LLM-as-a-judge paradigm, using GPT-4o to assess correctness. Answers are extracted according to the output format (see Appendix[B.2]). We analyze efficiency through the total number of generated tokens in the intermediate reasoning. We sample 150 questions from each dataset for the sake of computational costs, and report the averaged performance over three independent runs per question.

Citation

@misc{aytes2025sketchofthought,
  title={Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching},
  author={Simon A. Aytes, Jinheon Baek, and Sung Ju Hwang (2025)},
  year={2025},
  note={arXiv:2503.05179}
}

• arXiv: 2503.05179