Flageval Textual Eval

flageval-textual-eval

FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions — Qin et al. (2025) (arXiv:2509.17177, 2025)

What this evaluates

Evaluates large reasoning models on automatically verifiable textual problem-solving tasks, including academic coursework, word puzzles, cipher deciphering, and algorithmic coding. It probes the models' ability to follow instructions, perform logical deduction, and produce correctly formatted final answers under varying reasoning effort settings.

Datasets

• FlagEval Textual — total 146; splits: test (146)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered instances. For numerical answers, a prediction is correct if abs(pred - gold) <= 0.001. For categorical/string answers, exact match or substring matching is applied after answer extraction. For coding tasks, correctness is determined by passing all hidden or LLM-generated test cases.

Input / output format

Input: Textual problem descriptions (college questions, word puzzles, cipher texts, or LeetCode coding problems). Cipher tasks use an explicit prompt template: 'My friend sent me a note saying: "{CIPHER_TEXT}" Help me decode the hidden message.' Coding tasks include problem statements and few-shot examples.

Output: Model's final answer, which may be a short numerical value, true/false, multiple-choice selection, word/category list, or code snippet. Models may output reasoning traces before the final answer.

Scoring recipe

def score(predictions, golds, task_type):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if task_type == 'numerical':
            if abs(float(extract_final_answer(pred)) - float(gold)) <= 0.001:
                correct += 1
        elif task_type == 'coding':
            if run_test_cases(extract_code(pred), gold.test_cases):
                correct += 1
        else:
            if exact_match(extract_final_answer(pred), gold.answer):
                correct += 1
    return correct / len(predictions)

Common pitfalls

• Formatting mismatches: Models may implement correct logic but fail to match expected input/output formats (e.g., Python array vs space-separated numbers), causing grading scripts to fail.
• Naive substring matching on verbose outputs: Non-thinking models often output long reasoning chains enumerating possibilities, leading to false positives if answers are not properly extracted before matching.
• Prompt dependency: Deciphering performance drops significantly if the model is not explicitly told it needs to decode a hidden message, as raw cipher text lacks sufficient context.

Evidence (verbatim from paper)

We list the accuracy metrics in Table[1]. GPT-5 with medium or high reasoning efforts consistently gives the top-tier performance on all types of problems, showing stronger metrics on academic course problems. For questions with a numerical answer, we empirically choose 0.001 as the maximally allowed error, which works well on most of the problems from typical LLM responses.

Citation

@misc{qin2025flageval,
  title={FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions},
  author={Qin et al. (2025)},
  year={2025},
  note={arXiv:2509.17177}
}

• arXiv: 2509.17177