Kwbench Eval

kwbench-eval

KWBench: Measuring Unprompted Problem Recognition in Knowledge Work — Maloo (2026) (arXiv:2604.15760, 2026)

What this evaluates

This benchmark evaluates language models' ability to recognize formal game-theoretic structures (e.g., principal-agent conflict, signaling, strategic omission) in real-world knowledge work scenarios without explicit task hints. It measures the gap between a model's theoretical understanding of these concepts and its capacity for unprompted, practical problem framing.

Datasets

• KWBench — total 223; splits: test (223); repo https://github.com/ankitmaloo/fasteval

Metrics

• Pass Rate (primary) — range: percent
  • Calculated as the number of tasks where all mandatory criteria were met divided by the total number of evaluated tasks. Tasks failing the mandatory gate receive a score of zero.
• Mean Score — range: percent
  • The average score across all evaluated tasks, including tasks that failed the mandatory gate (which contribute zero to the average).
• Conditional — range: percent
  • The mean score computed only on tasks that successfully passed the mandatory gate.

Input / output format

Input: Real-world knowledge work scenario descriptions presented without task hints or explicit prompts asking for structural analysis.

Output: Identification/classification of the underlying formal game-theoretic structure governing the scenario.

Scoring recipe

passed = 0
total_score = 0.0
for task in tasks:
    pred = model.predict(task)
    if meets_mandatory_gate(pred):
        passed += 1
        total_score += get_task_score(pred)
pass_rate = passed / len(tasks)
mean_score = total_score / len(tasks)
conditional_score = total_score / passed if passed > 0 else 0.0

Common pitfalls

• Models frequently correctly articulate game-theoretic structures when explicitly prompted, but fail to recognize them unprompted, creating a false impression of capability.
• API failures or malformed outputs can cause some models to be evaluated on fewer than the full task set, making raw pass counts misleading without normalizing by evaluated tasks.
• Conditional scores converge across models (~72-84%), so ranking differences stem almost entirely from pass rates rather than execution quality on passed tasks.

Evidence (verbatim from paper)

Passed: tasks where all mandatory criteria were met. Pass Rate: passed / evaluated. Mean Score: average across all evaluated tasks (including zeros). Conditional: mean score on tasks that passed the gate.

Citation

@misc{maloo2026kwbench,
  title={KWBench: Measuring Unprompted Problem Recognition in Knowledge Work},
  author={Maloo (2026)},
  year={2026},
  note={arXiv:2604.15760}
}

• arXiv: 2604.15760