Iao Prompting Eval

iao-prompting-eval

IAO Prompting: Making Knowledge Flow Explicit in LLMs through Structured Reasoning Templates — Diallo et al. (2025) (arXiv:2502.03080, 2025)

What this evaluates

Evaluates LLMs' ability to perform structured reasoning and knowledge application across arithmetic, logical, commonsense, and symbolic tasks using a template-based prompting framework.

Datasets

• GSM8K — total ?; splits: test (-1)
• AQuA — total ?; splits: test (-1)
• Date Understanding — total ?; splits: test (-1)
• Object Tracking — total ?; splits: test (-1)
• StrategyQA — total ?; splits: test (-1)
• CommonsenseQA — total ?; splits: test (-1)
• Last Letter — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions, calculated as (number of correct predictions / total number of instances) * 100.

Input / output format

Input: Natural language question or problem statement, optionally preceded by a structured reasoning template (Input-Action-Output steps) in a zero-shot setting.

Output: A single extracted answer or final solution, obtained via a dedicated answer extraction prompt in a single API call.

Scoring recipe

def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100

Common pitfalls

• Evaluations are strictly zero-shot; any few-shot demonstrations violate the protocol.
• All baselines and IAO must use the identical answer extraction prompt in a single stage to ensure fair comparison.
• Model scale significantly impacts template adherence; results vary notably between PALM-2 and GPT-4.

Evidence (verbatim from paper)

Table 1: Evaluation results for PALM-2. Bold denotes best result. All methods use the same answer extraction prompt in a single stage for fair comparison. All methods are evaluated under the zero-shot setting.

Citation

@misc{diallo2025iaoprompting,
  title={IAO Prompting: Making Knowledge Flow Explicit in LLMs through Structured Reasoning Templates},
  author={Diallo et al. (2025)},
  year={2025},
  note={arXiv:2502.03080}
}

• arXiv: 2502.03080