Tact Eval

tact-eval

IE as Cache: Information Extraction Enhanced Agentic Reasoning — Hang Lv et al. (2026) (arXiv:2604.14930, 2026)

What this evaluates

Evaluates an agent's ability to perform logical deduction and multi-hop reasoning over long, noise-rich unstructured text without pre-defined schemas or tables. It probes the model's capacity to actively synthesize scattered evidence and filter out irrelevant distractors to arrive at a correct decision.

Datasets

• TACT — total ?; splits: test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • 1 if the predicted answer exactly matches the ground-truth answer, 0 otherwise. Measures whether the produced answer satisfies the target decision or constraint.

Input / output format

Input: Long-form unstructured text containing logical reasoning problems, paired with a query or instruction requiring deduction.

Output: A final answer string that satisfies the target decision or constraint.

Scoring recipe

def score_em(pred, gold):
    return 1.0 if pred.strip().lower() == gold.strip().lower() else 0.0

# Average over test set
em_scores = [score_em(p, g) for p, g in zip(predictions, gold_answers)]
final_metric = sum(em_scores) / len(em_scores)

Common pitfalls

• Models often fail to aggregate scattered evidence correctly when contexts are filled with irrelevant distractors.
• Exact match requires strict formatting; minor phrasing variations or extra punctuation will result in a score of 0.
• Static extraction baselines struggle because they cannot iteratively update their information state during multi-hop reasoning.

Evidence (verbatim from paper)

For TACT and Calendar Scheduling, we report Exact Match (EM), which directly measures whether the produced answer satisfies the target decision/constraints.

Citation

@misc{lv2026ieascache,
  title={IE as Cache: Information Extraction Enhanced Agentic Reasoning},
  author={Hang Lv et al. (2026)},
  year={2026},
  note={arXiv:2604.14930}
}

• arXiv: 2604.14930