Gistbench Eval

gistbench-eval

GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification — Fostiropoulos et al. (2026) (arXiv:2603.29112, 2026)

What this evaluates

Evaluates LLMs' ability to extract and verify user interests from interaction histories, focusing on factual grounding, specificity, and strict instruction following across heterogeneous engagement types.

Datasets

• Unspecified real-world engagement datasets — total ?; splits: (unstated); repo https://github.com/facebookresearch/GISTBench

Metrics

• IG (primary) — range: [0, 1]
  • Decomposed into precision and recall to penalize hallucinations and reward coverage of verified interests.
• IS — range: [0, 1]
  • Measures the distinctiveness of verified interests.

Input / output format

Input: Chunked user interaction history (UIH) with up to 100 engagements per prompt, formatted as numbered entries containing interaction type and object description, along with dataset-specific grounding thresholds in the system prompt.

Output: Model-generated list of identified interests with evidence citations referencing sequential object identifiers, ideally structured as JSON.

Scoring recipe

def compute_metrics(model_output, uih_chunks):
    pred = json_extract(repair(model_output))
    if pred is None: return None
    ig_precision = judge_l33_70b.verify_precision(pred.interests, uih_chunks)
    ig_recall = judge_l33_70b.verify_recall(pred.interests, uih_chunks)
    ig = harmonic_mean(ig_precision, ig_recall)
    is_score = judge_l33_70b.compute_distinctiveness(pred.interests)
    return {"IG": ig, "IS": is_score}

Common pitfalls

• JSON parsing failures require multi-stage extraction and repair before scoring.
• UIH chunking requires aggregation at the user level to avoid losing context across windows.
• Unparsable generations are strictly filtered out, which may bias results if a model consistently fails formatting.

Evidence (verbatim from paper)

The IG evidence-filtering judge and IS retrieval judge both used Llama-3.3-70B-Instruct. All intermediate outputs undergo best-effort JSON extraction: we first attempt standard parsing, then apply regex-based extraction to recover JSON objects embedded in free-text responses, and finally use automated JSON repair to fix common structural errors (unclosed brackets, trailing commas). Generations that remain unparsable after these recovery steps are filtered before metric computation.

Citation

@misc{fostiropoulos2026gistbench,
  title={GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification},
  author={Fostiropoulos et al. (2026)},
  year={2026},
  note={arXiv:2603.29112}
}

• arXiv: 2603.29112