Eureka Bench Eval

eureka-bench-eval

Eureka: Evaluating and Understanding Large Foundation Models — Balachandran et al. (2024) (arXiv:2409.10566, 2024)

What this evaluates

Granular, capability-level analysis of large foundation models across multimodal reasoning, language understanding, safety, and stability. It dissects performance across fine-grained subcategories (e.g., geometric depth vs. height, single vs. multi-object detection) to reveal persistent failures and complementary strengths across models.

Datasets

• EUREKA-BENCH — total ?; splits: test (-1); repo https://github.com/microsoft/eureka-ml-insights

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered instances or correctly classified items across tasks.
• instruction_following_rate — range: [0, 1]
  • Proportion of model outputs that satisfy explicit instruction constraints.
• fact_precision — range: [0, 1]
  • Query constraint satisfaction rate; proportion of generated facts that are correct.
• fact_recall — range: [0, 1]
  • Completeness rate; proportion of required facts successfully retrieved or generated.
• disagreement_rate — range: [0, 1]
  • Percentage of instances where outputs differ across three identical runs (temp=0, top_p=0.95).

Input / output format

Input: Text prompts, often paired with images for multimodal tasks. Inputs include constrained queries, long-context documents, and toxic/neutral content for safety evaluation.

Output: Text responses or classification labels. For determinism analysis, three identical outputs per prompt are collected and compared.

Scoring recipe

def compute_metrics(predictions, gold, runs=None):
    acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    if_rate = sum(1 for p in predictions if satisfies_instruction(p)) / len(predictions)
    precision = sum(1 for p in predictions if is_factually_correct(p)) / len(predictions)
    recall = sum(1 for p in predictions if covers_required_facts(p)) / len(gold)
    if runs:
        disagree = sum(1 for i in range(len(predictions)) if len(set(r[i] for r in runs)) > 1)
        disagree_rate = disagree / len(predictions)
    else:
        disagree_rate = 0
    return {'accuracy': acc, 'instruction_following_rate': if_rate, 'fact_precision': precision, 'fact_recall': recall, 'disagreement_rate': disagree_rate}

Common pitfalls

• Assuming a single aggregate leaderboard score reflects overall model capability; the benchmark shows highly complementary performance across models.
• Ignoring non-determinism; identical runs can yield 1–4% performance fluctuations at the subcategory level, skewing comparisons.
• Overlooking modality fusion effects; most models perform worse on multimodal tasks than equivalent language-only tasks, except for specific models like GPT-4o.

Evidence (verbatim from paper)

Amongst the studied language capabilities, instruction following is where most models are improving faster, potentially due to strong investments in instruction tuning processes, with most models now having an instruction following rate of higher than 75%.

Citation

@misc{balachandran2024eureka,
  title={Eureka: Evaluating and Understanding Large Foundation Models},
  author={Balachandran et al. (2024)},
  year={2024},
  note={arXiv:2409.10566}
}

• arXiv: 2409.10566