Core Capability Benchmarks Eval

core-capability-benchmarks-eval

The Amazon Nova Family of Models: Technical Report and Model Card — Amazon Nova Team et al. (2025) (arXiv:2506.12103, 2025)

What this evaluates

Evaluates foundational language, reasoning, instruction-following, and multimodal understanding capabilities across text, images, charts, documents, and video, alongside multilingual translation and agentic function-calling proficiency.

Datasets

• MMLU — total ?; splits: test (-1)
• GSM8K — total 1319; splits: test (1319)
• MATH — total 5000; splits: MATH5k (5000)
• IFEval — total 541; splits: test (541)
• Flores200 — total 842; splits: test (842)
• ChartQA — total 2500; splits: test (2500)
• DocVQA — total 5349; splits: test (5349)
• TextVQA — total 5000; splits: val (5000)
• Berkeley Function Calling Leaderboard (BFCL) — total ?; splits: test (-1)

Metrics

• exact match accuracy (primary) — range: [0, 1]
  • Proportion of predictions that exactly match the ground truth answer after normalization. Reported as macro average across subjects or tasks.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall computed over token-level matches between predicted and reference answers.
• ANLS — range: [0, 1]
  • Average Normalized Levenshtein Similarity; computes character-level edit distance between prediction and ground truth, normalized by the length of the longer string, with a threshold of 0.5 for zeroing out mismatches.
• CIDEr — range: [0, 1]
  • Consensus-based Image Description Evaluation metric that compares n-gram overlaps between generated captions and multiple reference captions, weighted by TF-IDF.
• spBleu — range: [0, 1]
  • Sentence-level BLEU score computed over tokenized translations, averaged across language pairs.

Input / output format

Input: Natural language prompts, optionally with Chain-of-Thought instructions. Multimodal inputs include images, charts, documents, or video frames paired with questions. Agentic inputs include tool/API descriptions and user requests.

Output: Textual answers, function call signatures (JSON/AST), or generated translations/captions. Evaluated against ground truth using exact match, relaxed match, F1, ANLS, CIDEr, or execution success.

Scoring recipe

def compute_metric(predictions, gold, metric_type):
    if metric_type == 'exact_match':
        return sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g)) / len(gold)
    elif metric_type == 'f1':
        return compute_token_f1(predictions, gold)
    elif metric_type == 'anls':
        return compute_levenshtein_similarity(predictions, gold, threshold=0.5)
    elif metric_type == 'execution':
        return sum(1 for p in predictions if execute_function_call(p) == gold) / len(predictions)
    return 0.0

Common pitfalls

• Prompting format varies significantly across benchmarks (0-shot, 0-shot CoT, few-shot, 3-shot, 25-shot); using the wrong format drastically changes scores.
• Confidence intervals are approximated using a Gaussian assumption for binary scores: CI = 1.96 * sqrt(S*(1-S)/N), which may not hold for small N or non-binary metrics.
• Some benchmarks require external OCR (e.g., DocVQA) or specific evaluation splits (e.g., MATH5k, GSM8K test set); using the wrong split or omitting OCR inflates/deflates results.

Evidence (verbatim from paper)

We use 0-shot Chain-of-Thought (CoT) [[79]] for prompting and report the macro average exact match accuracy across all subjects.

Citation

@misc{amazon2025nova,
  title={The Amazon Nova Family of Models: Technical Report and Model Card},
  author={Amazon Nova Team et al. (2025)},
  year={2025},
  note={arXiv:2506.12103}
}

• arXiv: 2506.12103