Nemotron Nano V2 Vl Eval

nemotron-nano-v2-vl-eval

NVIDIA Nemotron Nano V2 VL — Deshmukh et al. (2025) (arXiv:2511.03929, 2025)

What this evaluates

Evaluates a 12B vision-language model's capabilities across multimodal understanding, long-context reasoning, document/OCR processing, video comprehension, and pure text reasoning. It probes the model's ability to handle diverse visual inputs, follow instructions, and perform complex STEM and code reasoning under varying decoding and reasoning budget constraints.

Datasets

• MMBench V1.1 — total ?; splits: dev (-1)
• MMMU — total ?; splits: val (-1)
• OCRBench — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• LongVideoBench — total ?; splits: test (-1)
• MATH-500 — total ?; splits: test (-1)
• GPQA-Diamond — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions or tasks. Calculated as (number of correct predictions / total number of instances) * 100. Reported as raw percentage scores in tables.
• Pass@1 — range: percent
  • Fraction of problems solved correctly in a single attempt. Averaged over 16 runs for AIME-2025 and 4 runs for other benchmarks like MATH-500 and GPQA-Diamond.

Input / output format

Input: Multimodal inputs consisting of images, videos, or documents paired with text prompts/questions. For pure text evaluations, inputs are text-only prompts. Reasoning budget constraints (2K, 4K, 8K, 12K tokens) can be applied to limit generation length.

Output: Text responses generated by the model. For reasoning-off mode: greedy decoding, max 1,024 tokens (16,384 for RDTableBench). For reasoning-on mode: temperature 0.6, top-p 0.95, max 16,384 tokens.

Scoring recipe

def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
    return (correct / len(gold)) * 100

def calculate_pass_at_1(predictions, gold, runs=16):
    total_correct = 0
    for i in range(runs):
        total_correct += sum(1 for p, g in zip(predictions[i], gold) if p.strip() == g.strip())
    return (total_correct / (len(gold) * runs)) * 100

Common pitfalls

• Reasoning-on vs reasoning-off modes use drastically different decoding parameters (greedy vs temperature 0.6/top-p 0.95) and token limits, which significantly impact scores.
• Some benchmark scores are taken directly from official model reports, while others are independently reproduced using VLMEvalKit, leading to potential discrepancies.
• Token generation limits vary by dataset (e.g., 1,024 tokens standard, but 16,384 for RDTableBench), which can affect performance on complex tasks if not configured correctly.

Evidence (verbatim from paper)

We report Pass@1 average of 16 runs for AIME-2025; an average of 4 runs for MATH-500, GPQA-Diamond, LiveCodeBench, IFEval; and score of 1 run for SciCode and RULER. For the reasoning-off mode, we employ greedy decoding and cap the maximum number of generated tokens at 1,024 for all benchmarks except RDTableBench where we use a limit of 16,384 tokens. For reasoning-on evaluations, we set the temperature to 0.6, top-p to 0.95, and the maximum output length to 16,384 tokens.

Citation

@misc{deshmukh2025nvidianemotronnanov2vl,
  title={NVIDIA Nemotron Nano V2 VL},
  author={Deshmukh et al. (2025)},
  year={2025},
  note={arXiv:2511.03929}
}

• arXiv: 2511.03929