Tucano2 Portfolio Eval

tucano2-portfolio-eval

Tucano 2 Cool: Better Open Source LLMs for Portuguese — Corrêa et al. (2026) (arXiv:2603.03543, 2026)

What this evaluates

This evaluation protocol probes the language understanding, reasoning, and instruction-following capabilities of Portuguese LLMs across diverse domains including academic exams, natural language inference, physical commonsense, and code generation. It is specifically designed to provide reliable training signals during pretraining and assess post-training alignment.

Datasets

• ARC Challenge — total ?; splits: test (-1)
• Calame — total ?; splits: test (-1)
• Global PIQA — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• LAMBADA — total ?; splits: test (-1)
• ENEM — total ?; splits: test (-1)
• BLUEX — total ?; splits: test (-1)
• OAB — total ?; splits: test (-1)
• Belebele — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• IFEval-PT — total 300; splits: test (300)
• GSM8K-PT — total 1295; splits: test (1295)
• RULER-PT — total ?; splits: test (-1)
• HumanEval — total 164; splits: test (164)

Metrics

• accuracy (log-likelihood selection) (primary) — range: [0, 1]
  • For Multiple-Choice Format (MCF) and Cloze Formulation (CF), the model scores each candidate choice via conditional log-likelihood given the prompt. The option with the highest log-likelihood is selected as the prediction, and accuracy is computed as the fraction of correct selections.
• NPM — range: percent
  • Normalized Preferred Metric used to aggregate performance across benchmarks with different random baselines: NPM = (1/N) * sum_{i=1}^{N} 100 * (Preferred Metric_i - Random Score_i) / (Max Score_i - Random Score_i). Higher values indicate better relative performance.

Input / output format

Input: Prompt with 5-shot examples (pretraining suite) or 0-shot (HumanEval), followed by multiple-choice options or a continuation prompt. Post-training tasks include translated instruction prompts or code problems.

Output: Model generates a sequence of tokens. For MCF/CF tasks, the output is implicitly the selected choice via log-likelihood. For GSM8K-PT, HumanEval, and IFEval-PT, free-form text or code is generated.

Scoring recipe

def score_loglikelihood(prompt, choices, gold):
    log_probs = [model.log_prob(choice, prompt) for choice in choices]
    pred = choices[np.argmax(log_probs)]
    return pred == gold

def score_exact_match(pred, gold):
    return normalize(pred) == normalize(gold)

def score_pass_at_1(generated_code, test_cases):
    return run_tests(generated_code, test_cases)

Common pitfalls

• Using free-form generative scoring for pretraining benchmarks instead of log-likelihood selection, which causes high volatility and poor signal-to-noise ratios.
• Ignoring the distinction between Multiple-Choice Format (MCF) and Cloze Formulation (CF), as CF tasks provide earlier learning signals and higher SNR.
• Applying the same shot setting across all benchmarks; the protocol specifies 5-shot for the pretraining suite and 0-shot for HumanEval.

Evidence (verbatim from paper)

We also adopt the Normalized Preferred Metric (NPM) from Pires_2023 to aggregate performance across benchmarks with different random baselines: NPM = 1/N \sum_{i=1}^{N} 100 \times \frac{\text{Preferred Metric}{i}-\text{Random Score}{i}}{\text{Max Score}{i}-\text{Random Score}{i}}

Citation

@misc{correa2026tucano2cool,
  title={Tucano 2 Cool: Better Open Source LLMs for Portuguese},
  author={Corrêa et al. (2026)},
  year={2026},
  note={arXiv:2603.03543}
}

• arXiv: 2603.03543