Gervasio Pt Eval

gervasio-pt-eval

Advancing Generative AI for Portuguese with Open Decoder Gerv'asio PT* — Santos et al. (2024) (arXiv:2402.18766, 2024)

What this evaluates

Evaluates instruction-tuned decoder-only LLMs on Portuguese language understanding tasks, including natural language inference, paraphrase detection, commonsense reasoning, and multiple-choice question answering.

Datasets

• MRPC — total ?; splits: test (-1)
• RTE — total ?; splits: test (-1)
• COPA — total ?; splits: test (-1)
• ENEM 2022 — total ?; splits: test (-1)
• BLUEX — total ?; splits: test (-1)
• STS — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Macro-averaged F1 score computed over predicted and gold labels for classification tasks.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted classes for multi-choice tasks.
• Pearson correlation coefficient — range: [-1, 1]
  • Pearson r measuring linear correlation between predicted and gold similarity scores.

Input / output format

Input: Instruction prompt containing randomly selected few-shot examples and a test instance, formatted for a decoder-only language model.

Output: Free-form generated text. For classification tasks, the first word (e.g., 'sim'/'não') or first digit is extracted and matched against the gold label.

Scoring recipe

def score(predictions, golds, task):
    # Extract first word/digit from generation
    preds = [p.strip().split()[0] for p in predictions]
    if task in ['MRPC', 'RTE', 'COPA']:
        return f1_score(golds, preds, average='macro')
    elif task in ['ENEM 2022', 'BLUEX']:
        return accuracy_score(golds, preds)
    elif task == 'STS':
        return pearsonr(golds, preds)[0]

Common pitfalls

• Baseline scores (e.g., Sabiá) are reported from a single run, while Gervásio scores are averaged over three independent seeds, potentially inflating performance gaps.
• Sabiá uses constrained likelihood decoding over candidate classes, whereas Gervásio uses unconstrained generation with post-hoc exact matching, making direct metric comparison methodologically inconsistent.
• Only the first word or digit of the generated response is considered, which may discard valid multi-token answers or conversational fillers.

Evidence (verbatim from paper)

For every task under evaluation, we use the respective evaluation metrics commonly found in the literature, typically the F1 score or the Pearson correlation coefficient, as indicated below. Each performance score reported below is the average of the outcome of three independent runs using different seeds.

Citation

@misc{santos2024gervasiopt,
  title={Advancing Generative AI for Portuguese with Open Decoder Gerv'asio PT*},
  author={Santos et al. (2024)},
  year={2024},
  note={arXiv:2402.18766}
}

• arXiv: 2402.18766