Multipl E Low Resource Eval

multipl-e-low-resource-eval

Enhancing Code Generation for Low-Resource Languages: No Silver Bullet — Giagnorio et al. (2025) (arXiv:2501.19085, 2025)

What this evaluates

Evaluates large language models' ability to generate functionally correct code in low-resource programming languages (R and Racket). It probes how well in-context learning strategies and fine-tuning adapt pre-trained models to languages with limited training data and documentation.

Datasets

• MultiPL-E — total 161; splits: test (161)

Metrics

• pass@1 (primary) — range: [0, 1]
  • The probability that at least one of the n=50 generated code samples is functionally correct. Computed as 1 - (C(n-c, k) / C(n, k)) where c is the number of correct samples, n=50, and k=1. In practice, it scores 1.0 if any of the 50 samples passes all unit tests, else 0.0.

Input / output format

Input: Function docstring and signature, optionally augmented with in-context examples (Python-to-target translations, translation rules, or few-shot target-language examples) depending on the technique.

Output: Function body implementation (concrete code) in the target low-resource language (R or Racket).

Scoring recipe

def compute_pass_at_1(predictions, test_cases):
    # predictions: list of 50 generated code strings
    # test_cases: list of unit tests for the function
    passed = [run_tests(code, test_cases) for code in predictions]
    # pass@1 = 1 if at least one sample passes, else 0
    return 1.0 if any(passed) else 0.0

Common pitfalls

• Fine-tuned models are evaluated by reporting only the best epoch's results, not the final epoch, to avoid performance degradation from overfitting on small datasets.
• pass@1 is computed over 50 sampled outputs per program, but the metric only requires one successful sample to score 1.0, which can mask partial correctness or inconsistent generation.
• The 161 programs are specific to R and Racket subsets; they do not represent the full MultiPL-E benchmark which covers many other languages.

Evidence (verbatim from paper)

As in our previous study, our evaluation metric is the pass@1 rate with n = 50 repetitions. In this case, instead of performance across languages, we compare performance across techniques.

Citation

@misc{giagnorio2025enhancing,
  title={Enhancing Code Generation for Low-Resource Languages: No Silver Bullet},
  author={Giagnorio et al. (2025)},
  year={2025},
  note={arXiv:2501.19085}
}

• arXiv: 2501.19085