Helo Apr Eval

helo-apr-eval

HELO-APR: Enhancing Low-Resource Program Repair through Cross-Lingual Knowledge Transfer — Wang et al. (2026) (arXiv:2604.17016, 2026)

What this evaluates

Evaluates a cross-lingual program repair framework on low-resource programming languages (Ruby and Rust) by measuring functional correctness against unit tests and syntactic validity via compilation or parsing rates.

Datasets

• xCodeEval (Compact Set for Ruby and Rust) — total ?; splits: test (-1)
• Defects4Ruby — total ?; splits: test (-1)

Metrics

• Pass@k (primary) — range: [0, 1]
  • Unbiased estimator: Pass@k = E[1 - C(n-c, k)/C(n, k)], where n=5 candidates per problem, c is the number of test-passing candidates, and k is the candidate budget (1, 3, or 5). Averages over all problems.
• Compilation Rate (CR) — range: percent
  • Percentage of top-1 generated patches that successfully compile (for Rust) or parse (for Ruby) in the ExecEval environment, independent of functional correctness.

Input / output format

Input: Buggy source code in Ruby or Rust, accompanied by problem descriptions and unit tests.

Output: A single fixed/patched source code file intended to resolve the bug.

Scoring recipe

def compute_pass_at_k(problems, n=5, k_values=[1,3,5]):
    results = {}
    for k in k_values:
        scores = []
        for prob in problems:
            c = sum(1 for cand in prob.candidates if cand.passes_tests())
            if n >= k:
                scores.append(1 - math.comb(n - c, k) / math.comb(n, k))
            else:
                scores.append(0.0)
        results[f'Pass@{k}'] = sum(scores) / len(scores)
    return results

def compute_cr(problems, top_1_patches):
    compiled = sum(1 for patch in top_1_patches if patch.compiles_or_parses())
    return (compiled / len(problems)) * 100

Common pitfalls

• Using the biased Pass@k estimator instead of the specified unbiased formula, which underestimates performance for small n.
• Ignoring the Compilation Rate metric, which is critical for assessing syntactic validity in low-resource languages where models often generate unparseable code.
• Failing to execute patches in the isolated ExecEval sandbox, leading to inconsistent unit test results across different evaluation runs.

Evidence (verbatim from paper)

We employ two metrics to evaluate both the functional correctness and the syntactic validity of the generated patches. Pass@k. Following standard practice, we use Pass@k as the primary metric to evaluate repair performance. Pass@k measures the probability that at least one of the top-k generated patches passes all unit tests, and is computed using the unbiased estimator: Pass@k := E[1 - C(n-c, k)/C(n, k)], where n is the number of generated candidates per problem, c is the number of test-passing candidates under the ExecEval environment, and k denotes the candidate budget. In our experiments, we set n=5 and report Pass@1, Pass@3, and Pass@5. Compilation Rate (CR). We use the compilation rate to assess the syntactic correctness of generated patches. It measures the percentage of top-1 generated patches that compile (for Rust) or parse successfully (for Ruby) in the ExecEval environment, independent of their functional correctness.

Citation

@misc{wang2026heloapr,
  title={HELO-APR: Enhancing Low-Resource Program Repair through Cross-Lingual Knowledge Transfer},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.17016}
}

• arXiv: 2604.17016