Pass At K

pass-at-k — Unbiased pass@k for code generation

Source: Chen et al., "Evaluating Large Language Models Trained on Code" (Codex paper, arXiv:2107.03374, §2.1). The unbiased estimator avoids the bias of the naive 1 - (1 - p)^k plug-in when you only sampled a finite n.

When to invoke this skill

• User has, per problem: n total samples, c of which pass the unit tests, and wants pass@k for some k ≤ n
• User says "compute pass@k", "HumanEval-style scoring", "pass@1 / pass@10 / pass@100"
• User mentions code-gen benchmarks (HumanEval, MBPP, APPS, LiveCodeBench, BigCodeBench)

Definition

For a single problem with n total samples, c correct samples, and target cutoff k:

pass@k = 1 - C(n - c, k) / C(n, k)        if n - c >= k
       = 1                                 if n - c < k

Numerically stable form (paper-standard):

pass@k = 1 - prod_{i=n-c+1..n} (1 - k / i)        # iff k <= n - c

Then average over all problems in the dataset. Use the unbiased estimator above, not 1 - (1 - c/n)^k.

Reference implementation

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased pass@k for one problem. n total samples, c correct, cutoff k."""
    if k > n:
        raise ValueError(f"k={k} > n={n}; need to sample at least k completions")
    if n - c < k:
        return 1.0
    # 1 - prod_{i in [n-c+1 .. n]} (1 - k/i)
    return 1.0 - float(np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

def mean_pass_at_k(results: list[tuple[int, int]], k: int) -> float:
    """results = list of (n, c) per problem."""
    return float(np.mean([pass_at_k(n, c, k) for n, c in results]))

# example usage
# Per problem you sampled n=20 times and ran tests on each.
# results = [(20, 14), (20, 0), (20, 7), ...]
print(mean_pass_at_k(results, k=1))   # 0.35 — typical
print(mean_pass_at_k(results, k=10))  # higher; closer to coverage

The HuggingFace evaluate library has the same impl: evaluate.load("code_eval").

Input contract

• Per problem: integer n (samples drawn) and integer c (samples that passed all unit tests).
• k: cutoff. Common reporting: pass@1 (greedy / temp 0.2 single sample), pass@10, pass@100. Must satisfy k ≤ n.
• Sampling: paper uses temperature 0.8 for pass@10/100, 0.2 for pass@1. Don't claim a high pass@k by sampling at temperature 0.

Output format

HumanEval (164 problems, n=200, T=0.8):
  pass@1  = 0.282
  pass@10 = 0.461
  pass@100= 0.722

Edge cases

• c == 0: contribution is exactly 0 for any k (no correct sample to draw).
• c == n: contribution is exactly 1 for any k ≤ n.
• k > n: undefined — your sampling protocol is incompatible. Either raise or downgrade k to n.
• Non-determinism in unit tests (timing flakes): re-run flaky tests, or wrap with a 60s timeout + fixed seeds. Do NOT count timeouts as pass.

Don'ts

• Don't use 1 - (1 - c/n)^k — biased upward when c is small.
• Don't compute pass@1 by sampling once and calling it pass@1 if n=1 — variance is enormous; always sample n≥10 even for pass@1 and use the unbiased estimator.
• Don't rely on the model-output being valid Python — wrap unit-test execution in a sandbox + timeout. The paper warns about this explicitly.

Citation

@article{chen2021codex,
  title={Evaluating Large Language Models Trained on Code},
  author={Chen, Mark and Tworek, Jerry and Jun, Heewoo and others},
  journal={arXiv:2107.03374}, year={2021}
}