Vibepass Eval

vibepass-eval

VIBEPASS: Can Vibe Coders Really Pass the Vibe Check? — Bansal et al. (2026) (arXiv:2603.15921, 2026)

What this evaluates

Evaluates LLMs on fault-targeted test generation and fault-targeted program repair. It probes discriminative fault detection, fault hypothesis generation, and the ability to debug subtle semantic bugs under diagnostic guidance.

Datasets

• VIBEPASS — total ?; splits: test (-1)

Metrics

• D_{IO} (primary) — range: percent
  • Discriminative input/output validity. Measures whether a generated test case has valid inputs and outputs AND successfully triggers a fault in the buggy code.
• J+D_{IO} — range: percent
  • Joint metric requiring both correct bug judgment (J) and discriminative test generation (D_{IO}).
• SR — range: percent
  • Success Rate for fault-targeted program repair, measuring whether the repaired code passes all validation checks.
• P@1 — range: percent
  • Pass@1 for code generation or repair, indicating the proportion of instances where the first generated solution is correct.

Input / output format

Input: Buggy code snippets with problem descriptions. For test generation: optionally a bug label (Bug-Aware) or none (Bug-Discovery). For repair: buggy code, problem description, and optionally a test case (NoTest, IntTest, ExtTest).

Output: Generated test cases (inputs and expected outputs), a binary judgment on whether the code is buggy, and/or a repaired version of the code.

Scoring recipe

def score_dio(generated_test, buggy_code):
    valid_io = check_input_output_validity(generated_test)
    triggers_fault = run_test_on_code(generated_test, buggy_code)
    return valid_io and triggers_fault

def score_sr(repaired_code, test_suite):
    return all(run_test_on_code(test, repaired_code) for test in test_suite)

Common pitfalls

• Confusing syntactic input validity (V_I) with discriminative fault-triggering capability (D_I), as models often generate format-compliant tests that fail to expose bugs.
• Assuming external test guidance always improves debugging performance; the evaluation shows self-generated tests or no-test baselines often outperform externally provided tests.
• Treating raw code generation ability (P@1) as a reliable proxy for fault-targeted program repair success.

Evidence (verbatim from paper)

In the Bug-Aware, the model knows the code is buggy, and performance measures FT-Test generation quality directly: $V_I / V_{IO}$ for input/output validity, and $D_I / D_{IO}$ for discriminative effectiveness. In the Bug-Discovery, J denotes judgment accuracy (buggy or not), and joint metrics $(J^{+*})$ require both correct judgment and the corresponding quality criterion.

Citation

@misc{bansal2026vibepass,
  title={VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?},
  author={Bansal et al. (2026)},
  year={2026},
  note={arXiv:2603.15921}
}

• arXiv: 2603.15921