Slump Eval

slump-eval

When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents — Lu Yan et al. (2026) (arXiv:2603.17104, 2026)

What this evaluates

Measures how much final code fidelity degrades when a system's design is progressively disclosed through multi-turn interaction rather than provided upfront. It evaluates semantic faithfulness to a committed design and structural integration of dependencies in long-horizon coding agents.

Datasets

• SLUMP benchmark — total 20; splits: test (20)

Metrics

• MCF — range: [0, 4]
  • Mean Component Faithfulness. Computes the average rubric score $s_i \in {0,1,2,3,4}$ across all $N$ components in a paper: $\frac{1}{N} \sum_{i=1}^N s_i$.
• DIR — range: [0, 1]
  • Dependency Integration Ratio. Measures structural integration by dividing the number of public symbols actually reused by downstream code by the total number of exported symbols: $|E_{used}| / |E_{total}|$.
• IF50 (primary) — range: [0, 1]
  • Implementation Faithfulness. Normalized summary metric combining semantic faithfulness and structural integration: $\frac{1}{2}(\frac{MCF}{4}) + \frac{1}{2}DIR$. Serves as the primary endpoint metric for SLUMP.

Input / output format

Input: Multi-turn interaction script (emergent condition) or complete paper text in a single prompt (single-shot control).

Output: Final integrated code repository implementing the target design.

Scoring recipe

def compute_metrics(code, components):
    # MCF: rubric score 0-4 per component
    mcf_scores = [rubric_score(comp) for comp in components]
    MCF = sum(mcf_scores) / len(components)
    
    # DIR: count exported vs reused public symbols
    E_total = count_public_symbols_exported(code)
    E_used = count_public_symbols_reused_in_downstream(code)
    DIR = E_used / E_total if E_total > 0 else 0.0
    
    # IF50: primary endpoint
    IF50 = 0.5 * (MCF / 4) + 0.5 * DIR
    return MCF, DIR, IF50

Common pitfalls

• Executable test pass rates do not reliably detect faithfulness loss, as they may validate the produced code without verifying alignment with the intended design.
• Faithfulness loss occurs during final integration of previously implemented components, not merely during isolated implementation.

Evidence (verbatim from paper)

We report three metrics. Mean Component Faithfulness (MCF) measures semantic faithfulness to the committed design. Let $s_i \in {0,1,2,3,4}$ be the rubric score for component $i$ and let $N$ be the number of components in the paper. Then MCF = 1/N \sum_{i=1}^N s_i. MCF ranges from 0 to 4. Dependency Integration Ratio (DIR) (Le Hai et al., 2025) measures structural integration: DIR = |E_used| / |E_total|, where $E_{total}$ is the set of public symbols exported by standalone modules and $E_{used}$ is the subset actually reused by downstream integration code. DIR ranges from 0 to 1. To summarize both dimensions, we define implementation faithfulness IF50 = 1/2 (MCF/4) + 1/2 DIR. IF50 ranges from 0 to 1 and serves as our primary endpoint metric for SLUMP.

Citation

@misc{yan2026slump,
  title={When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents},
  author={Lu Yan et al. (2026)},
  year={2026},
  note={arXiv:2603.17104}
}

• arXiv: 2603.17104