Eval Regression Detector

Regression Detector

When to use this

The regression detector is called automatically at the end of every [[eval-benchmark-runner]] run, after scores are computed and before [[eval-leaderboard-updater]] records them. It compares current scores to the previous run for the same model and raises alerts if quality has degraded.

It is also the system that blocks automatic promotion from staging to production if a blocking-rubric regression is detected.

Inputs

Logic

Regression thresholds

A hard block means the CI pipeline returns a non-zero exit code and the deployment is not promoted. It must be manually overridden by a senior engineer with a written justification.

Regression computation

function detectRegressions(current: RunScores, previous: RunScores): RegressionReport {
  const regressions: Regression[] = [];

  for (const rubric of RUBRICS) {
    const currentScore = current.rubrics[rubric];
    const prevScore = previous.rubrics[rubric];
    const delta = currentScore - prevScore;
    const pct = delta / prevScore;

    if (pct < -THRESHOLDS[rubric]) {
      regressions.push({
        rubric,
        currentScore,
        prevScore,
        delta,
        pctChange: pct,
        severity: SEVERITIES[rubric],
        blocking: BLOCKING_RUBRICS.includes(rubric),
      });
    }
  }

  return {
    hasRegressions: regressions.length > 0,
    blockDeployment: regressions.some(r => r.blocking),
    regressions,
  };
}

Outputs

On regression detected:

1. Slack alert to #eng-quality with diff:

   ⚠️ Quality regression detected — claude-sonnet-4-5
   legal_soundness: 4.2 → 3.9 (-7.1%) [BLOCKING]
   citation_quality: 3.8 → 3.6 (-5.3%)
   Run: 2026-05-14 12:00 UTC
   [View details → Langfuse]
   

2. Linear ticket auto-created with:
   • Title: "Quality regression — {model} — {rubric} -{pct}%"
   • Description: regression context, top failing prompts, link to run report
   • Assignee: on-call engineer
   • Priority: P0 (for blocking) or P1 (for non-blocking)
3. Block deployment if blockDeployment = true — CI pipeline exits non-zero.

On no regression:

• Update leaderboard silently (no alert needed).
• Post a brief ✓ to #eng-quality only on the weekly summary, not every run.

Investigation flow

When a regression is detected, follow this procedure before manual override:

1. Identify which prompts regressed — sort prompts by score delta; focus on the bottom 10%.
2. Compare current vs previous responses side-by-side — often a single prompt change reveals the root cause.
3. Inspect skill routing — did the router send certain prompts to a different skill? Check skills_routed in the run log.
4. Check for model API changes — did the underlying model version or API behavior change (e.g., new system-prompt behavior, token limits)?
5. Check for config drift — did a system prompt, context window size, or temperature change?
6. Rollback if P0 — if legal_soundness or hallucination regressed and the cause is not identified within 2 hours, rollback the deployment.

Observability

Connect to:

• PostHog: emit eval_regression_detected event with rubric and delta properties.
• Langfuse: link regression report to the run trace for full observability.

Both signals feed the weekly engineering review.

Caveats & currency

• Some score variance between runs is expected (~1–2%) due to LLM judge non-determinism. Only flag regressions that exceed the threshold consistently across 2+ runs before treating them as genuine regressions.
• Recalibrate thresholds annually as the product matures. A threshold that was tight in early development may be too tight once the model quality plateaus at a high level.
• Track trend over absolute score — a model running at 4.2 for 3 months is more trustworthy than one that scored 4.5 once and is declining.

Related skills

• [[eval-benchmark-runner]] — calls this detector after each run
• [[eval-llm-as-judge-system-prompt]] — produces the scores that are compared
• [[eval-leaderboard-updater]] — records results after this detector runs
• [[eval-rubric-legal-soundness]] — primary blocking rubric
• [[eval-rubric-hallucination-detection]] — secondary blocking rubric