Eng Langfuse Eval Runner

Langfuse Eval Runner

What it does

The eval runner executes systematic quality evaluations of skill outputs, using Langfuse as the orchestration layer. For each skill under test, it:

1. Takes a dataset of reference input/output pairs (golden set).
2. Runs the current skill version against the inputs.
3. Scores the outputs against the golden outputs using a judge model or custom rubric.
4. Writes scores back to Langfuse traces.
5. Computes aggregated pass rates and quality metrics.
6. Feeds results into the feature-flag promotion decision ([[eng-feature-flag-rollout-skills]]).

In a legal AI product, where the cost of a wrong output is potential malpractice exposure, systematic evaluation is non-optional for any skill that produces client-facing legal content.

Evaluation dataset structure

A Langfuse dataset item for a legal skill:

{
  "dataset_id": "efirm-conflict-check-eval-v1",
  "item_id": "ulid",
  "input": {
    "skill_id": "efirm-conflict-check",
    "context": {
      "new_client": "Al Baraka Holdings Ltd",
      "counterparties": ["Noor Capital SAOC", "Ali Hassan (individual)"],
      "matter_description": "Acquisition of minority stake in a KSA-incorporated company"
    },
    "user_message": "Run a conflict check for this new matter."
  },
  "expected_output": {
    "result": "CONCERN",
    "dimension_flagged": "former_client_check",
    "description": "Noor Capital SAOC was represented by the firm in matter 2023-UAE-0081 which may be substantially related"
  },
  "metadata": {
    "skill_version": "1.0",
    "practice_area": "corporate",
    "jurisdiction": "KSA",
    "difficulty": "medium",
    "created_by": "legal-qa-team",
    "reviewed_by": "partner-id"
  }
}

Rubrics by skill category

Conflict check rubrics

Drafting skill rubrics (engagement letter, NDA, fee quote)

Advisory / analysis skill rubrics

Judge model configuration

For automated scoring (as opposed to human review), the judge model evaluates each output:

judge_config:
  model: claude-sonnet-4-6
  system_prompt: |
    You are a senior legal quality reviewer. Evaluate the AI output against the 
    reference output and the rubric. Score each dimension 0–1 (continuous). 
    Return JSON: {dimension: score, overall: float, pass: bool, notes: string}.
    
    Legal accuracy is paramount. A single fabricated statute number or incorrect 
    jurisdiction claim is an automatic fail regardless of other scores.
    
  pass_threshold: 0.75        # overall score ≥ 0.75 = pass
  legal_accuracy_hard_fail: true   # any legal_accuracy < 0.9 = fail regardless

The judge model should never be the same model as the model being evaluated — use a different model or a more capable model as judge.

Running an eval

Single skill, current version

langfuse.run_eval(
    dataset_name="efirm-conflict-check-eval-v1",
    skill_id="efirm-conflict-check",
    skill_version="current",
    judge_config="legal-qa-rubric",
    run_name="conflict-check-v1-2025-05-14"
)

A/B comparison (two skill versions)

langfuse.run_eval(
    dataset_name="efirm-engagement-letter-eval-v2",
    experiments=[
        {"skill_version": "1.0", "name": "control"},
        {"skill_version": "1.1-test", "name": "treatment"}
    ],
    judge_config="drafting-rubric",
    significance_threshold=0.05
)

Eval result schema (written back to Langfuse)

{
  "run_id": "run_xxx",
  "skill_id": "efirm-conflict-check",
  "skill_version": "1.0",
  "dataset_id": "efirm-conflict-check-eval-v1",
  "n_items": 50,
  "pass_rate": 0.88,
  "avg_score": 0.83,
  "scores_by_dimension": {
    "result_classification": 0.92,
    "dimension_identification": 0.86,
    "no_false_negatives": 0.84,
    "description_quality": 0.79
  },
  "failures": [
    {"item_id": "xxx", "score": 0.61, "notes": "Missed corporate group adversity"}
  ],
  "promotion_recommendation": "PASS — meets 0.75 threshold",
  "run_timestamp": "ISO-8601"
}

Promotion gate

Integrate with [[eng-feature-flag-rollout-skills]]:

• If pass_rate ≥ 0.85 AND avg_score ≥ 0.80: auto-promote to next rollout stage.
• If pass_rate < 0.75 OR any legal-accuracy hard fail: block promotion; alert product team.
• Between 0.75 and 0.85: manual review by legal QA partner before promotion.

Golden set maintenance

• Golden sets must be reviewed and updated by a legal practitioner (not just engineering) at least every 6 months.
• When new legal developments occur (new legislation, regulatory guidance), update affected golden items.
• Target: minimum 30 items per skill for statistically meaningful results; 50–100 items for P0 skills.

Related skills

• [[eng-langfuse-trace-inspector]]
• [[eng-feature-flag-rollout-skills]]
• [[eng-latency-slo-by-skill]]
• [[eng-cost-per-message-tracker]]