Contract
- Input: model list or candidate, task description, quality bar, budget constraints.
- Output: benchmark suite + comparison report + harness configuration.
- Scope: designs and documents evaluation; does not approve production model changes unless explicitly instructed.
- Rule: designs and documents evaluation; does not approve production model changes unless explicitly instructed.
- Rule: documented standards override defaults; explicit project rules take precedence.
- Rule: if blocked by missing context or dependencies, surface the blocker before proceeding.
Provenance
| Question |
Answer |
| What is the source of truth? |
The user's request, originating spec/issue, and the skill's declared outputs |
| What is in scope? |
Work covered by the skill's acceptance criteria and completion rules |
| What is explicitly out of scope? |
Files, behaviors, and decisions outside the skill's declared boundary |
| Who or what consumes this artifact afterward? |
The next skill in the workflow or the user |
| What evidence proves it is done? |
Completion criteria met, artifact saved, validation passed |
| What risk remains? |
Subjective judgment calls, missing context, or external dependency failures |
Artifact
Emit LlmEvalArtifact as both:
- JSON:
.agents/skills/platform/artifacts/llm-eval/{identifier}.json
- Markdown view: same filename with
.md extension
LLM Evaluation Harness
Design an LLM evaluation harness — benchmark design, model comparison, automated evals — with reproducible scoring, confidence intervals, and regression detection.
Process
1. Define evaluation scope
- Task types (classification, generation, reasoning, coding, tool use).
- Model candidates (base, instruct, fine-tuned, RAG-augmented).
- Success criteria (accuracy threshold, cost ceiling, latency ceiling).
- Evaluation audience (internal review, external publication, regulatory).
Completion criterion: evaluation scope and success criteria documented.
2. Benchmark design
- Task set (size, difficulty distribution, contamination checks).
- Prompt templates (system, few-shot, chain-of-thought variants).
- Golden answers or reference outputs (human-written or high-quality model outputs).
- Input formatting and sanitisation.
Completion criterion: benchmark tasks, prompts, and golden answers defined.
3. Scoring rubrics
- Automatic metrics (exact match, F1, BLEU, ROUGE, BERTScore, code pass rate).
- LLM-as-judge (structured rubric, pairwise comparison, calibration).
- Human-in-the-loop (sampling plan, adjudication rules, inter-annotator agreement).
- Confidence intervals and statistical tests (t-test, bootstrap, ANOVA).
Completion criterion: scoring rubric with calibration plan defined.
4. Model comparison protocol
- Paired evaluation (same prompts, same order, same grader).
- Blind or anonymised labels to reduce bias.
- Cost and latency tracking (tokens, wall time, $/1K tokens).
- Regression detection (delta thresholds, alerting rules).
Completion criterion: comparison protocol with regression rules defined.
5. Harness configuration
- Runner (local, distributed, CI-triggered).
- Model access (API keys, rate limiting, retries, fallback models).
- Result storage (database, artifact store, versioned dataset).
- CI integration (PR gating, scheduled runs, diff alerts).
Completion criterion: harness configuration with CI integration defined.
6. Reporting
- Leaderboard (primary metric, secondary metrics, cost/latency columns).
- Statistical significance (p-values, confidence intervals, effect sizes).
- Error analysis (failure modes, prompt sensitivity, edge cases).
- Reproducibility checklist (seeds, versions, environment).
Completion criterion: report template with reproducibility checklist defined.
Rules
- Rule: separate test set from development set; never tune prompts on test data.
- Rule: report variance across runs (temperature, sampling, seeds) alongside point estimates.
- Rule: track cost and latency as first-class metrics, not afterthoughts.
- Rule: version prompts, datasets, and model snapshots together for reproducibility.
- Rule: define regression thresholds before running experiments to avoid p-hacking.
Completion
- the skill's completion criteria are explicitly checked
- the artifact is saved and validated
- any blockers or skipped validations are documented
- the next consumer is identified or the work is handed off
@include .agents/skills/platform/contract-base.xml
1---2name: "llm-eval"3category: "ai"4maturity: "stable"5version: "1"6description: "Design LLM evaluation harnesses — benchmark design, model comparison, automated evals — with reproducible scoring, confidence intervals, and regression detection."7license: "MIT"8capabilities:9 - apply llm eval methods within explicit ai boundaries10 - analyze design llm evaluation harnesses — benchmark design, model comparison, automated evals — with reproducible scoring, confidence intervals, and regression detection with assumptions, constraints, and failure modes11 - produce and validate a ai artifact with evidence, decisions, and next steps12outputs: "benchmark suite + comparison report + harness configuration."13sideEffects: []14dependencies: []15stopCondition: "Benchmark suite defined; comparison protocol documented; harness configuration complete."16risk: "medium"17trustTier: "3"18maxIterations: "6"19promptVersion: "2.0"20artifactType: "ai"21modelTier: "reasoning"22evaluators: ["behavioral", "regression"]23fixturesPath: .agents/skills/platform/fixtures/behavioral/llm-eval.json24diataxis: "how-to"25tags: ["ai"]26compatibility: []27approvalRequired: "true"28approvalFor: []29metadata:30 source: https://skillmd.com/skills/quantumquirkxyz/llm-eval31---3233## Contract3435- Input: model list or candidate, task description, quality bar, budget constraints.36- Output: benchmark suite + comparison report + harness configuration.37- Scope: designs and documents evaluation; does not approve production model changes unless explicitly instructed.38- Rule: designs and documents evaluation; does not approve production model changes unless explicitly instructed.39- Rule: documented standards override defaults; explicit project rules take precedence.40- Rule: if blocked by missing context or dependencies, surface the blocker before proceeding.4142## Provenance4344| Question | Answer |45|---|---|46| What is the source of truth? | The user's request, originating spec/issue, and the skill's declared outputs |47| What is in scope? | Work covered by the skill's acceptance criteria and completion rules |48| What is explicitly out of scope? | Files, behaviors, and decisions outside the skill's declared boundary |49| Who or what consumes this artifact afterward? | The next skill in the workflow or the user |50| What evidence proves it is done? | Completion criteria met, artifact saved, validation passed |51| What risk remains? | Subjective judgment calls, missing context, or external dependency failures525354## Artifact5556Emit `LlmEvalArtifact` as both:57- JSON: `.agents/skills/platform/artifacts/llm-eval/{identifier}.json`58- Markdown view: same filename with `.md` extension596061# 62# 63# LLM Evaluation Harness6465Design an **LLM evaluation harness** — benchmark design, model comparison, automated evals — with reproducible scoring, confidence intervals, and regression detection.6667## Process6869### 1. Define evaluation scope70- Task types (classification, generation, reasoning, coding, tool use).71- Model candidates (base, instruct, fine-tuned, RAG-augmented).72- Success criteria (accuracy threshold, cost ceiling, latency ceiling).73- Evaluation audience (internal review, external publication, regulatory).7475**Completion criterion:** evaluation scope and success criteria documented.7677### 2. Benchmark design78- Task set (size, difficulty distribution, contamination checks).79- Prompt templates (system, few-shot, chain-of-thought variants).80- Golden answers or reference outputs (human-written or high-quality model outputs).81- Input formatting and sanitisation.8283**Completion criterion:** benchmark tasks, prompts, and golden answers defined.8485### 3. Scoring rubrics86- Automatic metrics (exact match, F1, BLEU, ROUGE, BERTScore, code pass rate).87- LLM-as-judge (structured rubric, pairwise comparison, calibration).88- Human-in-the-loop (sampling plan, adjudication rules, inter-annotator agreement).89- Confidence intervals and statistical tests (t-test, bootstrap, ANOVA).9091**Completion criterion:** scoring rubric with calibration plan defined.9293### 4. Model comparison protocol94- Paired evaluation (same prompts, same order, same grader).95- Blind or anonymised labels to reduce bias.96- Cost and latency tracking (tokens, wall time, $/1K tokens).97- Regression detection (delta thresholds, alerting rules).9899**Completion criterion:** comparison protocol with regression rules defined.100101### 5. Harness configuration102- Runner (local, distributed, CI-triggered).103- Model access (API keys, rate limiting, retries, fallback models).104- Result storage (database, artifact store, versioned dataset).105- CI integration (PR gating, scheduled runs, diff alerts).106107**Completion criterion:** harness configuration with CI integration defined.108109### 6. Reporting110- Leaderboard (primary metric, secondary metrics, cost/latency columns).111- Statistical significance (p-values, confidence intervals, effect sizes).112- Error analysis (failure modes, prompt sensitivity, edge cases).113- Reproducibility checklist (seeds, versions, environment).114115**Completion criterion:** report template with reproducibility checklist defined.116117## Rules118119- Rule: separate test set from development set; never tune prompts on test data.120- Rule: report variance across runs (temperature, sampling, seeds) alongside point estimates.121- Rule: track cost and latency as first-class metrics, not afterthoughts.122- Rule: version prompts, datasets, and model snapshots together for reproducibility.123- Rule: define regression thresholds before running experiments to avoid p-hacking.124125## Completion126127- the skill's completion criteria are explicitly checked128- the artifact is saved and validated129- any blockers or skipped validations are documented130- the next consumer is identified or the work is handed off131---132@include .agents/skills/platform/contract-base.xml