Agent Evals and Observability
Evaluation asks whether behavior meets a defined criterion on a declared dataset or production sample. Observability supplies traces, logs, metrics, correlations, and diagnostic context. Use both; neither proves what the other does.
Workflow
- Define the decision, risk, task contract, trajectory contract, and unacceptable outcomes. Select evidence by harm, reversibility, and deployment stage, not a staged completeness scale.
- Create an immutable dataset version and manifest before comparing versions. Declare provenance, rights/consent, slices, fixtures, expected side effects, contamination risk, limitations, retention, and changelog.
- Select complementary graders that observe the claimed property. Use deterministic checks for observable mechanics; use execution/environment checks for state and side effects; use human, model-judge, pairwise/ranking, domain, and safety review where appropriate.
- Run the candidate and baseline under comparable conditions. Preserve run configuration, stochastic repeats where variability affects the decision, failures/timeouts, and trajectory evidence.
- Report a multidimensional profile and uncertainty. Use paired comparisons where possible; inspect slices, missingness, base rates, confounders, effect sizes, and multiple comparisons. Non-significance is not equivalence.
- Apply a risk-tiered release gate: hard safety, privacy, authorization, and side-effect invariants cannot be averaged away. Record authority, insufficient-evidence outcomes, rollback, and follow-up. Load release-engineering when this evidence must be incorporated into artifact promotion, deployment, rollback, or a broader release train.
- Instrument production with minimized, redacted telemetry. Feed verified incidents and near misses into cases after consent, transformation, and contamination review.
Stop when the supported decision, evidence gaps, residual risks, and responsible owner are recorded. Escalate rather than infer a pass when required evidence is unavailable or conflicts.
Load By Need
| Need |
Load |
| Choose evaluation evidence and contracts |
references/evaluation-design.md |
| Build immutable cases, fixtures, and provenance |
references/datasets.md |
| Select or calibrate graders |
references/graders.md |
| Define measures or compare runs |
references/metrics-and-statistics.md |
| Review tools, state, recovery, or side effects |
references/trajectory-review.md |
| Triage a regression or decide release readiness |
references/regression-and-release.md |
| Design traces, logs, metrics, or privacy controls |
references/production-observability.md |
| Interoperate with OpenTelemetry |
references/opentelemetry-genai.md |
| Need framework-specific implementation |
references/framework-routing.md |
| Exercise the methodology with safe probes |
references/synthetic-scenarios.md |
| Benchmark an agent's Supabase task competence with the official harness |
supabase agent evals reference |
| Verify a source claim or status |
references/source-index.md |
Templates
Use templates/eval-plan.md, templates/dataset-manifest.md, templates/grader-specification.md, templates/trace-review.md, templates/run-report.md, and templates/release-gate.md. They are decision records, not checklists that manufacture evidence.
Guardrails
- Do not use a scalar score, keyword hit, schema validation, or repeated output as proof beyond the property it directly observes.
- Do not capture prompts, outputs, tool arguments, intermediate reasoning, credentials, or personal data by default. Minimize before export, redact early, restrict access, set retention/deletion paths, and prepare incident response.
- Do not treat human judgment as definitive truth. Human and model graders require explicit rubrics, calibration, disagreement analysis, and revalidation when conditions change.
- Keep prevalence-oriented production samples separate from risk-enriched challenge cases. Do not silently reweight either into the other.
When Not To Use
Use systematic-debugging to investigate one active defect before proposing fixes. Use verification-methodology for general evidence-backed completion claims, secure-software-engineering to design preventive controls, and framework skills for SDK-specific code. To benchmark an agent's competence on Supabase tasks with the official scored scenarios, use the supabase agent evals reference — this skill covers methodology, not a named harness.
1---2name: agent-evals-and-observability3description: Design, run, review, or release framework- and vendor-neutral evaluations and observability for AI agents. Use when defining agent evals, datasets, graders, trajectory review, regression analysis, release gates, production traces, or privacy-aware telemetry. Covers task and trajectory contracts, statistical comparisons, and incident-to-case learning; route framework implementation to pydanticai or langgraph when needed. Do not use this skill for unrelated requests; route to the nearest named specialist.4license: MIT5---67# Agent Evals and Observability89Evaluation asks whether behavior meets a defined criterion on a declared dataset or production sample. Observability supplies traces, logs, metrics, correlations, and diagnostic context. Use both; neither proves what the other does.1011## Workflow12131. Define the decision, risk, task contract, trajectory contract, and unacceptable outcomes. Select evidence by harm, reversibility, and deployment stage, not a staged completeness scale.142. Create an immutable dataset version and manifest before comparing versions. Declare provenance, rights/consent, slices, fixtures, expected side effects, contamination risk, limitations, retention, and changelog.153. Select complementary graders that observe the claimed property. Use deterministic checks for observable mechanics; use execution/environment checks for state and side effects; use human, model-judge, pairwise/ranking, domain, and safety review where appropriate.164. Run the candidate and baseline under comparable conditions. Preserve run configuration, stochastic repeats where variability affects the decision, failures/timeouts, and trajectory evidence.175. Report a multidimensional profile and uncertainty. Use paired comparisons where possible; inspect slices, missingness, base rates, confounders, effect sizes, and multiple comparisons. Non-significance is not equivalence.186. Apply a risk-tiered release gate: hard safety, privacy, authorization, and side-effect invariants cannot be averaged away. Record authority, insufficient-evidence outcomes, rollback, and follow-up. Load [release-engineering](../release-engineering/SKILL.md) when this evidence must be incorporated into artifact promotion, deployment, rollback, or a broader release train.197. Instrument production with minimized, redacted telemetry. Feed verified incidents and near misses into cases after consent, transformation, and contamination review.2021Stop when the supported decision, evidence gaps, residual risks, and responsible owner are recorded. Escalate rather than infer a pass when required evidence is unavailable or conflicts.2223## Load By Need2425| Need | Load |26|---|---|27| Choose evaluation evidence and contracts | [references/evaluation-design.md](references/evaluation-design.md) |28| Build immutable cases, fixtures, and provenance | [references/datasets.md](references/datasets.md) |29| Select or calibrate graders | [references/graders.md](references/graders.md) |30| Define measures or compare runs | [references/metrics-and-statistics.md](references/metrics-and-statistics.md) |31| Review tools, state, recovery, or side effects | [references/trajectory-review.md](references/trajectory-review.md) |32| Triage a regression or decide release readiness | [references/regression-and-release.md](references/regression-and-release.md) |33| Design traces, logs, metrics, or privacy controls | [references/production-observability.md](references/production-observability.md) |34| Interoperate with OpenTelemetry | [references/opentelemetry-genai.md](references/opentelemetry-genai.md) |35| Need framework-specific implementation | [references/framework-routing.md](references/framework-routing.md) |36| Exercise the methodology with safe probes | [references/synthetic-scenarios.md](references/synthetic-scenarios.md) |37| Benchmark an agent's Supabase task competence with the official harness | [supabase agent evals reference](../supabase/references/agent-evals.md) |38| Verify a source claim or status | [references/source-index.md](references/source-index.md) |3940## Templates4142Use [templates/eval-plan.md](templates/eval-plan.md), [templates/dataset-manifest.md](templates/dataset-manifest.md), [templates/grader-specification.md](templates/grader-specification.md), [templates/trace-review.md](templates/trace-review.md), [templates/run-report.md](templates/run-report.md), and [templates/release-gate.md](templates/release-gate.md). They are decision records, not checklists that manufacture evidence.4344## Guardrails4546- Do not use a scalar score, keyword hit, schema validation, or repeated output as proof beyond the property it directly observes.47- Do not capture prompts, outputs, tool arguments, intermediate reasoning, credentials, or personal data by default. Minimize before export, redact early, restrict access, set retention/deletion paths, and prepare incident response.48- Do not treat human judgment as definitive truth. Human and model graders require explicit rubrics, calibration, disagreement analysis, and revalidation when conditions change.49- Keep prevalence-oriented production samples separate from risk-enriched challenge cases. Do not silently reweight either into the other.5051## When Not To Use5253Use [systematic-debugging](../systematic-debugging/SKILL.md) to investigate one active defect before proposing fixes. Use [verification-methodology](../verification-methodology/SKILL.md) for general evidence-backed completion claims, [secure-software-engineering](../secure-software-engineering/SKILL.md) to design preventive controls, and framework skills for SDK-specific code. To benchmark an agent's competence on Supabase tasks with the official scored scenarios, use the [supabase agent evals reference](../supabase/references/agent-evals.md) — this skill covers methodology, not a named harness.
magnus919