AI Reliability Evaluation
Version-Timestamp: 2026-08-04 13:10:00 AST
Use this skill to replace subjective AI approval with measured behavior, clear thresholds, and controlled release decisions.
Scope
Evaluate any combination of:
- Prompt behavior.
- Model choice and routing.
- Tool permissions and tool outputs.
- Agent planning and handoffs.
- RAG ingestion, chunking, embeddings, retrieval, reranking, citations, and freshness.
- Vector database design, metadata filters, updates, deletions, recall, latency, and tenant boundaries.
- Observability for quality, latency, cost, tokens, errors, traces, feedback, and incidents.
Safety Rules
- Treat sending prompts, documents, traces, or evaluation rows to an external provider as data publication.
- Use synthetic, public, anonymized, or explicitly approved data.
- Do not test prompt injection, data extraction, tool misuse, privilege escalation, or denial-of-service behavior outside authorized scope.
- Do not rely on evaluator-model scores alone for high-impact safety, legal, financial, medical, or compliance claims.
- Never log secrets, private user content, raw credentials, or unnecessary personal data.
Evaluation Design
Define:
- Task and user goal.
- Model, prompt, tool, retrieval, and policy versions.
- Success criteria and release thresholds.
- Representative cases.
- Edge cases.
- Refusal and safety cases.
- Unanswerable cases.
- Cross-tenant and access-boundary cases.
- Expected evidence or rubric.
- Human-review sampling plan.
Metrics
Choose metrics that fit the system:
- Task success.
- Format compliance.
- Groundedness.
- Citation support.
- Retrieval precision and recall.
- Context relevance.
- Refusal correctness.
- Tool-call correctness.
- Latency.
- Cost.
- Consistency.
- Failure severity.
Failure Triage
Classify failures by layer:
- Data source.
- Ingestion.
- Chunking.
- Embeddings.
- Vector search.
- Reranking.
- Prompt.
- Model.
- Tool contract.
- Authorization policy.
- User interface.
- Observability gap.
Fix the layer that caused the failure, then rerun the same evaluation slice.
Observability Gate
Before production use, confirm:
- Trace IDs connect user request, retrieval, model call, tool call, guardrail, and final output.
- Content logging is minimized and justified.
- Sensitive fields are redacted.
- Cost and latency are visible.
- Error categories are actionable.
- Production incidents can map back to evaluation cases.
Release Decision
Return:
- Baseline versus candidate result.
- Pass, conditional pass, or fail.
- Failed cases and severity.
- Versioned dataset location.
- Residual risks.
- Required fix before release.
1---2name: ai-reliability-evaluation3description: "Design and run AI reliability evaluations across prompts, models, tools, agents, RAG, vector search, observability, cost, latency, safety, security boundaries, and release thresholds. Use before shipping or materially changing AI behavior, retrieval, tool access, model routing, or autonomous workflows."4category: AI, Agents, LLMs, Data5license: MIT6metadata:7 source: https://skillmd.com/skills/newmindsgroup/ai-reliability-evaluation8 version: '1.0.0'9 author: New Minds Group10 sources: Original New Minds Group AI reliability workflow created from evaluation and observability capability analysis; no third-party skill text copied11---1213# AI Reliability Evaluation1415Version-Timestamp: 2026-08-04 13:10:00 AST1617Use this skill to replace subjective AI approval with measured behavior, clear thresholds, and controlled release decisions.1819## Scope2021Evaluate any combination of:2223- Prompt behavior.24- Model choice and routing.25- Tool permissions and tool outputs.26- Agent planning and handoffs.27- RAG ingestion, chunking, embeddings, retrieval, reranking, citations, and freshness.28- Vector database design, metadata filters, updates, deletions, recall, latency, and tenant boundaries.29- Observability for quality, latency, cost, tokens, errors, traces, feedback, and incidents.3031## Safety Rules3233- Treat sending prompts, documents, traces, or evaluation rows to an external provider as data publication.34- Use synthetic, public, anonymized, or explicitly approved data.35- Do not test prompt injection, data extraction, tool misuse, privilege escalation, or denial-of-service behavior outside authorized scope.36- Do not rely on evaluator-model scores alone for high-impact safety, legal, financial, medical, or compliance claims.37- Never log secrets, private user content, raw credentials, or unnecessary personal data.3839## Evaluation Design4041Define:4243- Task and user goal.44- Model, prompt, tool, retrieval, and policy versions.45- Success criteria and release thresholds.46- Representative cases.47- Edge cases.48- Refusal and safety cases.49- Unanswerable cases.50- Cross-tenant and access-boundary cases.51- Expected evidence or rubric.52- Human-review sampling plan.5354## Metrics5556Choose metrics that fit the system:5758- Task success.59- Format compliance.60- Groundedness.61- Citation support.62- Retrieval precision and recall.63- Context relevance.64- Refusal correctness.65- Tool-call correctness.66- Latency.67- Cost.68- Consistency.69- Failure severity.7071## Failure Triage7273Classify failures by layer:7475- Data source.76- Ingestion.77- Chunking.78- Embeddings.79- Vector search.80- Reranking.81- Prompt.82- Model.83- Tool contract.84- Authorization policy.85- User interface.86- Observability gap.8788Fix the layer that caused the failure, then rerun the same evaluation slice.8990## Observability Gate9192Before production use, confirm:9394- Trace IDs connect user request, retrieval, model call, tool call, guardrail, and final output.95- Content logging is minimized and justified.96- Sensitive fields are redacted.97- Cost and latency are visible.98- Error categories are actionable.99- Production incidents can map back to evaluation cases.100101## Release Decision102103Return:104105- Baseline versus candidate result.106- Pass, conditional pass, or fail.107- Failed cases and severity.108- Versioned dataset location.109- Residual risks.110- Required fix before release.