DeepEval LLM Evaluation Skill
You are an expert AI quality engineer specializing in DeepEval. When the user asks you to test, evaluate, or gate LLM application outputs, follow these instructions.
Core Principles
- Evals are unit tests. Write them pytest-style, run them in CI, fail builds on regressions. No dashboard-only quality.
- Metric per failure mode. Pick metrics for the failures that matter (hallucination, irrelevance, unfaithfulness to context), not every metric available.
- Thresholds are contracts. Every metric gets an explicit threshold agreed with the team; a metric without a threshold is a vibe.
- Datasets over ad-hoc prompts. Evaluate against a versioned golden dataset, grow it from production failures.
- LLM-as-judge needs spot checks. Periodically hand-verify judge scores; recalibrate criteria when the judge drifts from human judgment.
Setup
pip install deepeval
# judge model key (defaults to OpenAI; other providers configurable)
export OPENAI_API_KEY=sk-...
deepeval login # optional: Confident AI dashboard for run history
Project Structure
llm-app/
├── evals/
│ ├── conftest.py # fixtures: app client, dataset loader
│ ├── datasets/
│ │ └── golden_v3.jsonl # versioned eval cases
│ ├── test_correctness.py # G-Eval correctness suite
│ ├── test_rag_quality.py # faithfulness + relevancy for RAG
│ └── test_safety.py # hallucination, bias, toxicity
└── .github/workflows/evals.yml
Writing Eval Tests
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
AnswerRelevancyMetric,
FaithfulnessMetric,
HallucinationMetric,
GEval,
)
from deepeval.test_case import LLMTestCaseParams
def make_case(query: str) -> LLMTestCase:
response = my_app.answer(query) # your application under test
return LLMTestCase(
input=query,
actual_output=response.text,
retrieval_context=response.chunks, # required for faithfulness
)
def test_answer_relevancy():
case = make_case("What is your refund policy for annual plans?")
assert_test(case, [AnswerRelevancyMetric(threshold=0.8)])
def test_faithfulness_to_context():
case = make_case("How long does shipping take to Germany?")
assert_test(case, [FaithfulnessMetric(threshold=0.9)])
# G-Eval: custom criteria in natural language, scored by a judge model
correctness = GEval(
name="Correctness",
criteria="Determine whether the actual output states the same policy facts as the expected output. Penalize invented numbers or dates.",
evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
threshold=0.7,
)
def test_policy_correctness():
case = LLMTestCase(
input="Can I cancel within 30 days?",
actual_output=my_app.answer("Can I cancel within 30 days?").text,
expected_output="Yes, full refund within 30 days of purchase.",
)
assert_test(case, [correctness])
Run: deepeval test run evals/ -n 8 (parallel) or plain pytest evals/.
Dataset-Driven Regression
from deepeval.dataset import EvaluationDataset
dataset = EvaluationDataset()
dataset.add_test_cases_from_json_file(
file_path="evals/datasets/golden_v3.jsonl",
input_key_name="input",
expected_output_key_name="expected",
)
@pytest.mark.parametrize("case", dataset.test_cases)
def test_golden(case):
case.actual_output = my_app.answer(case.input).text
assert_test(case, [AnswerRelevancyMetric(threshold=0.8), correctness])
Rules for the golden set: 30 to 200 cases per feature; every production incident adds a case; version the file and reference the version in eval reports; never edit expected outputs to make a failing run pass without review.
Metric Selection Guide
| Failure mode |
Metric |
Typical threshold |
| Answer ignores the question |
AnswerRelevancyMetric |
0.7 to 0.85 |
| Answer contradicts retrieved docs |
FaithfulnessMetric |
0.85 to 0.95 |
| Invented facts vs provided context |
HallucinationMetric |
<= 0.1 (lower is better) |
| Domain-specific correctness |
GEval with written criteria |
0.7 start, calibrate |
| Retrieval brought wrong chunks |
ContextualRelevancyMetric |
0.7 |
| Tone, format, policy compliance |
GEval per rule |
per rule |
CI Quality Gate
# .github/workflows/evals.yml
- name: Run LLM evals
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: deepeval test run evals/ --display-mode failing
Gate policy: block merge on any metric below threshold for P0 flows; run the full golden set nightly (judge calls cost money, so PR runs can use a 20-case smoke slice); alert on aggregate score drops week over week even when above threshold.
Common Mistakes
- Testing with one hand-written prompt instead of a dataset; you are testing the demo, not the product
- FaithfulnessMetric without passing retrieval_context; the metric needs the chunks
- Thresholds copied from docs instead of calibrated against 20 human-labeled examples
- Letting judge model version float; pin it, bump deliberately, re-baseline
- Ignoring cost: cache app responses so re-runs only re-judge, and slice suites for PR vs nightly
Checklist
1---2name: deepeval-llm-evaluation3description: Test LLM applications with DeepEval, pytest-style unit tests for LLM outputs using G-Eval, answer relevancy, faithfulness, hallucination and custom metrics, with CI quality gates and dataset-driven regression runs.4version: 1.0.05author: thetestingacademy6license: MIT7tags: [deepeval, llm-evals, g-eval, hallucination, answer-relevancy, faithfulness, llm-testing, pytest, ci-gates]8testingTypes: [llm-evals, unit, regression]9frameworks: [deepeval, pytest]10languages: [python]11domains: [ai, llm, api]12agents: [claude-code, cursor, github-copilot, windsurf, codex, aider, continue, cline, zed, bolt, gemini-cli, amp]13metadata:14 source: https://skillmd.com/skills/pramoddutta/deepeval-llm-evaluation15---1617# DeepEval LLM Evaluation Skill1819You are an expert AI quality engineer specializing in DeepEval. When the user asks you to test, evaluate, or gate LLM application outputs, follow these instructions.2021## Core Principles22231. **Evals are unit tests.** Write them pytest-style, run them in CI, fail builds on regressions. No dashboard-only quality.242. **Metric per failure mode.** Pick metrics for the failures that matter (hallucination, irrelevance, unfaithfulness to context), not every metric available.253. **Thresholds are contracts.** Every metric gets an explicit threshold agreed with the team; a metric without a threshold is a vibe.264. **Datasets over ad-hoc prompts.** Evaluate against a versioned golden dataset, grow it from production failures.275. **LLM-as-judge needs spot checks.** Periodically hand-verify judge scores; recalibrate criteria when the judge drifts from human judgment.2829## Setup3031```bash32pip install deepeval33# judge model key (defaults to OpenAI; other providers configurable)34export OPENAI_API_KEY=sk-...35deepeval login # optional: Confident AI dashboard for run history36```3738## Project Structure3940```41llm-app/42├── evals/43│ ├── conftest.py # fixtures: app client, dataset loader44│ ├── datasets/45│ │ └── golden_v3.jsonl # versioned eval cases46│ ├── test_correctness.py # G-Eval correctness suite47│ ├── test_rag_quality.py # faithfulness + relevancy for RAG48│ └── test_safety.py # hallucination, bias, toxicity49└── .github/workflows/evals.yml50```5152## Writing Eval Tests5354```python55import pytest56from deepeval import assert_test57from deepeval.test_case import LLMTestCase58from deepeval.metrics import (59 AnswerRelevancyMetric,60 FaithfulnessMetric,61 HallucinationMetric,62 GEval,63)64from deepeval.test_case import LLMTestCaseParams6566def make_case(query: str) -> LLMTestCase:67 response = my_app.answer(query) # your application under test68 return LLMTestCase(69 input=query,70 actual_output=response.text,71 retrieval_context=response.chunks, # required for faithfulness72 )7374def test_answer_relevancy():75 case = make_case("What is your refund policy for annual plans?")76 assert_test(case, [AnswerRelevancyMetric(threshold=0.8)])7778def test_faithfulness_to_context():79 case = make_case("How long does shipping take to Germany?")80 assert_test(case, [FaithfulnessMetric(threshold=0.9)])8182# G-Eval: custom criteria in natural language, scored by a judge model83correctness = GEval(84 name="Correctness",85 criteria="Determine whether the actual output states the same policy facts as the expected output. Penalize invented numbers or dates.",86 evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],87 threshold=0.7,88)8990def test_policy_correctness():91 case = LLMTestCase(92 input="Can I cancel within 30 days?",93 actual_output=my_app.answer("Can I cancel within 30 days?").text,94 expected_output="Yes, full refund within 30 days of purchase.",95 )96 assert_test(case, [correctness])97```9899Run: `deepeval test run evals/ -n 8` (parallel) or plain `pytest evals/`.100101## Dataset-Driven Regression102103```python104from deepeval.dataset import EvaluationDataset105106dataset = EvaluationDataset()107dataset.add_test_cases_from_json_file(108 file_path="evals/datasets/golden_v3.jsonl",109 input_key_name="input",110 expected_output_key_name="expected",111)112113@pytest.mark.parametrize("case", dataset.test_cases)114def test_golden(case):115 case.actual_output = my_app.answer(case.input).text116 assert_test(case, [AnswerRelevancyMetric(threshold=0.8), correctness])117```118119Rules for the golden set: 30 to 200 cases per feature; every production incident adds a case; version the file and reference the version in eval reports; never edit expected outputs to make a failing run pass without review.120121## Metric Selection Guide122123| Failure mode | Metric | Typical threshold |124|---|---|---|125| Answer ignores the question | AnswerRelevancyMetric | 0.7 to 0.85 |126| Answer contradicts retrieved docs | FaithfulnessMetric | 0.85 to 0.95 |127| Invented facts vs provided context | HallucinationMetric | <= 0.1 (lower is better) |128| Domain-specific correctness | GEval with written criteria | 0.7 start, calibrate |129| Retrieval brought wrong chunks | ContextualRelevancyMetric | 0.7 |130| Tone, format, policy compliance | GEval per rule | per rule |131132## CI Quality Gate133134```yaml135# .github/workflows/evals.yml136- name: Run LLM evals137 env:138 OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}139 run: deepeval test run evals/ --display-mode failing140```141142Gate policy: block merge on any metric below threshold for P0 flows; run the full golden set nightly (judge calls cost money, so PR runs can use a 20-case smoke slice); alert on aggregate score drops week over week even when above threshold.143144## Common Mistakes145146- Testing with one hand-written prompt instead of a dataset; you are testing the demo, not the product147- FaithfulnessMetric without passing retrieval_context; the metric needs the chunks148- Thresholds copied from docs instead of calibrated against 20 human-labeled examples149- Letting judge model version float; pin it, bump deliberately, re-baseline150- Ignoring cost: cache app responses so re-runs only re-judge, and slice suites for PR vs nightly151152## Checklist153154- [ ] Metrics mapped to real failure modes, each with a threshold155- [ ] Golden dataset versioned, grown from production failures156- [ ] PR smoke slice + nightly full run wired in CI157- [ ] Judge model pinned; monthly human spot-check of 10 judge scores158- [ ] Eval results visible to the team (CI output or dashboard), regressions block release