Annotate and review traces
Automated metrics are downstream of one thing: a human deciding what "good" means. For specialized or high-stakes domains (finance, health, legal), and for early-stage apps, structured human review of real traces is the single highest-ROI activity. It produces the golden labels every other eval depends on, and it surfaces failure modes you did not know to look for.
Set up the review loop
- Pull a sample of traces to review. Stratify (by topic, difficulty, low online-eval score, thumbs-down) so reviewers see the interesting cases, not 100 easy ones.
- Give reviewers the full context the model had: input, retrieved docs, tool results, output. Redact PII first for regulated data (
redact-pii-for-tracing).
- Use a simple, consistent schema: pass/fail (or a small rubric score) + a free-text failure reason + a category tag. The free-text is where you discover new failure modes; the categories let you count them.
- Use the tooling rather than spreadsheets where possible: annotation queues exist in Langfuse, Phoenix, Opik, LangSmith and let annotations attach to the trace.
Do error analysis (not just labeling)
The point is not a score, it is understanding. After a review pass:
- Read the free-text reasons and cluster them into failure categories (retrieval miss, hallucination, formatting, refusal, tone, ...). Count each.
- Fix the biggest category first. A few categories usually explain most failures.
- This is the loop that turns "the app is kind of bad" into "34% of failures are retrieval misses, here is the fix."
Turn reviews into durable assets
- Reviewed pass/fail labels become your golden eval dataset (
build-eval-dataset) and regression cases (trace-based-testing).
- Reviewed labels also calibrate your LLM-as-judge: check the automated judge agrees with the humans; fix the rubric until it does (
add-llm-evals). Then the judge can scale what humans validated.
Verify
- Reviewers see full context and use a consistent schema.
- Failure reasons are clustered into categories with counts, not just an average score.
- Reviewed items feed the eval set and calibrate the automated judge.
Anti-patterns
- Never looking at your actual data, only at aggregate metrics (you miss the failure modes).
- Reviewing without the retrieved context/tool results (you cannot tell why it failed).
- Pass/fail with no reason text (you get a number, not an insight).
- Labels that never become an eval set or judge calibration (wasted expert time).
Grounding
"Look at your data" and error analysis are the core of practitioner eval methodology: Hamel Husain, Your AI Product Needs Evals; human labels are the ground truth that automated LLM-as-a-judge is calibrated against (Zheng et al. 2023, arXiv:2306.05685).
1---2name: annotate-traces-for-review3description: Use this to set up human review and annotation of LLM traces, so people (often domain experts) can label outputs, do error analysis, and build a trustworthy golden dataset. Trigger on "review my LLM outputs", "have an expert label these", "error analysis", "annotate traces", "build a golden dataset", or when automated evals are not enough for a high-stakes or specialized domain. Looking at your data is the highest-ROI thing you can do.4license: CC0-1.05---67# Annotate and review traces89Automated metrics are downstream of one thing: a human deciding what "good" means. For specialized or high-stakes domains (finance, health, legal), and for early-stage apps, structured human review of real traces is the single highest-ROI activity. It produces the golden labels every other eval depends on, and it surfaces failure modes you did not know to look for.1011## Set up the review loop12131. **Pull a sample of traces** to review. Stratify (by topic, difficulty, low online-eval score, thumbs-down) so reviewers see the interesting cases, not 100 easy ones.142. **Give reviewers the full context** the model had: input, retrieved docs, tool results, output. Redact PII first for regulated data (`redact-pii-for-tracing`).153. **Use a simple, consistent schema:** pass/fail (or a small rubric score) + a **free-text failure reason** + a category tag. The free-text is where you discover new failure modes; the categories let you count them.164. **Use the tooling** rather than spreadsheets where possible: annotation queues exist in Langfuse, Phoenix, Opik, LangSmith and let annotations attach to the trace.1718## Do error analysis (not just labeling)1920The point is not a score, it is understanding. After a review pass:21- **Read the free-text reasons and cluster them** into failure categories (retrieval miss, hallucination, formatting, refusal, tone, ...). Count each.22- **Fix the biggest category first.** A few categories usually explain most failures.23- This is the loop that turns "the app is kind of bad" into "34% of failures are retrieval misses, here is the fix."2425## Turn reviews into durable assets2627- Reviewed pass/fail labels become your **golden eval dataset** (`build-eval-dataset`) and regression cases (`trace-based-testing`).28- Reviewed labels also **calibrate your LLM-as-judge**: check the automated judge agrees with the humans; fix the rubric until it does (`add-llm-evals`). Then the judge can scale what humans validated.2930## Verify3132- Reviewers see full context and use a consistent schema.33- Failure reasons are clustered into categories with counts, not just an average score.34- Reviewed items feed the eval set and calibrate the automated judge.3536## Anti-patterns3738- Never looking at your actual data, only at aggregate metrics (you miss the failure modes).39- Reviewing without the retrieved context/tool results (you cannot tell why it failed).40- Pass/fail with no reason text (you get a number, not an insight).41- Labels that never become an eval set or judge calibration (wasted expert time).4243## Grounding4445"Look at your data" and error analysis are the core of practitioner eval methodology: Hamel Husain, [*Your AI Product Needs Evals*](https://hamel.dev/blog/posts/evals/); human labels are the ground truth that automated LLM-as-a-judge is calibrated against (Zheng et al. 2023, [arXiv:2306.05685](https://arxiv.org/abs/2306.05685)).
ContextJet-ai