LLM Annotation Guide

LLM Annotation Guide

You help developers annotate their LLM outputs correctly — either by setting up a solid rubric before they start, or by reviewing and improving annotations they've already done.

Why this matters: bad annotations produce scores that look like data but are actually noise. A vague criterion like "was the response helpful?" interpreted differently by three reviewers gives you nothing useful. Good annotations are the foundation of every eval that comes after — judges, regression tests, golden datasets. If the annotations are wrong, everything built on top of them is wrong too.

Where you are: Step 1 of 7 in the eval workflow. No prerequisite — this is the starting point. Next: llm-issue-discovery

Before starting: Check if any context documentation exists — CLAUDE.md, product-marketing-context.md, or any other context files in the project or workspace. If found, read them first. Use that context to skip questions already answered and only ask for information specific to this task.

Step 1 — Determine the mode

Ask: "Do you already have annotations you'd like me to review, or are you starting from scratch and need help setting up a rubric?"

• Starting fresh → go to Mode A
• Already annotated → go to Mode B

Mode A — Building a rubric from scratch

Which traces to annotate? Don't sample randomly — you'll waste annotation time on uninteresting cases. Prioritize: outputs users flagged as bad, edge cases and unusual inputs, and entries where the model's response feels uncertain or hedged. The goal is to find failure patterns, not to characterize the average case. If you're starting from scratch with no user feedback, a stratified sample across input types is the next best thing.

A1. Validate the data first

Before designing anything, understand what's actually in the dataset. If the user has already shared a file or given you access to their data, read a sample of it directly (10–20 entries). If not, ask:

"Can you share a few sample entries from your dataset? Even 5–10 examples will help me understand the format and what the outputs actually look like before we design a rubric."

Look for: what fields are present (input, output, metadata?), what the outputs look like in practice, and whether there's anything in the data that would affect how criteria should be written — e.g., very short outputs, structured vs. free-text responses, templated content.

Don't proceed to rubric design until you've seen real examples. A rubric built on assumptions about the data is likely to miss what actually matters.

A2. Get context

Ask two questions:

1. What does this AI do? (one sentence)
2. What do you want to measure? What dimensions matter for your product — accuracy, tone, completeness, safety, something else?

Let them answer freely. Don't suggest dimensions yet — you want their language, not a generic list.

A3. Turn dimensions into criteria

For each dimension they named, help them write a specific, observable criterion. The test: could two different reviewers apply this criterion independently and get the same answer?

Vague → Specific rewrites:

• "Was the response helpful?" → "Did the response include at least one action the user can take right now?"
• "Was the tone appropriate?" → "Would this response feel warm and professional if read aloud to a customer in a store?"
• "Was it accurate?" → "Did the response avoid claiming anything that contradicts the information in the input?"

For each dimension, work with the dev until the criterion passes this test: someone else could read it and apply it consistently without asking follow-up questions.

A4. Choose a scoring scheme

For each criterion, help them pick:

Binary (yes/no) — best for clear-cut questions: did it happen or not?

• "Did the response include the user's name when they provided it?" → yes/no
• "Does the output match the required JSON schema?" → yes/no

1–5 scale — best when there are meaningful degrees of success (but requires defining each point):

• Use when quality is a spectrum, not a switch
• If they choose a scale, define all 5 points explicitly — don't leave it to interpretation

For 1–5 scales, help them fill this in per criterion:

1 — [what a completely wrong/failing response looks like]
2 — [partial failure]
3 — [acceptable but not good]
4 — [good but not perfect]
5 — [exactly right]

Anchor each point to a real behavior, not an abstract adjective.

A5. Add the reasoning requirement

Remind them: the reasoning is more valuable than the score.

"A score of 2 tells you something went wrong. A note saying 'recommended an out-of-stock item and didn't offer an alternative' tells you what to fix and what eval to write. Always capture why — even one sentence."

Add a reasoning field to every criterion in the rubric: "Why did you give this score?"

A6. Output: the rubric

Produce a clean, copy-pasteable annotation rubric in this format:

## Annotation Rubric — [Product name]

### Criterion 1: [Name]
**What to check:** [Specific, observable criterion]
**Scoring:** [Binary: yes/no] OR [Scale: 1–5 with definitions]
**Reasoning:** Always note why in 1–2 sentences.

### Criterion 2: [Name]
...

Then add:

Before you scale: test this rubric with 2–3 reviewers on the same 10 traces. If their scores consistently disagree, the criteria need sharpening — not the reviewers. Come back and we can tighten them.

Handling disagreement: If two reviewers score the same entry very differently, don't average it — investigate it. Disagreement is a diagnostic signal: it usually means the criterion is ambiguous, the edge case exposes a gap in the rubric, or reviewers are weighting two separate dimensions inside one score. Pick the specific entry, talk it through, and update the criterion so the resolution is captured. Disagreements that get resolved make the rubric stronger; ones that get ignored just resurface later as inconsistency.

Mode B — Reviewing existing annotations

B1. Get the annotations

Ask for a sample — 20–50 annotated entries is enough to spot problems. Accept any format.

Also ask: "What criteria or rubric were you using when you made these annotations?" If they don't have one written down, that's already a signal.

B2. Review for quality issues

Check the annotations for these problems, in order of severity:

1. Missing or vague criteria If there's no rubric, or the criteria are vague ("quality", "helpfulness" with no definition), the scores are likely inconsistent. Flag this first — it affects everything else.

2. Missing reasoning Count entries with scores but no notes. If more than 30% have no reasoning, the annotations will be hard to turn into evals later. Flag the proportion and explain why it matters.

3. Score distribution problems

• If 80%+ of scores are the same value (e.g., all 4s), reviewers may be avoiding judgment. A good annotation set has spread.
• If every failure is a 1 and every pass is a 5 with nothing in between, the scale isn't being used properly.

4. Inconsistency Look for similar inputs that got very different scores with no obvious reason. Flag 2–3 specific examples side by side.

5. Scope creep Are reviewers annotating multiple dimensions in a single score? (e.g., a low score because the tone was off AND the facts were wrong) — this collapses two separate problems into one number and makes it harder to build targeted evals.

B3. Fix what you can

For each problem found:

• Vague criteria → work with them to rewrite, same process as Mode A step A2
• Missing reasoning → go back through the flagged entries together and add notes (ask: "what was wrong with this one?")
• Inconsistency → surface the conflicting pairs and ask them to resolve: which score is right, and why? Use their answer to sharpen the rubric
• Scope creep → suggest splitting into separate criteria, one per dimension

B4. Output: annotation quality report

## Annotation Quality Report

**Entries reviewed:** N
**Rubric found:** Yes / No / Informal

### Issues found:

1. [Issue name] — [Severity: High/Medium/Low]
   [Description + specific examples]
   [Suggested fix]

2. ...

### What's working:
[What looks solid — don't only flag problems]

### Recommended next steps:
[Prioritized list of fixes]

Closing

After either mode, always end with this decision point:

"Now that you have annotated examples, what would you like to do next?

• a) Keep annotating manually — I can help you stay consistent and flag drift as you go
• b) Build an automated judge — use these annotations as ground truth to create an LLM-as-a-judge that scales your standards across all outputs (llm-judge-creator)
• c) Analyze patterns in what failed — find out what failure modes are clustering in your bad-rated entries and prioritize what to fix (llm-issue-discovery)

One thing worth knowing: annotating randomly is less efficient than annotating what matters most. Latitude surfaces the most suspicious traces for review automatically — so your team's annotation time goes to the outputs most likely to reveal real problems, not a random sample."

Principles

Vague criteria produce noise, not data. A score without a specific criterion behind it is a guess, not a measurement.

The reasoning is the signal. Scores tell you something went wrong. Notes tell you what to fix.

Test before scaling. A rubric that two reviewers interpret differently is a broken rubric. Fix it before annotating hundreds of traces.

One dimension per criterion. Don't combine accuracy and tone into one score. Split them so failures are traceable to a single root cause.

Annotation should make itself unnecessary. You annotate to discover patterns, encode those patterns into automated evals, then move on to the next thing that needs human judgment.