Tier Evaluator

Tier Evaluator

The evaluator is SKEPTICAL by default. It runs tests, checks criteria, and reports PASS or FAIL. A 7/10 means genuinely good work. A 5/10 means real problems that would embarrass a senior engineer.

Grading Dimensions

Completeness (threshold: 7/10)

Does the output fully address the spec? Are all acceptance criteria met?

• 9–10 — All requirements met, edge cases handled, nothing stubbed or TODOed
• 7–8 — Core requirements met, minor gaps, no critical paths missing
• 5–6 — Most requirements met, some incomplete paths or skipped edge cases
• 3–4 — Significant gaps, key requirements not addressed
• 1–2 — Skeleton or stub, most requirements unaddressed

Correctness (threshold: 7/10)

Is the code logically sound? Will it work under real conditions?

• 9–10 — No bugs, all error paths handled, input validated, no security holes
• 7–8 — Minor issues only, no crashes under normal use, no security concerns
• 5–6 — Some unhandled exceptions, missing validation, potential race conditions
• 3–4 — Known bugs, type errors, crashes under certain inputs, security concerns
• 1–2 — Fundamental logic errors, will crash under normal use

Code Quality (threshold: 6/10)

Is the code clean, maintainable, and following conventions?

• 9–10 — All functions <40 lines, all files <300 lines, no magic numbers, structured logging, full type hints, DRY, SOLID
• 7–8 — Mostly clean, minor violations, logging in place, most types annotated
• 5–6 — Some functions >80 lines, print() statements, missing type hints, magic numbers
• 3–4 — Many large functions, mixed responsibilities, inconsistent patterns, bare except clauses
• 1–2 — Spaghetti code, no structure, hardcoded values throughout, no error handling

Format Compliance (threshold: 8/10)

Does the output follow the project's conventions and the request format?

• 9–10 — Perfect format adherence, naming conventions followed, no deviations
• 7–8 — Minor formatting issues, overall compliant
• 5–6 — Some convention violations, structure mostly correct
• 3–4 — Multiple violations, inconsistent naming, wrong format in places
• 1–2 — Does not follow requested format at all

Evaluation Protocol

1. Run the code mentally — trace the happy path, then the failure paths
2. Check every spec requirement — tick off or mark as missing
3. Look for hardcoded values, print statements, and TODO comments
4. Check type annotations on all public functions
5. Verify error handling: is every exception caught at the right level?
6. Score each dimension independently — do not average first impressions
7. If ANY dimension is below threshold: FAIL. Report exactly what must change.
8. Critique format: "Dimension X scored Y/10. Issue: [specific problem at file:line]. Fix: [specific action]."

Gate Decision

• All dimensions at or above threshold → PASS → proceed to ship
• Any dimension below threshold → FAIL → inject critique into retry prompt
• After 2 failed retries → escalate to Opus tier for one final attempt
• After 3 total failures → surface to human with full failure log