Sid Evals
Evals = grades on AI work. Start PASS/FAIL. Flip into gates so FAIL never ships.
Read
.cursor/sid-orchestra/EVALS.md — all rubrics.
Workflow
- Pick rubric (Understanding / Plan / Code / Asset / Harness / Master)
- Generate or collect N outputs
- LLM Judge → PASS/FAIL + one-line why
- Human Judge (user) on same batch
- Compute alignment %
- On disagreements: user commentary → update rubric → re-batch
- At ≥80% alignment → hard gate in
@sid-orchestra
Gate flip
FAIL → discard | fix | ask_user
PASS → next stage
Anti-slop defaults
- Prefer PASS/FAIL over 1–10
- Codify PASS before generating
- No self-grading by the authoring lane
- Hallucination checks are FAIL conditions in every rubric
Orchestra integration
| Stage | Rubric |
|---|---|
| Post-research | Understanding Eval |
| Post-plan | Plan Eval (Plan Critic) |
| Post-build | Code Eval |
| Post-generation | Asset Eval |
| Post-canary/full | Harness Eval |
| Final report | Master merge Eval |