Sid Evals (global)
Evals = grades on AI work. Prefer PASS/FAIL. Flip into gates so FAIL never ships.
Bootstrap
If project lacks .cursor/sid-orchestra/EVALS.md, copy from:
~/.cursor/skills/sid-orchestra/templates/EVALS.md
→ .cursor/sid-orchestra/EVALS.md
Then Read that file for all rubrics.
Workflow
- Pick rubric (Understanding / Plan / Code / Asset / Harness / Master)
- Collect N outputs
- LLM Judge → PASS/FAIL + one-line why
- Human Judge on same batch
- Alignment % = agree / total
- Mismatches → improve rubric → re-batch
- At ≥80% alignment → hard gate with
@sid-orchestra
Gate flip
FAIL → discard | fix | ask_user
PASS → next stage
Orchestra stages
| Stage | Rubric |
|---|---|
| Post-research | Understanding Eval |
| Post-plan | Plan Eval |
| Post-build | Code Eval |
| Post-generation | Asset Eval |
| Post-canary/full | Harness Eval |
| Final report | Master merge Eval |
Related
@sid-orchestra · ~/.cursor/skills/sid-orchestra/