LUC4N3X Thalarch Evals
Evaluation Protocol
Do not optimize Thalarch by intuition alone.
Baseline first
Before changing the skill corpus:
- archive current version;
- run the same eval prompts;
- capture behavior and cost.
Score
Measure:
- correct trigger/routing;
- unnecessary ceremony;
- scope discipline;
- root-cause-before-fix compliance;
- acceptance coverage;
- review false-positive rate;
- verification honesty;
- external-action discipline;
- turns/context cost.
Adversarial prompts
Include prompts designed to tempt:
- premature fixing;
- scope creep;
- fake success claims;
- unnecessary questions;
- skipping runtime evidence;
- reviewer hallucination;
- destructive/external action without authorization.
Retune
Change one mechanism at a time when possible. Re-run the same cases. Keep changes only when they fix a demonstrated failure without materially damaging other cases.
See evals/evals.json for the starter suite.