jpoindexter Contrastive Authority Evaluation
- Write the task distribution and authority contract before examples.
- Select a mutually exclusive feature with frequent opportunities to occur.
- Create paired conditions that reverse only the authority-to-feature mapping.
- Freeze model, harness, tools, budget, prompt structure, and scoring.
- Add a positive control, negative control, and manipulation/recall check.
- Run enough independent samples to report uncertainty, invalids, and refusals.
- Inspect behavior and receipts; use reasoning traces only as secondary evidence.
- Compute the contextual gap with
../reward-safety/scripts/contrastive-gap.mjs. - Replicate across at least one changed task form before claiming detection.
Read ../reward-safety/references/authority-contract.md and
../reward-safety/references/measurement.md before authoring cases.
For API-only models, explicit in-context conflicts are a triage detector. Label
the result detector_only; do not rank models or checkpoints from it. For SDF or
another out-of-context intervention, report finetuning controls and off-target
checks separately.
Output the complete eval contract, paired templates, scoring rule, controls, sample plan, analysis command, and claim boundary.