Evaluating Bitrouter Routes

Evaluate BitRouter Routes

Evaluate outcomes outside BitRouter's serving path. Produce an immutable result and stop after BitRouter reports its admission status. Do not run the optimizer or use low-level policy publication from the evaluator workflow.

Read the Eval Exchange reference before forming a subject or result. It is the exact current wire and authority contract.

Classify the evaluation

Choose scope from the observable outcome boundary:

Choose evaluator.kind from the actual source:

Build the evaluator packet

1. Copy every decision's decision_id, policy, route_projection, request_key, selected_tier, baseline_tier, policy_digest, and optional experiment object from router-authored evidence. Preserve the experiment object verbatim; never invent or edit its id, arm, assignment unit, assignment-id digest, or challenger propensity.

2. Redact evidence before it leaves its private source. Retain raw messages, tool arguments, code, and evaluator output with the evaluator; place safe, content-addressed evidence items in the subject.

3. List only dimensions the evaluator was asked to judge. Leave unsupported dimensions absent. Use inconclusive when evidence cannot support a verdict.

4. Set confidence_ppm to the evaluator's confidence that its verdict is correct. Use null when the evaluator or rubric does not supply confidence. For a task or episode cost result, submit the complete unit cost as cost.usd_micros with unit micro_usd; never substitute one request's price for the complete task or episode.

5. Write a draft subject with an empty evidence_digest, then seal it:

   bro eval subject seal subject-draft.json --output subject.json
   

6. For a multi-decision subject, derive decision_credit from the fixed evaluator credit policy:

   • Exact supported decision/metric mappings: emit only those mappings.
   • No policy or no exact mapping: use {} or omit the serde-defaulted field. The result remains a record but produces no per-route evidence. For a one-decision subject, empty credit means implicit full credit. When an inconclusive evaluator intentionally withholds attribution, emit that decision with weight_ppm: 0 instead. Keep hypothetical or illustrative weights outside submit-ready JSON.

Submit and hand off

1. Insert the sealed subject and submit a result that repeats its exact eval_id and evidence_digest.

   bro eval subject put subject.json --config bitrouter.yaml
   bro eval result submit result.json --config bitrouter.yaml
   

2. Treat an admitted response as eligible evidence. Preserve held_out, rejected, and disputed responses as non-training records.

3. Hand off the sealed subject, result, submission response, and private evidence references, then stop. A later bro optimize run invocation is a separate autonomous authorization; do not review, publish, or run it as part of evaluation.

Keep the packet consistent

• Use subject seal for canonical evidence hashing and JSON.
• Attribute metrics only to evidence-supported decisions.
• Never copy a task- or episode-level reward onto each request. Use a fixed causal policy (for example, a matched control plus one changed route family) or withhold credit.
• Preserve the router-authored baseline and selected tiers.
• Preserve router-authored experiment references exactly. Optimizer membership never comes from the evaluator-owned cohort string.
• Treat inconclusive as zero quality evidence even if an old or malformed packet assigns positive quality credit. Attribute cost or latency separately.
• Keep evaluator identity, rubric/config digest, evidence references, confidence, and idempotency key stable for an equivalent retry.
• Keep eval_id, result, and evidence identities attempt-specific. For task scope, keep subject_id stable for the canonical task inside its explicit run/source/policy namespace so repeated attempts cannot inflate the generic compiler's independent-task count.