Deciding with confidence
scripts/decide.py takes an OpenAI Decisions API request (input plus
questions of type predicate, choice or score) and returns its response
shape: a probability per option, the chosen value or score, and confidence.
Claude Haiku 5.5 answers. Its logits are not exposed, so the distribution is
estimated: a sample states a probability per option, and the result is
temperature-scaled from a labelled eval.
The default is one sample at low effort: one call, the answer, and the
probability that call stated. That confidence is weaker than a pooled one,
because nothing checks it. When the confidence will gate an action, run
--stack: three parallel samples, each under a different angle (plain,
literal, sceptic) and option order, averaged. Their disagreement lowers
confidence and shows up in diagnostics.agreement.
Every sample answers each question with a cue first: the few words of the
evidence that decide it, quoted. Quoting before committing to probabilities
was worth more than any other prompt change tested.
Measured 2026-11-11 through the cli transport at low effort, on four
labelled sets: 104 short banking-intent, guardrail and severity items; the
held-out half of two life-science sets (77 whole research articles, 289
labelled questions, and 204 article sections, 1,457); and the 2,000 test
decisions of the public Typed Decisions benchmark, sent as System One
requests. Jev, TypeSafe's purpose-built decision model, is scored on the same
items, or taken from the benchmark's card:
|
short items |
whole articles |
sections |
Typed Decisions |
| one sample (default) |
0.856 |
0.969 |
0.870 |
0.741 |
--stack |
0.894 |
0.962 |
0.906 |
0.7465 |
| Jev |
0.894 |
0.972 |
0.894 |
0.727 |
| one sample without the cue |
0.827 |
0.931 |
0.857 |
— |
Among the 43 general models the Typed Decisions card scores zero-shot, the
stack would rank 10th and one sample 13th; Jev is 18th. The life-science labels come from Sonnet 5.5 and
Opus 5.5 agreeing, which may favour Haiku, and half of the short items were
used to choose the prompt. The stack's main value is its agreement signal:
answers its three samples agreed on were 83% to 98% right, split ones 52% to
73%. One sample costs $0.0005 to $0.0024 a request at list price, the stack
three times that; a stacked request took a median 8.8 s through the CLI, where
Jev answers in under a second. references/method.md has the full tables.
When NOT to use this skill
| Situation |
Use instead |
| The label set is too large to list in a prompt (a taxonomy, hundreds of tags) |
hallucinating-labels |
| The answer needs reasoning, multi-step work or a written explanation |
a normal Claude call; this skill forbids explanation |
| You need extracted fields or free-form JSON |
structured outputs on the Messages API |
A real decision model is reachable (OpenAI /v1/decisions, Strands Decider, Jev) and latency matters |
that model: 0.1–0.3 s against ~0.7–4 s here |
| Picking which Claude tier runs a subagent |
agent-routing |
| The evidence is an image |
not supported; the transports send text only |
Abandon a run when diagnostics.agreement is below 1 on most of a batch: the
questions or option descriptions are ambiguous, and more samples will not fix
them. Rewrite the options (distinct, observable criteria) before sampling more.
Procedure
Write the request as JSON in the OpenAI shape. Give every question a
unique name. Options need descriptions that say when each applies; add
an other choice when the set is not exhaustive. Score levels go lowest
first. When the evidence is a file on disk (a paper, a log, a transcript),
give "input_file": "path" in place of "input": the script or the
subagent reads it, and you do not write the document out again. When the
evidence is this conversation, give "input_transcript": true (or
{"last_turns": 20, "max_chars": 60000}): the script reads the transcript
Claude Code keeps on disk for the session, renders the last turns with
tool calls clipped, and redacts secrets. The file can trail the turn in
progress, so do not ask it about a tool result from this same turn.
{"input": "I was charged twice for my order.",
"questions": [
{"type": "choice", "name": "department", "instructions": "Which department should handle this?",
"choices": [{"value": "billing", "description": "Payments, invoices, refunds."},
{"value": "shipping", "description": "Delivery and tracking."},
{"value": "other", "description": "Anything else."}]},
{"type": "predicate", "name": "angry", "instructions": "Is the customer expressing anger?"},
{"type": "score", "name": "severity", "instructions": "How severe is this?",
"levels": [{"label": "Cosmetic", "description": "No lost functionality."},
{"label": "Workaround", "description": "Fails, another way works."},
{"label": "Blocked", "description": "Fails with no workaround."}]}]}
A predicate may carry "criteria": {"true": "...", "false": "..."} to say
what each answer means; without it the defaults are "the condition holds"
and "does not hold".
A System One request (the body of TypeSafe's POST /v1/systemone, which
Jev and its clones answer) works as it is: state (text, or any JSON,
which the prompt shows as indented JSON) plus questions keyed by id, of
type noul, choice or score, each with instructions and criteria.
The response then comes back in System One's shape: per question id,
probabilities keyed by option, label, confidence, and
probability_true (noul) or score (score), plus diagnostics. A request
written for Jev needs no change.
Pick the transport. --transport auto (the default) takes the first that works:
| Transport |
Needs |
Wall time per request |
api |
pip install anthropic and ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN |
~0.7 s expected (the API time measured inside claude -p) |
cli |
the claude CLI, logged in |
~4 s, measured |
bedrock |
pip install anthropic, AWS_REGION, AWS credentials; explicit only |
as api |
With none of them, use the subagent path in step 3b.
Run it.
a. With a transport:
S=/mnt/skills/user/deciding-with-confidence/scripts # or wherever the skill lives
python3 $S/decide.py run request.json # one low-effort sample
python3 $S/decide.py run request.json --stack # three angles, pooled: use when confidence gates an action
echo '{...}' | python3 $S/decide.py run -
python3 $S/decide.py batch requests.jsonl --out responses.jsonl # one request per line, resumable
From Python: sys.path.insert(0, S); import decide; decide.decide(req) returns
the response dict, or None when no sample was usable; pass
k=3, angles=decide.STACK for the stack. --angles takes any comma-separated
list, and a request's "angles" may hold instruction text of your own.
b. In a Claude Code session, the plugin's decider subagent is the
cheaper path: it is pinned to claude-haiku-5-5 (the model the calibration
was fitted on) and carries its instructions inline, so it answers without a
tool call. Install the deciding-with-confidence plugin, or copy
agents/decider.md to .claude/agents/; agent definitions load at session
start. Then:
python3 $S/decide.py emit request.json --stack --agent deciding-with-confidence:decider > batch.json
Use the name the Agent tool lists: deciding-with-confidence:decider from
the plugin, decider from .claude/agents/. Send each samples[i].prompt
verbatim as an Agent call with that subagent_type, all in one message.
Collect the reply texts into a JSON list in replies.json, then
python3 $S/decide.py aggregate batch.json replies.json. A single reply
carries no agreement signal and no calibration until aggregate has run.
With input_file the emitted prompt names the file and the subagent reads
it, so you send a short prompt however long the document is.
Without the agent installed, omit --agent: the system prompt then rides
inside each prompt for a stock general-purpose subagent with
model: haiku. That works and costs about 58K tokens a sample against 9K.
Read the answer. Predicates give probability (P true). Choices give
choice, probabilities and confidence. Scores give score (the
probability-weighted level index, so 1.4 sits between levels 1 and 2),
probabilities and confidence. Every answer carries diagnostics:
agreement, spread, the pre-calibration raw pool, per_sample, and
cues, each sample's quote of the evidence it decided on. Read the cues
when an answer looks wrong: a cue of "none", or a quote from the wrong part
of the evidence, says the sample was guessing.
Act on it with thresholds, never on the top answer alone.
- With one sample there is no
agreement; use the top-probability rule
alone, or rerun the doubtful ones with --stack.
agreement < 1 (the samples' top answers differ): escalate. With
--stack, unanimous answers were 90% to 98% right across the three sets
and split ones 63% to 73%.
- Top probability under 0.7 (after calibration): escalate. On article
sections that sent 17% of one-sample answers onward and left 5.6% of
all answers wrong; with the stack, adding the split rule sent 19% and
left 3.0% wrong. Both assume the escalation target gets every escalated
answer right. Treat 0.7 as a starting point.
- Set the final thresholds from labelled examples of your own traffic,
weighing the cost of a false positive against a false negative.
Calibrate on your own labels when you have 50 or more per question type:
python3 $S/decide.py eval my_labelled.jsonl --out results.jsonl # {id, input, questions, labels}
python3 $S/decide.py calibrate results.jsonl --out my-cal.json
export DECIDER_CALIBRATION=$PWD/my-cal.json
Run eval with --stack so the file holds both temperature sets. One sample
and a pool of three need different temperatures, because averaging three
samples flattens the distribution. calibrate fits single-sample
temperatures from each request's first sample and pooled ones from the
pools, and a type with fewer than 50 labels keeps T = 1. The bundled
assets/calibration.json was fitted through the cli transport on the
1,798 labelled answers above. Every temperature is below 1, so Haiku's
stated probabilities are underconfident and the calibration sharpens
them. Single-sample T is 0.74 for predicates, 0.60 for choices and 0.81
for scores; the pooled T values are 0.44, 0.52 and 0.50.
Earned exceptions
| Rule |
Banned when |
Earned when |
| One sample |
the confidence decides whether something is escalated, blocked or sent on |
you need the answer, and a wrong one is cheap or gets reviewed anyway |
--stack |
cost is the constraint and nothing reads the confidence |
the confidence gates an action: parallel samples cost no wall time and 3x the tokens |
| Bundled calibration |
your transport or domain differs from the eval's (cli; banking, support and life-science text) |
you have not yet labelled 50 items of your own |
Escalate on agreement < 1 |
nothing stronger sits behind it |
a stronger model or a human can take the split items |
Common failure modes
decide: no usable sample; first error: TypeError: "Could not resolve authentication method..." → the api transport found no key. Set
ANTHROPIC_API_KEY, or pass --transport cli.
decide: no transport → no key and no claude on PATH. Use step 3b.
diagnostics.samples below k → some replies did not parse. Usually a
transport timeout under heavy concurrency; lower DECIDER_WORKERS (default 8)
or raise DECIDER_TIMEOUT (default 60 s).
- Every answer near uniform (confidence around 0.1–0.2 throughout) →
options without descriptions, or descriptions that overlap. Rewrite them so
each has distinct, observable criteria; more samples will not help.
decide: input_transcript: no session id or no transcript for session
→ the script is not running inside a Claude Code session, or the transcript
lives elsewhere. Pass {"path": "/path/to/session.jsonl"}.
- A subagent reply with prose around the JSON →
aggregate takes the
outermost {...} and copes. A reply with no JSON at all counts as a failed
sample.
- Transcript text showing up as evidence in the subagent path → the
[no-context] line was dropped from the prompt. Send the emitted prompt
verbatim.
Verification
Run the tests, then one live request:
python3 -m pytest /mnt/skills/user/deciding-with-confidence/tests -q # 56 collected, no network
echo '{"input":"Nobody can log in since the 2pm deploy.","questions":[{"type":"predicate","name":"outage","instructions":"Is a service down for users?"}]}' \
| python3 $S/decide.py run -
The live answer should have probability above 0.9 and diagnostics.samples
equal to 1 (3 with --stack). A bad success is an answer with samples: 1
from a --stack run: it looks normal but has lost the agreement signal. Check
samples when a batch runs under load.
Diagnosed failures
- 2026-10-09, first eval: 14 of 520 samples were discarded because Haiku
left the options it ruled out of 8-way choices. Missing keys now share the
probability mass the sample left unassigned, and the re-run lost none. The
prompt also asks for ruled-out options at 0.001.
- 2026-10-09:
claude -p --json-schema doubled API time (2.2 s against
0.8 s) with a second turn and three times the output tokens. The cli
transport sends the JSON contract in the system prompt instead.
- 2026-10-09: the claude-workspace PreToolUse hook that appends
parent-transcript chunks to Agent prompts would have put them in the
evidence.
emit appends [no-context], which that hook honours.
- 2026-11-10: the
cli transport passed the sample prompt as a command-line
argument, which caps a single argument at about 128 KB and so cannot carry a
long document. The prompt now goes in on stdin.
- 2026-11-10: a session that pasted a document into each subagent prompt
paid for the document in its own output tokens, once per sample. A request
with
input_file is emitted as a path for the subagent to read.
- 2026-11-10: a decision about the conversation itself meant restating the
conversation in the request.
input_transcript reads it from the session's
transcript file; on a long session one call over the last six turns read
38,580 characters and cost $0.0036.
- 2026-11-11: one temperature per question type, fitted on pools,
applied to single samples and pools alike. With the cue prompt the stack's
pools were further from calibrated (ECE 0.10 to 0.15) than single samples
were, because averaging flattens them. Pools now have their own
temperatures, and cross-validated ECE is 0.01 to 0.05 for both.
1---2name: deciding-with-confidence3description: "Routes, triages, flags and rates a piece of text with a probability for every option: which department or queue a ticket goes to, which intent a message expresses, whether a yes/no condition holds (is this tool call grounded, should the agent ask first, is the customer angry), and how severe or urgent it is on a rubric, each with a calibrated confidence. Runs Claude Haiku in the request and response shapes of OpenAI's Decisions API, or of TypeSafe's System One API (Jev's), for when no purpose-built decision model is at hand. Use for \"route this ticket\", \"triage these\", \"classify with a confidence\", \"give me a probability that\", \"rate the severity\", \"score against this rubric\", guardrail checks before a tool call, \"a Decisions API without OpenAI\", or a stand-in for Jev."4metadata:5 source: https://skillmd.com/skills/oaustegard/deciding-with-confidence6 version: 0.4.07---89# Deciding with confidence1011`scripts/decide.py` takes an OpenAI Decisions API request (`input` plus12`questions` of type `predicate`, `choice` or `score`) and returns its response13shape: a probability per option, the chosen value or score, and `confidence`.14Claude Haiku 5.5 answers. Its logits are not exposed, so the distribution is15estimated: a sample states a probability per option, and the result is16temperature-scaled from a labelled eval.1718The default is **one sample at low effort**: one call, the answer, and the19probability that call stated. That confidence is weaker than a pooled one,20because nothing checks it. When the confidence will gate an action, run21`--stack`: three parallel samples, each under a different angle (plain,22literal, sceptic) and option order, averaged. Their disagreement lowers23`confidence` and shows up in `diagnostics.agreement`.2425Every sample answers each question with a `cue` first: the few words of the26evidence that decide it, quoted. Quoting before committing to probabilities27was worth more than any other prompt change tested.2829Measured 2026-11-11 through the `cli` transport at low effort, on four30labelled sets: 104 short banking-intent, guardrail and severity items; the31held-out half of two life-science sets (77 whole research articles, 28932labelled questions, and 204 article sections, 1,457); and the 2,000 test33decisions of the public Typed Decisions benchmark, sent as System One34requests. Jev, TypeSafe's purpose-built decision model, is scored on the same35items, or taken from the benchmark's card:3637| | short items | whole articles | sections | Typed Decisions |38|---|---|---|---|---|39| one sample (default) | 0.856 | 0.969 | 0.870 | 0.741 |40| `--stack` | 0.894 | 0.962 | 0.906 | 0.7465 |41| Jev | 0.894 | 0.972 | 0.894 | 0.727 |42| one sample without the cue | 0.827 | 0.931 | 0.857 | — |4344Among the 43 general models the Typed Decisions card scores zero-shot, the45stack would rank 10th and one sample 13th; Jev is 18th. The life-science labels come from Sonnet 5.5 and46Opus 5.5 agreeing, which may favour Haiku, and half of the short items were47used to choose the prompt. The stack's main value is its agreement signal:48answers its three samples agreed on were 83% to 98% right, split ones 52% to4973%. One sample costs $0.0005 to $0.0024 a request at list price, the stack50three times that; a stacked request took a median 8.8 s through the CLI, where51Jev answers in under a second. `references/method.md` has the full tables.5253## When NOT to use this skill5455| Situation | Use instead |56|---|---|57| The label set is too large to list in a prompt (a taxonomy, hundreds of tags) | `hallucinating-labels` |58| The answer needs reasoning, multi-step work or a written explanation | a normal Claude call; this skill forbids explanation |59| You need extracted fields or free-form JSON | structured outputs on the Messages API |60| A real decision model is reachable (OpenAI `/v1/decisions`, Strands Decider, Jev) and latency matters | that model: 0.1–0.3 s against ~0.7–4 s here |61| Picking which Claude tier runs a subagent | `agent-routing` |62| The evidence is an image | not supported; the transports send text only |6364Abandon a run when `diagnostics.agreement` is below 1 on most of a batch: the65questions or option descriptions are ambiguous, and more samples will not fix66them. Rewrite the options (distinct, observable criteria) before sampling more.6768## Procedure69701. **Write the request** as JSON in the OpenAI shape. Give every question a71 unique `name`. Options need `description`s that say when each applies; add72 an `other` choice when the set is not exhaustive. Score `levels` go lowest73 first. When the evidence is a file on disk (a paper, a log, a transcript),74 give `"input_file": "path"` in place of `"input"`: the script or the75 subagent reads it, and you do not write the document out again. When the76 evidence is this conversation, give `"input_transcript": true` (or77 `{"last_turns": 20, "max_chars": 60000}`): the script reads the transcript78 Claude Code keeps on disk for the session, renders the last turns with79 tool calls clipped, and redacts secrets. The file can trail the turn in80 progress, so do not ask it about a tool result from this same turn.8182 ```json83 {"input": "I was charged twice for my order.",84 "questions": [85 {"type": "choice", "name": "department", "instructions": "Which department should handle this?",86 "choices": [{"value": "billing", "description": "Payments, invoices, refunds."},87 {"value": "shipping", "description": "Delivery and tracking."},88 {"value": "other", "description": "Anything else."}]},89 {"type": "predicate", "name": "angry", "instructions": "Is the customer expressing anger?"},90 {"type": "score", "name": "severity", "instructions": "How severe is this?",91 "levels": [{"label": "Cosmetic", "description": "No lost functionality."},92 {"label": "Workaround", "description": "Fails, another way works."},93 {"label": "Blocked", "description": "Fails with no workaround."}]}]}94 ```9596 A predicate may carry `"criteria": {"true": "...", "false": "..."}` to say97 what each answer means; without it the defaults are "the condition holds"98 and "does not hold".99100 A **System One request** (the body of TypeSafe's `POST /v1/systemone`, which101 Jev and its clones answer) works as it is: `state` (text, or any JSON,102 which the prompt shows as indented JSON) plus `questions` keyed by id, of103 type `noul`, `choice` or `score`, each with `instructions` and `criteria`.104 The response then comes back in System One's shape: per question id,105 `probabilities` keyed by option, `label`, `confidence`, and106 `probability_true` (noul) or `score` (score), plus `diagnostics`. A request107 written for Jev needs no change.1081092. **Pick the transport.** `--transport auto` (the default) takes the first that works:110111 | Transport | Needs | Wall time per request |112 |---|---|---|113 | `api` | `pip install anthropic` and `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` | ~0.7 s expected (the API time measured inside `claude -p`) |114 | `cli` | the `claude` CLI, logged in | ~4 s, measured |115 | `bedrock` | `pip install anthropic`, `AWS_REGION`, AWS credentials; explicit only | as `api` |116117 With none of them, use the subagent path in step 3b.1181193. **Run it.**120121 a. With a transport:122 ```bash123 S=/mnt/skills/user/deciding-with-confidence/scripts # or wherever the skill lives124 python3 $S/decide.py run request.json # one low-effort sample125 python3 $S/decide.py run request.json --stack # three angles, pooled: use when confidence gates an action126 echo '{...}' | python3 $S/decide.py run -127 python3 $S/decide.py batch requests.jsonl --out responses.jsonl # one request per line, resumable128 ```129 From Python: `sys.path.insert(0, S); import decide; decide.decide(req)` returns130 the response dict, or None when no sample was usable; pass131 `k=3, angles=decide.STACK` for the stack. `--angles` takes any comma-separated132 list, and a request's `"angles"` may hold instruction text of your own.133134 b. In a Claude Code session, the plugin's `decider` subagent is the135 cheaper path: it is pinned to `claude-haiku-5-5` (the model the calibration136 was fitted on) and carries its instructions inline, so it answers without a137 tool call. Install the `deciding-with-confidence` plugin, or copy138 `agents/decider.md` to `.claude/agents/`; agent definitions load at session139 start. Then:140 ```bash141 python3 $S/decide.py emit request.json --stack --agent deciding-with-confidence:decider > batch.json142 ```143 Use the name the Agent tool lists: `deciding-with-confidence:decider` from144 the plugin, `decider` from `.claude/agents/`. Send each `samples[i].prompt`145 verbatim as an Agent call with that `subagent_type`, all in one message.146 Collect the reply texts into a JSON list in `replies.json`, then147 `python3 $S/decide.py aggregate batch.json replies.json`. A single reply148 carries no agreement signal and no calibration until `aggregate` has run.149 With `input_file` the emitted prompt names the file and the subagent reads150 it, so you send a short prompt however long the document is.151152 Without the agent installed, omit `--agent`: the system prompt then rides153 inside each prompt for a stock general-purpose subagent with154 `model: haiku`. That works and costs about 58K tokens a sample against 9K.1551564. **Read the answer.** Predicates give `probability` (P true). Choices give157 `choice`, `probabilities` and `confidence`. Scores give `score` (the158 probability-weighted level index, so 1.4 sits between levels 1 and 2),159 `probabilities` and `confidence`. Every answer carries `diagnostics`:160 `agreement`, `spread`, the pre-calibration `raw` pool, `per_sample`, and161 `cues`, each sample's quote of the evidence it decided on. Read the cues162 when an answer looks wrong: a cue of "none", or a quote from the wrong part163 of the evidence, says the sample was guessing.1641655. **Act on it with thresholds, never on the top answer alone.**166 - With one sample there is no `agreement`; use the top-probability rule167 alone, or rerun the doubtful ones with `--stack`.168 - `agreement < 1` (the samples' top answers differ): escalate. With169 `--stack`, unanimous answers were 90% to 98% right across the three sets170 and split ones 63% to 73%.171 - Top probability under 0.7 (after calibration): escalate. On article172 sections that sent 17% of one-sample answers onward and left 5.6% of173 all answers wrong; with the stack, adding the split rule sent 19% and174 left 3.0% wrong. Both assume the escalation target gets every escalated175 answer right. Treat 0.7 as a starting point.176 - Set the final thresholds from labelled examples of your own traffic,177 weighing the cost of a false positive against a false negative.1781796. **Calibrate on your own labels** when you have 50 or more per question type:180 ```bash181 python3 $S/decide.py eval my_labelled.jsonl --out results.jsonl # {id, input, questions, labels}182 python3 $S/decide.py calibrate results.jsonl --out my-cal.json183 export DECIDER_CALIBRATION=$PWD/my-cal.json184 ```185 Run `eval` with `--stack` so the file holds both temperature sets. One sample186 and a pool of three need different temperatures, because averaging three187 samples flattens the distribution. `calibrate` fits single-sample188 temperatures from each request's first sample and pooled ones from the189 pools, and a type with fewer than 50 labels keeps T = 1. The bundled190 `assets/calibration.json` was fitted through the `cli` transport on the191 1,798 labelled answers above. Every temperature is below 1, so Haiku's192 stated probabilities are underconfident and the calibration sharpens193 them. Single-sample T is 0.74 for predicates, 0.60 for choices and 0.81194 for scores; the pooled T values are 0.44, 0.52 and 0.50.195196## Earned exceptions197198| Rule | Banned when | Earned when |199|---|---|---|200| One sample | the confidence decides whether something is escalated, blocked or sent on | you need the answer, and a wrong one is cheap or gets reviewed anyway |201| `--stack` | cost is the constraint and nothing reads the confidence | the confidence gates an action: parallel samples cost no wall time and 3x the tokens |202| Bundled calibration | your transport or domain differs from the eval's (`cli`; banking, support and life-science text) | you have not yet labelled 50 items of your own |203| Escalate on `agreement < 1` | nothing stronger sits behind it | a stronger model or a human can take the split items |204205## Common failure modes206207- **`decide: no usable sample; first error: TypeError: "Could not resolve208 authentication method..."`** → the `api` transport found no key. Set209 `ANTHROPIC_API_KEY`, or pass `--transport cli`.210- **`decide: no transport`** → no key and no `claude` on PATH. Use step 3b.211- **`diagnostics.samples` below k** → some replies did not parse. Usually a212 transport timeout under heavy concurrency; lower `DECIDER_WORKERS` (default 8)213 or raise `DECIDER_TIMEOUT` (default 60 s).214- **Every answer near uniform** (confidence around 0.1–0.2 throughout) →215 options without descriptions, or descriptions that overlap. Rewrite them so216 each has distinct, observable criteria; more samples will not help.217- **`decide: input_transcript: no session id` or `no transcript for session`**218 → the script is not running inside a Claude Code session, or the transcript219 lives elsewhere. Pass `{"path": "/path/to/session.jsonl"}`.220- **A subagent reply with prose around the JSON** → `aggregate` takes the221 outermost `{...}` and copes. A reply with no JSON at all counts as a failed222 sample.223- **Transcript text showing up as evidence in the subagent path** → the224 `[no-context]` line was dropped from the prompt. Send the emitted prompt225 verbatim.226227## Verification228229Run the tests, then one live request:230231```bash232python3 -m pytest /mnt/skills/user/deciding-with-confidence/tests -q # 56 collected, no network233echo '{"input":"Nobody can log in since the 2pm deploy.","questions":[{"type":"predicate","name":"outage","instructions":"Is a service down for users?"}]}' \234 | python3 $S/decide.py run -235```236237The live answer should have `probability` above 0.9 and `diagnostics.samples`238equal to 1 (3 with `--stack`). A **bad success** is an answer with `samples: 1`239from a `--stack` run: it looks normal but has lost the agreement signal. Check240`samples` when a batch runs under load.241242## Diagnosed failures243244- **2026-10-09, first eval:** 14 of 520 samples were discarded because Haiku245 left the options it ruled out of 8-way choices. Missing keys now share the246 probability mass the sample left unassigned, and the re-run lost none. The247 prompt also asks for ruled-out options at 0.001.248- **2026-10-09:** `claude -p --json-schema` doubled API time (2.2 s against249 0.8 s) with a second turn and three times the output tokens. The `cli`250 transport sends the JSON contract in the system prompt instead.251- **2026-10-09:** the claude-workspace PreToolUse hook that appends252 parent-transcript chunks to Agent prompts would have put them in the253 evidence. `emit` appends `[no-context]`, which that hook honours.254- **2026-11-10:** the `cli` transport passed the sample prompt as a command-line255 argument, which caps a single argument at about 128 KB and so cannot carry a256 long document. The prompt now goes in on stdin.257- **2026-11-10:** a session that pasted a document into each subagent prompt258 paid for the document in its own output tokens, once per sample. A request259 with `input_file` is emitted as a path for the subagent to read.260- **2026-11-10:** a decision about the conversation itself meant restating the261 conversation in the request. `input_transcript` reads it from the session's262 transcript file; on a long session one call over the last six turns read263 38,580 characters and cost $0.0036.264- **2026-11-11:** one temperature per question type, fitted on pools,265 applied to single samples and pools alike. With the cue prompt the stack's266 pools were further from calibrated (ECE 0.10 to 0.15) than single samples267 were, because averaging flattens them. Pools now have their own268 temperatures, and cross-validated ECE is 0.01 to 0.05 for both.