LLM Output Parsing

LLM Output Parsing

The dangerous bug in reading a model's output is the one that looks like success. You regex the first approve out of a judge's reply — but the judge reasoned "my instinct was to reject… on reflection, approve", so you extract reject, the exact opposite of the verdict, with total confidence and no error raised. Preventing that silent-wrong-answer is the point: when the text carries conflicting values this skill reports ambiguous and refuses to pick, because a wrong value that looks certain is worse than an honest "I couldn't tell — ask again."

That refusal is one rule — never invent a value — applied across the two shapes a consumed output arrives in: prose you can't get as JSON, and "mostly JSON" wrapped in fences and preambles. Each corrupts the pipeline silently if mishandled; this skill turns either into a trustworthy typed value or a clean re-ask.

Two problems, one boundary

• Prose you can't get as JSON. The producer is a component you don't control, a model answering conversationally, or an LLM judge burying its verdict in reasoning: "my first instinct was to reject this, but on reflection I'd approve it, maybe 85% confident." You still need the decision out of that text.
• "Mostly JSON." You asked for JSON and got JSON with company: a ```json fence, a "Here's the result:" preamble, a "let me know if you need changes" coda, a trailing comma learned from a million code samples. Every one makes json.loads(raw) throw.

Different inputs, different failure modes — but the same consumer boundary and the same refusal to guess. Handle the one you have.

Half 1 — recovering a signal from prose

This is the half that catches the silent-wrong-answer above. Extraction reports one of four honest outcomes per target:

• ok — exactly one unambiguous, valid value. Use it.
• not_found — no value present. The caller re-asks or applies a documented default — a knowing choice, not a silent zero.
• ambiguous — multiple conflicting values (both "approve" and "reject" appear). The caller must not guess. Re-ask, or escalate to a human. This is the outcome the naive regex never surfaces and the whole reason this half exists.
• invalid — a value was found but fails its type, enum, or bounds (a score of 150, an enum word not in the set).

Prefer labels, fall back to scanning. Extraction is most reliable when the signal is labeled — "Verdict: approve", "Score: 85", "Blocking: no" — so it looks for name: value / name = value / name is value first. Only if there's no label does it scan the whole text, and that's exactly where ambiguity is likely (reasoning and conclusion both mention enum values). Practical guidance: prompt the producer to label its answers, even in prose. A labeled prose output is nearly as parseable as JSON.

Half 2 — validating "mostly JSON"

When you did get JSON, the boundary is a four-stage pipeline, and its safety rule is the same: locating and de-fencing JSON is deterministic and lossless; filling in a missing field or fixing a wrong one is authorship — the model's job, via a bounded retry, not something a parser papers over.

1. Extract. Pull the JSON from a fenced block if present; otherwise scan for the outermost balanced {...}/[...] and ignore the prose around it. Most "invalid" model output is valid JSON wearing a coat.
2. Repair — safely only. Strip trailing commas and nothing else. Safe repairs cannot change the parsed value; anything that could (quoting a bareword, inventing a close brace) is off-limits — a silently-wrong object is worse than a clean failure.
3. Parse. json.loads the candidate. If it still won't parse after the safe repair, that's a re-ask, not a heroics problem.
4. Validate. Required fields present, types correct, enums and numeric bounds respected — where a structurally-valid but semantically-wrong output (an out-of-range score, an invented enum value) is caught before it reaches a consumer.

The three JSON outcomes map to the three exit codes: valid (0) hand downstream; invalid (1) re-ask with the specific error as feedback ("field confidence must be 0–100; you returned 150") up to a bounded number of retries; input error (2) the schema itself is unreadable — a wiring problem, not a model one.

Which half, and when

If you control the producer, ask for JSON and use Half 2 — it's more reliable. Half 1 is the fallback for when you can't change the producer (a component you don't own, an inherently narrative answer, a judge that reasons out loud). Prose extraction is second choice, not the default.

Run vs read

python3 .../parse_output.py    --spec spec.json           --output raw.txt   # prose
python3 .../output_validate.py --schema json-validation/out.schema.json --output raw.txt   # mostly-JSON

Common pitfalls

• First-match extraction (prose). Grabbing the first enum word ignores that the model may argue the other side first — the silent-wrong-answer bug this is built to prevent.
• json.loads(raw) with no extraction (JSON). Crashes on the fence, the preamble, the coda — i.e. on most real outputs. Extract first.
• "Repairing" by guessing. Adding a missing field or coercing a wrong enum hides a real model error and ships a plausible-looking lie. Repair only what can't change meaning; escalate the rest to a re-ask.
• Guessing on ambiguity. If the prose says both, the honest answer is "ask again", never a coin flip dressed as certainty.
• Retrying without the error. "Try again" gets the same output; "field X must be an integer, you sent a string" gets a fix. Feed the specific failure back.
• Unbounded retries. A model that can't produce valid output in 2–3 tries won't on the 10th; cap it and surface the failure (see two-layer-critic's retry cap).
• Treating not_found as a default value. "No verdict found" is not "reject"; surface it and let the caller decide.

Verification checklist

• sh examples/selftest.sh passes (prose clean → 0, ambiguous/missing/out-of-range → 1, missing spec → 2; JSON clean/fenced/prose/trailing-comma → 0, bad enum-or-bounds and non-JSON → 1, missing schema → 2).
• Conflicting prose values produce ambiguous, and the caller re-asks rather than guessing.
• Only value-preserving JSON repairs are applied; nothing is invented.
• Invalid output triggers a bounded re-ask carrying the specific error, not a crash or a coercion.
• JSON (Half 2) is used instead of prose extraction wherever the producer can be changed.

Related skills in this bundle

• prompt-contracts — validates the output schema statically (enums narrow the registry); this validates a runtime instance against that schema.
• event-envelope — the envelope carries an opaque payload; this is how a consumer safely turns that payload text into a checked object.
• cross-model-verification / two-layer-critic — a verifier's or judge's verdict often arrives as prose; this is how you extract it, ambiguity and all, then validate it before acting.
• eval-harness — both refuse to manufacture certainty a source doesn't support; here it's the extracted value, there (its grounding check) it's the factual claim.