Structured Output And Tool Calling

Structured Output & Tool Calling Reliability

The moment an LLM's output feeds deterministic code — a parser, a database write, a payment call — "usually valid" is a production incident waiting on a percentage. This skill is about closing that gap: making the structure guaranteed, the contents validated, the execution idempotent, and the failures recoverable.

Core mental model — three independent guarantees, do not conflate them:

1. Structural validity (is it parseable JSON matching the schema shape?) — solved by constrained decoding.
2. Semantic validity (are the values correct, in-range, referencing real things?) — solved only by validation + repair.
3. Execution safety (does running it twice double-charge?) — solved only by idempotency.

Constrained decoding gives you #1 for free and nothing else. You still own #2 and #3.

1. Structured output: JSON mode vs constrained decoding

The single most important distinction in this whole skill:

Why constrained decoding eliminates malformed JSON: the decoder maintains a grammar state machine. Before sampling each token, it computes the set of tokens that keep the partial output grammar-valid and sets every other token's logit to -inf. The model literally cannot sample a } where the schema requires more fields, or a string where it requires an integer. There is no "the model usually closes its braces" — closing is the only legal move. This is the difference between guaranteed-valid and best-effort. If you have access to constrained decoding for a schema-shaped task, use it; it deletes an entire class of bugs.

Provider mapping (capability-level — verify exact model support before relying on it)

• OpenAI Structured Outputs — response_format: {type:"json_schema", json_schema:{name, strict:true, schema:{...}}}. With strict:true this is constrained decoding: output is guaranteed to match the JSON Schema (subset: every field required, additionalProperties:false, limited type expressivity). The older {type:"json_object"} is JSON mode — best-effort, validate it.
• OpenAI tool calls — pass tools with JSON-Schema parameters; strict:true on a function applies the same constrained guarantee to its arguments.
• Anthropic (Claude) — no native json_schema response_format. The stable pattern is tool-schema coercion: define a single tool whose input_schema is your target schema, then force it with tool_choice:{type:"tool", name:"..."}. The model fills the tool's arguments = your structured object. Read it from the tool_use block. Validation still required; coercion biases shape but is not token-masked grammar enforcement.
• Open / local — Outlines (regex + JSON-Schema → FSM-masked logits), llama.cpp GBNF grammars, vLLM/SGLang/TGI guided-decoding backends (guided_json, guided_grammar). These are true constrained decoding and work with any JSON Schema or hand-written grammar.

GBNF sketch — a grammar IS the contract:
root   ::= "{" ws "\"action\"" ws ":" ws action ws "," ws "\"id\"" ws ":" ws integer ws "}"
action ::= "\"create\"" | "\"update\"" | "\"delete\""   # enum, structurally enforced
integer::= "-"? [0-9]+

Validate at the boundary — always, even with constrained decoding

Constrained decoding guarantees shape, not meaning. A schema-valid object can still say quantity: -5 or country: "Wakanda". Put a real validator at the trust boundary and parse into a typed object so the rest of your code touches only validated data.

// Zod — single source of truth: feed .toJSON()→JSON Schema to the model, parse the reply
const Order = z.object({
  action: z.enum(["create", "update", "delete"]),
  quantity: z.number().int().positive(),        // catches -5 that the schema "shape" allows
  sku: z.string().regex(/^[A-Z]{3}-\d{4}$/),
});
const parsed = Order.safeParse(JSON.parse(raw));
if (!parsed.success) return repairLoop(raw, parsed.error);   // see §3

# Pydantic — model_validate_json does parse + validate + coerce in one step
class Order(BaseModel):
    action: Literal["create", "update", "delete"]
    quantity: PositiveInt
    sku: constr(pattern=r"^[A-Z]{3}-\d{4}$")
try:
    order = Order.model_validate_json(raw)
except ValidationError as e:
    return repair_loop(raw, e)

Do: derive the model-facing JSON Schema from your Zod/Pydantic type (one source of truth). Don't: hand-write a JSON Schema for the API and a separate validator — they drift, and the gap is exactly where bad data flows.

2. Failure modes of structured output

Even good setups fail. Know the catalogue so you can detect and repair each one specifically.

The repair loop

Treat the validator's error as a signal to the model, not just a 500 to the user. Bounded, escalating, idempotent.

validate(output)
  ├─ ok ────────────────────────────────► return typed object
  └─ fail
       ├─ attempts < N (e.g. 2):
       │     feed back: original output + the EXACT validator error
       │     ("field `quantity` must be a positive integer, got -5; return corrected JSON only")
       │     re-call SAME model → validate again (loop)
       ├─ attempts exhausted:
       │     ESCALATE → stronger model, same prompt + error
       └─ still failing:
             FALLBACK → deterministic default / queue for human / typed error to caller

Rules that make repair work:

• Bound it. 1–2 repair attempts, then escalate. Unbounded retry loops burn tokens and latency on a model that's stuck.
• Feed the real error back verbatim. "Invalid input" teaches the model nothing. "sku must match ^[A-Z]{3}-\d{4}$, got abc" gets a fix in one shot.
• Idempotent repair. A repair retry must not have side effects — never execute the action mid-repair; only re-derive the object. Execution happens once, after a clean validate.
• Partial-parse / streaming JSON. When streaming, use a tolerant incremental parser (partial-json, best-effort-json-parser, jsonrepair) to render in-progress UI, but only validate and act on the final completed object. Never trigger a tool from a partial parse.
• Truncation is special. If finish_reason === "length", the object isn't wrong — it's incomplete. Don't "repair" it as malformed; re-request with higher max_tokens or use a continuation strategy.

3. Function / tool calling reliability

A tool call is structured output whose schema is a function signature — every structured-output rule applies, plus the brutal new fact: the output triggers real-world side effects. A hallucinated number in a summary is cosmetic; a hallucinated argument to refund(amount) is money.

Tool contracts — design for the model, not just the compiler

{
  name: "search_orders",                       // verb_noun, unambiguous, no overlap with siblings
  description: "Search orders by customer email. Returns up to `limit` orders, newest first. " +
               "Use ONLY for lookups — does not modify anything.",   // says what it does AND when to use it
  parameters: {
    type: "object",
    properties: {
      email:  { type: "string", format: "email", description: "Exact customer email" },
      status: { type: "string", enum: ["open","shipped","cancelled"] },  // constrain, don't free-text
      limit:  { type: "integer", minimum: 1, maximum: 50, default: 20 },
    },
    required: ["email"],                        // required vs optional, explicit
    additionalProperties: false,
  },
}

Validate arguments BEFORE executing — never trust LLM args

The model's tool call is a request, not an authorization. Re-validate against the schema and business rules in your code before the side effect runs.

function dispatch(call: ToolCall) {
  const tool = registry[call.name];
  if (!tool) return toolError(call, `Unknown tool '${call.name}'. Available: ${Object.keys(registry)}`);  // hallucinated tool
  const args = tool.schema.safeParse(call.arguments);
  if (!args.success) return toolError(call, args.error.message);   // hallucinated/invalid args → back to model
  if (!authorize(tool, args.data)) return toolError(call, "Not permitted");  // business rule, not the LLM's call
  return tool.run(args.data);   // only now does anything happen
}

The hallucinated tool call comes in two flavors, both caught above by not trusting the call:

• Tool that doesn't exist — model invents delete_everything. Catch: registry lookup fails → return a tool-result error listing real tools; the model self-corrects next turn.
• Wrong / invented arguments — right tool, garbage args (made-up order ID, out-of-enum status). Catch: schema validation fails → return the error as the tool result; do not execute.

Returning the error as a tool result (not throwing) keeps the model in the loop and lets it retry with corrected input — far better than crashing the run.

Tool-choice forcing & parallel calls

• tool_choice — auto (model decides), required/any (must call some tool), or force a specific tool (= structured-output coercion, §1). Force when you know a tool is needed; leave auto for genuine routing.
• Parallel tool calls — providers may return several tool_use blocks in one turn. Execute them, but: validate each independently, watch for the same write issued twice (dedupe via idempotency, §4), and return all results before the next model turn or you desync the conversation.

Too-many-tools degradation — cap and route

Accuracy of tool selection drops as the toolset grows (more near-synonym descriptions to disambiguate, more schema in context). Past ~15–20 tools, expect wrong-tool picks and ignored tools.

Mitigations, cheapest first: (1) cap the tools exposed per call to the handful relevant to the task; (2) route — a cheap first-stage model/retriever picks the relevant 5–10 tools, then the main call sees only those; (3) namespace/group tools and expose one group at a time; (4) merge near-duplicate tools behind one with an action enum. Don't paste your entire API surface into every request.

4. Idempotency — so retries don't double-charge

Repair loops, parallel calls, network retries, and at-least-once queues all mean a write tool will be invoked more than once for one logical intent. If executing it twice does damage, you have a latent incident. Design every write tool so duplicate execution is a no-op.

Patterns:

• Idempotency keys. Derive a stable key from the logical intent (hash(tool + canonical_args) or a request-scoped UUID minted once before any retry). Pass it to the downstream system (Stripe-style Idempotency-Key) or store seen_keys; on a duplicate key, return the cached prior result instead of re-executing.
• Dedupe at the boundary. Before executing a write, check whether this exact (key) already ran this session. The repair loop must reuse the same key across attempts — minting a new key per retry defeats the whole mechanism.
• Read vs write safety. Auto-execute read tools; gate write tools through validation + idempotency; gate destructive tools through an explicit confirmation step (surface the action to a human/approval queue before running).
• Make the model's retry cheap and safe. Because tool-arg errors are fed back for repair, the same call can arrive twice with one tweak — idempotency keys keyed on intent (not on the exact bytes) absorb that.

5. Fallback chains

Compose the mechanisms into a single degrade-gracefully path. Each stage is strictly more expensive/slower than the last; stop at the first success.

1. Constrained decoding (json_schema strict / GBNF)   ← structurally valid, cheapest
2. Validate (Zod/Pydantic)                            ← semantic check
3. Repair loop ×N (feed error back, same model)       ← fixes most semantic misses
4. Escalate to a stronger model (same prompt+error)   ← for genuinely hard cases
5. Deterministic default / human review / typed error ← never crash the caller

The non-negotiable bottom rung: a typed, actionable failure, never an unhandled throw or a silent empty object. Downstream code should receive Result<T, StructuredOutputError>, not a surprise.

6. Evals for structured output

You cannot improve reliability you don't measure. Run these continuously on a fixed input set across model/prompt/schema changes.

Do: keep a golden set of inputs with expected structured outputs; assert per-field, not just "did it parse." A 100% valid-rate with 60% field accuracy is a worse failure than a parse error — it looks fine and ships wrong data. Don't: measure only valid-rate; constrained decoding can make it 100% while the contents are nonsense.

Quick reference — decision order

1. Can I use constrained decoding for this schema? → Do it. Malformed JSON gone.
2. Validate the result at the boundary with Zod/Pydantic into a typed object — always, even after #1.
3. On failure: repair (≤2×, feed the exact error) → escalate model → fallback to a typed default/human.
4. For tools: tight contracts, validate args before executing, catch hallucinated calls by not trusting them, cap/route when tools are many.
5. Make every write idempotent; gate destructive actions on confirmation.
6. Eval valid-rate and field accuracy continuously — the second is the one that lies quietly.