Cost-Performance Scorer
Overview
Routing strategies are only as good as the data that informs them. Selective
intelligence works only if you can measure it, and the book names two metrics
that matter:
- Cost per successful completion. Cost per task completed correctly, not
cost per token. A cheap model that fails 40% of the time is not cheaper than
an expensive model that succeeds on the first attempt — the wasted spend on
failures is amortized over the successes.
- Quality parity threshold. The minimum acceptable quality per node type.
The AlertClassifier may need 0.99 (a wrong validation is worse than none); the
QueryAnalyst may tolerate 0.90 because downstream nodes recover from
misclassification.
The book's harness wraps every node and logs a NodeInvocation per call
(Example 8-3). Success is judged per node — matching a human-labeled severity
for the AlertClassifier, an SRE accepting the recommendation for
PredictionSynthesis. Each candidate model is scored against a per-node
evaluation set with domain failure weights (Example 8-4): a P1 alert
misclassified as P3 is 10x worse than the reverse, an asymmetry a generic
accuracy metric cannot express. Kakao's AI Shopping Mate is the worked anchor:
restructuring GPT-4o-everywhere into a workflow graph of fine-tuned 27-32B
models moved format adherence from 0.655 to 0.987 and accuracy from 0.578 to
0.890, with the smaller models preferred over GPT-4o in 63% of 2,100 turns.
When to Use
- You have run production traffic through a routed pipeline and need to know
which node is over- or under-provisioned.
- You are choosing a specialist SLM and must evaluate it on your data, not MMLU.
- A cheap node looks cheap per token but you suspect its failure rate erases the
savings.
- You need a per-node quality gate before promoting a routing change.
Phrases that should invoke this skill: "cost per successful completion", "is
this model actually cheaper", "per-node evaluation set", "quality parity
threshold", "score the routing policy", "weighted error rate".
When NOT to Use
- Deciding routing before you run — that is
model-routing-selector (Ch8).
This skill measures what happened; that one decides what should happen.
- Generic capability benchmarking — MMLU/HumanEval scores do not predict
performance on your alert taxonomy (Ch8 Common Pitfalls: "benchmarking models
on the wrong evaluation set").
- Latency / KV-cache budgeting — that is
kv-cache-latency-budgeter.
- When you have no ground-truth success signal. Cost-per-success requires a
per-node evaluator (gold labels or downstream acceptance). Without it, you can
only report cost per call, which is the metric this skill exists to replace.
Process
| Step |
Input |
Action |
Output |
Verification |
| 1 |
Invocation log JSON |
_load_tracker(path) |
CostTracker of NodeInvocations |
Each row has node_name, cost_usd, success |
| 2 |
Node name |
tracker.cost_per_success(node) |
$/successful completion |
Finite, > 0; higher than $/call by the failure ratio |
| 3 |
Node name |
tracker.node_report(node) |
success_rate + p95 latency + $/success |
success_rate in [0,1] |
| 4 |
(gold, pred) pairs + eval set |
lib.evaluate_candidate(preds, set) |
accuracy + meets_threshold + weighted_error_rate |
weighted_error applies the domain failure weights |
| 5 |
Full tracker |
lib.score_policy(tracker) |
Pipeline total + blended $/success |
Sums per-node; blended ≥ max single-node $/success/pipeline-len |
| 6 |
Two model profiles |
lib.compare_cost_per_success(...) |
Which is cheaper per call vs per success |
Cheaper-per-call and cheaper-per-success can disagree |
Rationalizations
| Agent rationalization |
Documented rebuttal |
| "This model is 10x cheaper per token, ship it." |
Cost per token is not cost per success. If it fails 40% of the time and each failure triggers a retry or a downstream error, the true cost per completed task can exceed the reliable model (Ch8). Measure cost-per-success. |
| "It scores 85% on MMLU, that's good enough." |
MMLU does not contain your alert-classification taxonomy. The book's pitfall is exactly this: build a per-node eval set from production data and re-evaluate when the task distribution shifts. |
| "Accuracy is 0.985, close enough to the 0.99 bar." |
For the AlertClassifier a missed P1 is catastrophic; the bar is 0.99 and 0.985 fails it. The weighted_error_rate exists to price that asymmetry (P1_as_P3 = 10x). |
| "One eval set for the whole pipeline is simpler." |
Nodes have different stakes: 0.99 for the AlertClassifier, 0.90 for the QueryAnalyst. A shared set over- or under-gates half the pipeline (Example 8-4). |
| "Smaller models always cost quality." |
Kakao's fine-tuned 27-32B models beat GPT-4o on their task (0.987 vs 0.655 format adherence) and were preferred in 63% of 2,100 turns. Specialization can improve quality because the training signal is focused. |
Red Flags
- Cost per success reported as cost per call. If success is not evaluated,
the headline metric is missing; wire a per-node evaluator first.
- A node's success_rate is 1.0 across thousands of calls. Either the
evaluator is a no-op or the eval set is trivial — real production nodes fail.
- weighted_error_rate equals the raw error rate. The failure weights are not
being applied; check the
<gold>_as_<pred> keys.
- Blended $/success unchanged after a routing change. The change moved cost
between nodes without improving completions; re-inspect per-node reports.
- CLI
--help exits non-zero. SKILL.md / CLI mismatch.
Non-Negotiable Verification
Before trusting a policy score:
Run the benchmark battery.
python cli.py benchmark
Confirms cost-per-success is computed, the domain failure weights apply
(P1_as_P3 = 10.0), the 0.99 threshold gate is honoured, and the
cost-per-success inversion holds (cheaper-per-call ≠ cheaper-per-success).
Score the sample policy and read the per-node table.
python cli.py score
Confirm the SLM nodes are far cheaper per success than the frontier synthesis
node, and that a node with seeded failures shows success_rate < 1.0.
Evaluate a candidate against a real per-node set.
python cli.py evaluate AlertClassifier
Confirm a single P1-as-P3 miss produces a weighted_error_rate of 10.0, not 1.0.
Domain test in the notebook. Run notebooks/ch8-optimization.ipynb;
confirm the cost-performance section reads moto-mocked CloudWatch cost and
latency signals into the tracker and computes cost-per-success per node.
Security Posture
- Read-only over the invocation log.
lib.py makes no network calls and
writes nothing; it consumes a log the caller supplies.
- Cost figures are inputs, not authority. The log's
cost_usd values come
from the caller's pricing; verify them against live pricing before acting.
- Untrusted logs. If the invocation log is machine-generated from an
external system, validate the schema (numeric costs, boolean success) before
scoring — a poisoned log could hide a failing node behind fabricated successes.
Composition
- Closes the loop with
model-routing-selector (Ch8): the cost-per-success
numbers here recalibrate the cascade thresholds and per-node bars there.
- Feeds the Ch7 self-evolution loop: cost-per-success and weighted-error are
evaluation signals the diagnostic engine can act on.
- Pairs with
kv-cache-latency-budgeter (Ch8): this scores cost/quality;
that scores latency/concurrency. Both gate a production routing change.
Source Attribution
Distilled from Agentic GraphRAG (O'Reilly, by Anthony Alcaraz and Sam Julien),
Chapter 8 — Optimization, "Measuring Cost-Performance Tradeoffs". Key references:
- Example 8-3
CostTracker / NodeInvocation (cost per successful completion)
- Example 8-4 per-node evaluation sets with domain failure weights
- Kakao AI Shopping Mate (0.655 -> 0.987 format adherence, 0.578 -> 0.890
accuracy, 63% preference over GPT-4o across 2,100 turns)
1---2name: cost-performance-scorer3description: Score a multi-model routing policy on cost versus quality using the two metrics that actually decide selective intelligence: cost per successful completion (not cost per token) and a per-node quality parity threshold with domain-specific failure weights. Wraps a NodeInvocation log, computes cost-per-success and p95 latency per node, and evaluates a candidate model against a per-node evaluation set drawn from production data. Use to justify or recalibrate a routing decision AFTER you have run traffic. NOT for deciding routing a priori (that is model-routing-selector), NOT for generic benchmarks (MMLU/HumanEval do not capture your alert taxonomy), NOT for latency/KV budgeting (that is kv-cache-latency-budgeter).4---56# Cost-Performance Scorer78## Overview910Routing strategies are only as good as the data that informs them. Selective11intelligence works only if you can measure it, and the book names two metrics12that matter:1314- **Cost per successful completion.** Cost per *task completed correctly*, not15 cost per token. A cheap model that fails 40% of the time is not cheaper than16 an expensive model that succeeds on the first attempt — the wasted spend on17 failures is amortized over the successes.18- **Quality parity threshold.** The minimum acceptable quality per node type.19 The AlertClassifier may need 0.99 (a wrong validation is worse than none); the20 QueryAnalyst may tolerate 0.90 because downstream nodes recover from21 misclassification.2223The book's harness wraps every node and logs a `NodeInvocation` per call24(Example 8-3). Success is judged per node — matching a human-labeled severity25for the AlertClassifier, an SRE accepting the recommendation for26PredictionSynthesis. Each candidate model is scored against a *per-node*27evaluation set with domain failure weights (Example 8-4): a P1 alert28misclassified as P3 is 10x worse than the reverse, an asymmetry a generic29accuracy metric cannot express. Kakao's AI Shopping Mate is the worked anchor:30restructuring GPT-4o-everywhere into a workflow graph of fine-tuned 27-32B31models moved format adherence from 0.655 to 0.987 and accuracy from 0.578 to320.890, with the smaller models preferred over GPT-4o in 63% of 2,100 turns.3334## When to Use3536- You have run production traffic through a routed pipeline and need to know37 which node is over- or under-provisioned.38- You are choosing a specialist SLM and must evaluate it on your data, not MMLU.39- A cheap node looks cheap per token but you suspect its failure rate erases the40 savings.41- You need a per-node quality gate before promoting a routing change.4243Phrases that should invoke this skill: "cost per successful completion", "is44this model actually cheaper", "per-node evaluation set", "quality parity45threshold", "score the routing policy", "weighted error rate".4647## When NOT to Use4849- **Deciding routing before you run** — that is `model-routing-selector` (Ch8).50 This skill measures what happened; that one decides what should happen.51- **Generic capability benchmarking** — MMLU/HumanEval scores do not predict52 performance on your alert taxonomy (Ch8 Common Pitfalls: "benchmarking models53 on the wrong evaluation set").54- **Latency / KV-cache budgeting** — that is `kv-cache-latency-budgeter`.55- **When you have no ground-truth success signal.** Cost-per-success requires a56 per-node evaluator (gold labels or downstream acceptance). Without it, you can57 only report cost per call, which is the metric this skill exists to replace.5859## Process6061| Step | Input | Action | Output | Verification |62|------|-------|--------|--------|--------------|63| 1 | Invocation log JSON | `_load_tracker(path)` | CostTracker of NodeInvocations | Each row has node_name, cost_usd, success |64| 2 | Node name | `tracker.cost_per_success(node)` | $/successful completion | Finite, > 0; higher than $/call by the failure ratio |65| 3 | Node name | `tracker.node_report(node)` | success_rate + p95 latency + $/success | success_rate in [0,1] |66| 4 | (gold, pred) pairs + eval set | `lib.evaluate_candidate(preds, set)` | accuracy + meets_threshold + weighted_error_rate | weighted_error applies the domain failure weights |67| 5 | Full tracker | `lib.score_policy(tracker)` | Pipeline total + blended $/success | Sums per-node; blended ≥ max single-node $/success/pipeline-len |68| 6 | Two model profiles | `lib.compare_cost_per_success(...)` | Which is cheaper per call vs per success | Cheaper-per-call and cheaper-per-success can disagree |6970## Rationalizations7172| Agent rationalization | Documented rebuttal |73|-----------------------|---------------------|74| "This model is 10x cheaper per token, ship it." | Cost per token is not cost per *success*. If it fails 40% of the time and each failure triggers a retry or a downstream error, the true cost per completed task can exceed the reliable model (Ch8). Measure cost-per-success. |75| "It scores 85% on MMLU, that's good enough." | MMLU does not contain your alert-classification taxonomy. The book's pitfall is exactly this: build a per-node eval set from production data and re-evaluate when the task distribution shifts. |76| "Accuracy is 0.985, close enough to the 0.99 bar." | For the AlertClassifier a missed P1 is catastrophic; the bar is 0.99 and 0.985 fails it. The weighted_error_rate exists to price that asymmetry (P1_as_P3 = 10x). |77| "One eval set for the whole pipeline is simpler." | Nodes have different stakes: 0.99 for the AlertClassifier, 0.90 for the QueryAnalyst. A shared set over- or under-gates half the pipeline (Example 8-4). |78| "Smaller models always cost quality." | Kakao's fine-tuned 27-32B models beat GPT-4o on their task (0.987 vs 0.655 format adherence) and were preferred in 63% of 2,100 turns. Specialization can *improve* quality because the training signal is focused. |7980## Red Flags8182- **Cost per success reported as cost per call.** If success is not evaluated,83 the headline metric is missing; wire a per-node evaluator first.84- **A node's success_rate is 1.0 across thousands of calls.** Either the85 evaluator is a no-op or the eval set is trivial — real production nodes fail.86- **weighted_error_rate equals the raw error rate.** The failure weights are not87 being applied; check the `<gold>_as_<pred>` keys.88- **Blended $/success unchanged after a routing change.** The change moved cost89 between nodes without improving completions; re-inspect per-node reports.90- **CLI `--help` exits non-zero.** SKILL.md / CLI mismatch.9192## Non-Negotiable Verification9394Before trusting a policy score:95961. **Run the benchmark battery.**97 ```98 python cli.py benchmark99 ```100 Confirms cost-per-success is computed, the domain failure weights apply101 (P1_as_P3 = 10.0), the 0.99 threshold gate is honoured, and the102 cost-per-success inversion holds (cheaper-per-call ≠ cheaper-per-success).1031042. **Score the sample policy and read the per-node table.**105 ```106 python cli.py score107 ```108 Confirm the SLM nodes are far cheaper per success than the frontier synthesis109 node, and that a node with seeded failures shows success_rate < 1.0.1101113. **Evaluate a candidate against a real per-node set.**112 ```113 python cli.py evaluate AlertClassifier114 ```115 Confirm a single P1-as-P3 miss produces a weighted_error_rate of 10.0, not 1.0.1161174. **Domain test in the notebook.** Run `notebooks/ch8-optimization.ipynb`;118 confirm the cost-performance section reads `moto`-mocked CloudWatch cost and119 latency signals into the tracker and computes cost-per-success per node.120121## Security Posture122123- **Read-only over the invocation log.** `lib.py` makes no network calls and124 writes nothing; it consumes a log the caller supplies.125- **Cost figures are inputs, not authority.** The log's `cost_usd` values come126 from the caller's pricing; verify them against live pricing before acting.127- **Untrusted logs.** If the invocation log is machine-generated from an128 external system, validate the schema (numeric costs, boolean success) before129 scoring — a poisoned log could hide a failing node behind fabricated successes.130131## Composition132133- **Closes the loop with** `model-routing-selector` (Ch8): the cost-per-success134 numbers here recalibrate the cascade thresholds and per-node bars there.135- **Feeds** the Ch7 self-evolution loop: cost-per-success and weighted-error are136 evaluation signals the diagnostic engine can act on.137- **Pairs with** `kv-cache-latency-budgeter` (Ch8): this scores cost/quality;138 that scores latency/concurrency. Both gate a production routing change.139140## Source Attribution141142Distilled from *Agentic GraphRAG* (O'Reilly, by Anthony Alcaraz and Sam Julien),143Chapter 8 — Optimization, "Measuring Cost-Performance Tradeoffs". Key references:144145- Example 8-3 `CostTracker` / `NodeInvocation` (cost per successful completion)146- Example 8-4 per-node evaluation sets with domain failure weights147- Kakao AI Shopping Mate (0.655 -> 0.987 format adherence, 0.578 -> 0.890148 accuracy, 63% preference over GPT-4o across 2,100 turns)
AnthonyAlcaraz