Andersonlimahw Token Benchmark
Required experiment design
- Define same task set, inputs, acceptance criteria, concurrency, warmup, and sample count.
- Capture one JSONL row per run using
references/usage-ledger.schema.json; pair variants bycase_id, pin tokenizer and evaluator versions, and keep provider/model/evidence constant within each variant. - Record actual provider usage when available. Otherwise label estimate and document tokenizer/method.
- Compare total input/output/cached/reasoning tokens, calls, latency median/p95, schema retries, parse failures, task accuracy/calibration, and USD using dated price snapshot.
- Do not compare token counts across different tokenizers as if units were identical. Show calls, latency, cost, and quality beside tokens.
- Run
python3 scripts/benchmark.py runs.jsonl --baseline baseline --candidate typed --output report.md --slides slides.md(Python standard library only). - Review generated report/slides; add methodology, limitations, and decision.
Claim policy
measured: observed in this experiment.estimated: derived from disclosed tokenizer or price assumptions.vendor_claimed: quoted claim with source/date; never present as reproduced result.
Provider terms may restrict publication of service benchmarks or use of brand names. Keep results internal unless applicable terms allow publication or written permission exists.
Do not optimize tokens alone. A cheaper invalid decision is regression.