How do I install the Bench skill?

Run npx skillmds@latest add zapgun-ai/bench in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Bench safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Bench?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Bench free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

zapgun-ai (@zapgun-ai) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Bench by zapgun-ai — Agent Skill

Name: Bench
Author: zapgun-ai

Bench

Analyze clawback turn-logs (NDJSON) to compare treatment vs passthrough arms. Produces report.md, report.csv, summary.json, manifest.json. Headline is billable input tokens reclaimed per turn (full-rate quota saved, no pricing) with a bootstrap 95% CI; per-turn $ and hit-rate are a demoted cost appendix, all stratified by inter-turn gap bucket. Use after the operator has collected counter-balanced windows.

zapgun-ai e05c8d9 3.5 KB Updated Sep 17, 2026

File contents

clawback benchmark analyzer

Analyze per-turn NDJSON logs emitted by a running clawback proxy.

Usage

node benchmark/bin/analyze.js --out runs/report-$(date +%s) <turn-log-path>...

Inputs may be individual .ndjson files or directories containing them. Each record's arm field determines whether it counts as treatment, passthrough, or treatment-ping (keep-alive overhead).

Inputs

Turn-logs are produced by clawback when started with --turn-log <path>:

clawback --turn-log ./runs/turns.ndjson ...          # treatment
clawback --turn-log ./runs/turns.ndjson --passthrough ...   # baseline

Both arms write to the same file — arm label is embedded per record.

Outputs

Written to the --out directory:

report.md — leads with the billable input tokens reclaimed vs passthrough headline (per-turn rate + bootstrap CI; full-rate quota clawback keeps off your bill — no pricing). The $/turn cost detail is a demoted appendix below it. Then two diagnostic sections:
- Prefix fragmentation — distinct clawback SESSION KEYs seen per stable system prefix, per knobProfile. 1 = one logical context maps to one Anthropic cache key (ideal); >1 (flagged ⚠️) means the same context was split across keys, each cold-starting Anthropic's cache — strip-ephemeral collapses this toward 1. This is the headline finding on hot loops where passthrough fragments but the stack does not.
- Keep-alive ping coverage — share of turns preceded by ≥1 treatment-ping during the gap, plus mean pings/turn. High coverage on a >5-min gap bucket alongside a high hit rate is keep-alive keeping the cache warm (the 15-min warmth test). Renders only when the log carries ping records.
report.csv — turn-level rows for downstream plotting. Includes pingsSincePrevTurn and msSinceLastPing per turn (ping coverage), alongside gapMs/gapBucket and the priced usd_estimate.
summary.json — machine-readable aggregates; the top-level tokens block is the headline (baseline vs treatment billable totals + mean per-turn, reclaimedPerTurn + CI, pctLessPerTurn, reclaimedTotalIsProjected). The prefixFragmentation array carries the per-prefix key counts, and each arm stratum carries meanPingsSincePrevTurn + pingCoverageShare.
manifest.json — input file list, pricing hash, clawback version, wall-clock coverage per arm.

All gap, ping-coverage, and fragmentation metrics are derived analyzer-side from the existing turn-log fields (ts, sessionKey, systemStableKey, and the treatment-ping records) — no proxy change is needed to collect them.

Reproducibility

Bootstrap CIs use a seeded PRNG; rerunning on the same inputs yields byte-identical reports (ignoring generatedAt timestamps).

When to use

After an operator has run treatment/passthrough windows (≥ 200 turns per arm, ideally across multiple days).
To generate the report.md that's committed alongside the study.
To regenerate historical reports when pricing updates (use the pricing-hash field in manifest.json to detect drift).

---name: benchdescription: Analyze clawback turn-logs (NDJSON) to compare treatment vs passthrough arms. Produces report.md, report.csv, summary.json, manifest.json. Headline is billable input tokens reclaimed per turn (full-rate quota saved, no pricing) with a bootstrap 95% CI; per-turn $ and hit-rate are a demoted cost appendix, all stratified by inter-turn gap bucket. Use after the operator has collected counter-balanced windows.---# clawback benchmark analyzerAnalyze per-turn NDJSON logs emitted by a running clawback proxy.## Usage```bashnode benchmark/bin/analyze.js --out runs/report-$(date +%s) <turn-log-path>...```Inputs may be individual `.ndjson` files or directories containing them.Each record's `arm` field determines whether it counts as treatment,passthrough, or `treatment-ping` (keep-alive overhead).## InputsTurn-logs are produced by clawback when started with `--turn-log <path>`:```bashclawback --turn-log ./runs/turns.ndjson ... # treatmentclawback --turn-log ./runs/turns.ndjson --passthrough ... # baseline```Both arms write to the same file — arm label is embedded per record.## OutputsWritten to the `--out` directory:- `report.md` — leads with the **billable input tokens reclaimed vs passthrough** headline (per-turn rate + bootstrap CI; full-rate quota clawback keeps off your bill — no pricing). The `$`/turn cost detail is a demoted appendix below it. Then two diagnostic sections: - **Prefix fragmentation** — distinct clawback SESSION KEYs seen per stable system prefix, per knobProfile. `1` = one logical context maps to one Anthropic cache key (ideal); `>1` (flagged ⚠️) means the same context was split across keys, each cold-starting Anthropic's cache — strip-ephemeral collapses this toward 1. This is the headline finding on hot loops where passthrough fragments but the stack does not. - **Keep-alive ping coverage** — share of turns preceded by ≥1 `treatment-ping` during the gap, plus mean pings/turn. High coverage on a >5-min gap bucket alongside a high hit rate is keep-alive keeping the cache warm (the 15-min warmth test). Renders only when the log carries ping records.- `report.csv` — turn-level rows for downstream plotting. Includes `pingsSincePrevTurn` and `msSinceLastPing` per turn (ping coverage), alongside `gapMs`/`gapBucket` and the priced `usd_estimate`.- `summary.json` — machine-readable aggregates; the top-level `tokens` block is the headline (baseline vs treatment billable totals + mean per-turn, `reclaimedPerTurn` + CI, `pctLessPerTurn`, `reclaimedTotalIsProjected`). The `prefixFragmentation` array carries the per-prefix key counts, and each arm stratum carries `meanPingsSincePrevTurn` + `pingCoverageShare`.- `manifest.json` — input file list, pricing hash, clawback version, wall-clock coverage per arm.All gap, ping-coverage, and fragmentation metrics are derivedanalyzer-side from the existing turn-log fields (`ts`, `sessionKey`,`systemStableKey`, and the `treatment-ping` records) — no proxy changeis needed to collect them.## ReproducibilityBootstrap CIs use a seeded PRNG; rerunning on the same inputs yieldsbyte-identical reports (ignoring `generatedAt` timestamps).## When to use- After an operator has run treatment/passthrough windows (≥ 200 turns per arm, ideally across multiple days).- To generate the `report.md` that's committed alongside the study.- To regenerate historical reports when pricing updates (use the pricing-hash field in `manifest.json` to detect drift).

Frequently asked questions