Ab

/tokenwise:ab — A/B test a task across model tiers

Run the same task at multiple tiers and compare outputs.

Parse $ARGUMENTS

Expected form: <task description> [--tiers haiku,sonnet,opus,fable]

• The task description is everything before the first -- flag (or the whole string if no flags)
• --tiers haiku,sonnet is the default (skip Opus and Fable by default — Opus is the baseline, Fable is the priciest lane)
• --tiers haiku,sonnet,opus runs the three cheaper tiers
• --tiers ...,fable adds Fable — useful for calibrating the Planning lane (e.g. checking whether a task class really needs it, or whether an override pinning it to sonnet/opus is good enough). Since Fable costs 2× Opus, the cost-confirm step below should flag it

If $ARGUMENTS is empty, ask the user:

What task should I A/B test? Provide a task description (e.g., "rename getCwd to getCurrentWorkingDirectory across the codebase").

Steps

1. Confirm cost upfront:

   A/B test will run this task <N> times (once per tier).
   Estimated cost: $<rough estimate based on task size>.
   Proceed? [Y/n]
   

   Estimate by treating the task as ~10k input + ~1k output per tier and summing. If fable is in --tiers, add one line before the prompt: Note: fable is ~2x Opus's rate and will dominate this estimate.

2. For each tier:

   • Spawn a Task at that tier: Task(description: <task>, subagent_type: "general-purpose", model: <tier>, prompt: <task>)
   • Capture: stdout, input_tokens, output_tokens, duration_ms, errors
   • If the Task tool's model: param is silently overridden (Anthropic Issue #47488), warn user and abort with:

     Cannot A/B test on this Claude Code build — subagent model routing is not honored. See /tokenwise:install probe results.

3. Compute diffs:

   • Structural: line count, character count, token overlap (Jaccard on word sets)
   • File-list diff: if outputs mention modified files, compare the lists
   • Semantic: spawn one more Task at Opus tier with the prompt:

     Compare these N outputs for the task "". Score each from 1-10 on (a) completeness, (b) correctness, (c) clarity. Return a JSON object: {"<tier>": {"completeness": N, "correctness": N, "clarity": N, "overall": N, "notes": "..."}}. Be honest — if two outputs are equivalent, give them the same score.

4. Compute cost per tier:

   • Use the same pricing as /tokenwise:report:
     • Fable 5: $10 input / $50 output per 1M tokens
     • Opus 4.7: $5 input / $25 output per 1M tokens
     • Sonnet 4.6: $3 / $15
     • Haiku 4.5: $1 / $5
   • cost = (input/1M)*input_price + (output/1M)*output_price

5. Write report to ./tokenwise-ab-<YYYYMMDD-HHMMSS>.md:

# A/B Test — <first 80 chars of task>
Run: <ISO 8601 timestamp>
Task: "<full task>"
Tiers tested: <list>

## Tier comparison

| Tier   | Tokens (in/out)  | Cost     | Duration | Quality |
|--------|------------------|----------|----------|---------|
| Haiku  | <in> / <out>     | $<cost>  | <ms>     | <q>/10  |
| Sonnet | ...              | ...      | ...      | ...     |

## Semantic scores

(table with completeness/correctness/clarity per tier, plus notes from the judge)

## Recommendation

(One paragraph: which tier is sufficient for this task class, what override rule to add to CLAUDE.md if any.)

## Output diffs

(For each tier, include the full output verbatim, or a truncated version with full output in a fenced block.)

6. Also log the A/B run to .tokenwise/log.ndjson with task_class: "ab-test" so it doesn't pollute regular routing stats.

7. Print summary to stdout:

   A/B test complete.
   Report: tokenwise-ab-<ts>.md
   Recommendation: <one-sentence recommendation>
   

Caveats to mention to the user if relevant

• Non-deterministic tasks (e.g., "suggest improvements") will have varying outputs across runs. The A/B is advisory in those cases.
• Multi-step tasks that need user input mid-flow cannot be A/B'd cleanly. Decompose first.
• A/B costs are not part of normal routing stats — they're tagged separately.

Tools

Task (for spawning per-tier runs and the semantic judge), Read, Write, Bash.