dtsong mcs-analysis-toolkit
Overview
Six first-principles recipes for proving a claim about this repo instead of asserting it: derive it by hand from source, confirm with read-only tools, and (for review claims) build a discriminating experiment that refutes a wrong belief. Each recipe ends in a worked example reproduced 2026-07-02, with a command to re-verify it.
When to use, and when NOT to
Use when: sizing a suite's worst-case context load before shipping, auditing whether a description will actually trigger, verifying a hook truly blocks before depending on it, sanity-checking a multi-agent cost estimate, deciding whether a counter is trustworthy enough to cite, or reviewing a "fix" before merge.
Do NOT use for: routine tool runs once you already trust them, that is mcs-diagnostics-and-tooling (inventory counts, budget reports, eval runs, exit-code tables). What counts as sufficient acceptance evidence is mcs-validation-and-qa. How a hunch becomes an accepted result over time is mcs-research-methodology.
Recipe 1: Token / context-load accounting
Formula: tokens = ceil(words * 1.33) (TOKEN_RATIO, pipeline/hooks/_utils.py:13). Worst-case simultaneous load, per specialist: coordinator + specialist + its single largest reference file (only one specialist and one reference load at runtime). Hard ceiling: 5,500 tokens (pipeline/config/budgets.json: max_simultaneous_tokens), overridable per suite. Root CLAUDE.md:57's "5,000" is a stale outlier; enforcement uses 5,500.
Procedure:
wc -wthe coordinator, specialist, and largestreferences/file; apply the formula to each, round up, sum.- Check
budgets.json.overridesfor a key equal to the exact relative path the hook computes, not a guessed path. - Confirm:
python3 pipeline/hooks/check_context_load.py <coordinator SKILL.md>(hard hook).
Worked example (skills/soc-security/skills/threat-model-skill): 468 + 2828 + 1275 words → 623 + 3762 + 1696 = 6081 tokens, 581 over ceiling. A 6,100-token override exists in budgets.json but never matches the hook's computed key (nested-suite path mismatch). Full evidence: references/worked-examples.md §1.
Recipe 2: Trigger-reliability analysis
Formula (pipeline/specs/SKILL-TRIGGER-RELIABILITY-SPEC.md §1.2): activation directive + positive triggers + 2-5 quoted phrases + a negative boundary naming the alternative. Target 40-80 words (§1.5); under 20 "almost always" under-activates. The only hard floor is 10 words (check_frontmatter.py: MIN_DESCRIPTION_WORDS = 10), so 10-40 words passes silently.
Procedure:
- Count the description's words; score against the formula (activation verb? quoted phrases? negative boundary?).
- Sum always-loaded description cost library-wide:
python3 pipeline/scripts/skill-audit.py --json(total_description_tokens,over_threshold). - Design positive/negative eval pairs using the as-built shape from an existing department (
{id, input, expected_skill, type, description}), not the spec's fuller markdown template (unimplemented).
Worked example: skill-audit.py --json over skills/: 123 skills, 5,966 description tokens, 0 over threshold. Under-target cluster: git-workflows (10-word description) and prompt-wizard (11 words) clear the floor but sit at a quarter of target, no quoted phrases or negative boundary; git-workflows is a coordinator (13 nested specialists) that per §1.6 should "cast a wide net" and does not. Full evidence: references/worked-examples.md §2.
Recipe 3: Hook-blocking verification experiment
Rule: a hook wired into settings.json is not proof it fires or blocks. Prove, in order: its event+exit-code contract (PreToolUse + exit 2 denies the tool, stderr feeds back as the reason; PostToolUse + any non-zero exit is non-blocking, whatever the script prints), which stream carries its message, then a live-fire test plus a control payload that should pass.
Procedure: Read the settings.json registration (event, matcher) and the script's stdin contract. echo '<crafted JSON>' | bash <hook.sh>, check exit code and stream. Repeat with a control payload, confirm exit 0. Trust the hook only after both pass.
Precondition (checked 2026-07-05): the live-fire result below needs a fresh (<ACCEPTANCE_GATE_STALE_HOURS, default 72; see mcs-change-control) contract with an unresolved | pending |/| failed | row. A stale or fully-resolved contract makes the payload exit 0 with no output, that is the staleness guard working, not the hook failing to block. Check the newest contract's mtime first (ls -lt .claude/council/sessions/*/acceptance-contract.md | head -1).
Worked example (hooks/acceptance-gate.sh): with a fresh (<72h) contract carrying a pending/failed row, a TaskUpdate → completed payload exits 2, message on stderr; a status: in_progress control exits 0. Same experiment would have caught the gate's 4-month decorative period: before 605112d (2026-06-22) the identical script was PostToolUse, printed to stdout, exited 1 (non-blocking), so completion always went through from creation (299a264, 2026-02-16). Full transcript: references/worked-examples.md §3.
Recipe 4: Cost/latency estimation for multi-agent runs
Model: the engine's table (commands/_council-engine.md §Cost Profiles & Model Routing) gives a 5-agent-baseline token range per mode × profile, scaled by selected_agents / 5: brainstorm 10-15K flat; quick 20-35K/30-50K/45-75K; standard 80-120K/120-180K/180-270K; deep = standard + 50-150K audit, for lean/balanced/max.
Procedure: Read mode + profile off the session; take the baseline range; multiply by agents_selected / 5. Label the result an estimate: do not convert to dollars from memory, the engine points to a README "Cost guide" for $/MTok that does not exist in README.md (verified 2026-07-02).
Worked example (session claude-config-model-optimization-20260702-0003): profile max, 7 agents, standard-mode artifacts. Baseline (5-agent, standard, max) = 180-270K tokens, scaled by 7/5 = 252K-378K tokens. Estimate, not a committed actual: no telemetry here captures real per-session totals as of 2026-07-02. Full derivation: references/worked-examples.md §4.
Recipe 5: Instrument-trust audit
Rule: before citing any counter, trace its write path: who writes it under what condition, prose or enforced code, does it land in a commit, can it be contaminated by something outside its scope.
Procedure: Grep for the field name in the code/prose that writes it; classify the step (prose vs. enforced code). git log -- <file>, diff HEAD vs. working tree, an uncommitted write is invisible to everyone else. Check for contamination (a machine-wide symlink shared across projects?). Trust the counter only after one full write-through-commit cycle by SHA.
Worked example (skills/council/registry.json): write step is prose only (commands/_council-engine.md:672-677), no code enforces it. Last commit before today: 729adcd (2026-04-24, all uses: 0); next: dc44611 (2026-07-02, 18 uses), a 69-day gap where the write never got committed. One session slug has no matching directory in this repo, because ~/.claude/skills symlinks here, so every workspace on this machine shares one file with no workspace tag. Cite it as "at least these sessions ran," never full usage history. Full evidence: references/worked-examples.md §5.
Recipe 6: Refute-first review
Rule: for any "fix," before approving it: state the exact wrong behavior it prevents as input → wrong output; build one discriminating experiment whose result differs pre-fix vs. post-fix; run it. Reading the diff and agreeing with its narrative is not verification.
Procedure: State the failure scenario in one sentence. Isolate the code path (inject a fake dependency/transport/stdin rather than the real network or tool). Run the same experiment against git show <fix_commit>^:<file> (pre-fix) and the current file (post-fix). Confirm the two disagree on a caller-visible field, not just internal structure.
Worked example (commit 898174d, mcp/openrouter/openrouter_client.py): claimed failure was that urlopen raises HTTPError on 4xx/5xx, misclassified as transport failure. Monkeypatching urlopen to raise HTTPError(code=429, ...) and reading consult(...)["error_kind"] gives "http" on the current file, "transport" on 898174d^, both reproduced directly: a caller-visible field changes, so the fix is genuinely load-bearing. PR #55 has zero GitHub reviews and comments: the review that mattered was this self-run experiment. Full script: references/worked-examples.md §6.
Gotchas
- WRONG: "the override is in budgets.json, covered." RIGHT: the key must equal the hook's computed relpath;
skills/<suite>/skills/<specialist>almost never matches a flatskills/<specialist>key (Recipe 1). classify_file()treats any path containing askills/segment as "specialist," even repo-rootskills/itself, so nearly every top-levelskills/*/SKILL.mdgets the 1,500-word target regardless of intent. Only a siblingskills/subdirectory makes it "coordinator" (git-workflows qualifies, prompt-wizard doesn't).- Passing
check_frontmatter.pyproves only the 10-word floor, not that a description will trigger; trigger reliability has no hard hook. PostToolUse+ non-zero exit is silently non-blocking; onlyPreToolUse+ exit 2 denies a call. Confusing the two shipped a 4-month decorative hook (Recipe 3).- A cost/token estimate from the engine's table is a range, not a telemetry-backed actual; nothing here records real per-session totals as of 2026-07-02.
- A counter present and non-zero may not be committed, or scoped to the project you're standing in; check both (Recipe 5).
- "Passed review" can mean zero human reviewers plus a local Claude review loop (
gh pr view <n> --json reviews,comments); a merged-without-review fix still needs your own experiment. skill-audit.py,budget-report.py,context-load-analysis.pyare read-only, never fail a commit; a clean report is not a passing hard gate.
Provenance and maintenance
Last verified: 2026-07-02, against a live ship run (session claude-config-model-optimization-20260702-0003, commits through dc44611). Volatile numbers (registry counts, token totals, PR/issue state) will drift as work continues; re-verify:
- Token ratio:
grep -n TOKEN_RATIO pipeline/hooks/_utils.py - Ceiling + override mismatch:
python3 pipeline/hooks/check_context_load.py skills/soc-security/SKILL.md - Description formula/word target:
sed -n '100,176p' pipeline/specs/SKILL-TRIGGER-RELIABILITY-SPEC.md - Library-wide description cost:
python3 pipeline/scripts/skill-audit.py --json - Hard description floor:
grep -n MIN_DESCRIPTION_WORDS pipeline/hooks/check_frontmatter.py - Hook live-fire (requires a fresh, unresolved contract first, see Recipe 3 precondition):
echo '{"tool_name":"TaskUpdate","tool_input":{"status":"completed"}}' | bash hooks/acceptance-gate.sh; echo $? - Cost table:
grep -n "Estimated session cost" -A 10 commands/_council-engine.md - registry.json commit history:
git log --oneline -- skills/council/registry.json - Discriminating-experiment source:
git show 898174d -- mcp/openrouter/openrouter_client.py - PR review count:
gh pr view 55 --json reviews,comments