Cold Tester

/cold-tester — test your docs with an AI that has zero context

Why this exists

You can't fairly test your own documentation. You already know what every step means, so you fill the gaps automatically and the doc reads fine — to you. A real newcomer doesn't have that context, and neither should the tester.

This skill runs the doc through an isolated model (a separate process, blank slate, given only the document + a persona + a kickoff). It tries to actually do the task, gate by gate, and every place it stalls, guesses, or writes "UNCLEAR — need to confirm" is a place your doc is missing something.

Your main agent (this session) does not play the tester — it already knows too much. It plays the approver: it reads each submission the cold tester produces and rules pass / conditional / reject, then synthesizes which doc sections need fixing.

How it works

Your main agent (orchestrator + approver)
  → passes the doc + persona + kickoff to an isolated model (the cold tester)
isolated model (zero context)
  → attempts each gate using ONLY the doc, flags every gap
  → returns submissions
Your main agent
  → rules pass / conditional / reject per gate, writes the report

Setup

Any OpenAI-compatible endpoint. Use a cheap model — the tester should be naive, not smart.

pip install openai pyyaml
export COLD_TESTER_API_KEY=sk-...                       # required
export COLD_TESTER_BASE_URL=https://api.openai.com/v1   # optional (default OpenAI)
export COLD_TESTER_MODEL=gpt-4o-mini                    # optional (or set in config)

Works with OpenAI, DeepSeek, Together, OpenRouter, or any local OpenAI-compatible server — just point COLD_TESTER_BASE_URL at it.

Run it

The script stays at its install path; --base points it at the repo under test, so you never copy it into the target project.

Config file (recommended):

python3 ~/.claude/skills/cold-tester/cold_tester.py --config test.yaml --base /path/to/repo

Or inline:

python3 ~/.claude/skills/cold-tester/cold_tester.py \
  --base /path/to/repo \
  --docs README.md,CONTRIBUTING.md \
  --persona "First-time open-source contributor, has never seen this repo" \
  --task "Open your first pull request" \
  --kickoff "Set up the project locally and open a PR fixing a typo." \
  --gates "setup,find-an-issue,open-pr" \
  --output results/contributing-test.md

The cold tester reasons through the docs as a newcomer would; it doesn't execute commands or touch the filesystem. The gaps it reports are real, the execution is simulated — that's the reading-comprehension test you want.

Config (YAML)

test_name: "CONTRIBUTING guide — first PR"
model: gpt-4o-mini          # optional; overrides COLD_TESTER_MODEL
persona: |
  You are a competent developer who has never contributed to this project.
  You know general Git/GitHub but nothing about this repo's conventions.
task: "Open your first pull request"
kickoff: |
  Set up the project locally and open a small PR.
documents:
  - path: README.md
  - path: CONTRIBUTING.md
gates:
  - id: setup
    name: "Local setup"
    approver: "you"
    instruction: "Follow the setup steps and report exactly where you got stuck."
  - id: open-pr
    name: "Open the PR"
    approver: "you"
    instruction: "Describe the exact commands/clicks you'd run to open the PR."

The approver's job (your main agent)

After the script writes the report, fill each gate's verdict:

• pass — a newcomer could complete this gate from the doc alone
• conditional — completable but with a guess/ambiguity worth fixing
• reject — the doc doesn't give enough to proceed

Then write the Summary: which sections caused stalls, and the specific doc edits that would fix them. A gate the cold tester marked "UNCLEAR" but you'd have known how to do is the highest-value finding — that's the curse-of-knowledge gap made visible.

Good things to cold-read

Onboarding guides, CONTRIBUTING / setup docs, runbooks and SOPs, API quickstarts, tutorials, any "follow these steps" document where the author's context silently fills the gaps.