Prompt Refine Audit
Audit and tune the UserPromptSubmit prompt-refine hook.
Files
- Log:
~/.claude/logs/prompt-refine.log
- Hook:
~/.claude/hooks/prompt-refine.sh (active)
- Source:
~/dev/claude-config/hooks/prompt-refine.sh (repo)
Workflow
Step 1: Read the log
Read ~/.claude/logs/prompt-refine.log. If it doesn't exist or is empty, tell the user no prompts have been evaluated yet.
Step 2: Summarize
Present a summary table:
| Metric |
Value |
| Total prompts evaluated |
count of --- entries |
| Triggered (YES) |
count |
| Not triggered (no) |
count |
| Skipped (aspell missing) |
count |
| Trigger rate |
percentage |
| Current threshold |
read from hook script (the 0.15 value on the NEEDS_REFINE line) |
Step 3: Flag issues
Scan for these problems:
False positives (triggered but shouldn't have):
- Triggered entries where most "misspelled" words are proper nouns, tech terms, or common informal words (e.g.,
dont, bc, im, hm, ok, wanna, gonna, repo, env, config, api, cli, pr, ui, ux, db, ts, js, css, html, json, yaml, npm, pnpm, vps, ssh, mcp, lsp)
- Entries where the prompt is clearly understandable despite typo count
False negatives (didn't trigger but should have):
- Non-triggered entries where the prompt text has obvious garbled words or unclear intent
aspell gaps:
- Any
SKIP (aspell not found) entries — means aspell isn't installed on that machine
Step 4: Recommend fixes
Based on the analysis, suggest concrete changes:
- Threshold too low (too many false positives): suggest raising the
0.15 value
- Threshold too high (missing messy prompts): suggest lowering it
- Tech terms triggering aspell: suggest adding a personal wordlist at
~/.aspell.en.pws with commonly flagged terms. Generate the wordlist content based on the most frequently flagged non-typo words in the log.
- aspell missing: provide install command for the detected platform:
- Linux:
sudo apt install aspell aspell-en or sudo dnf install aspell aspell-en
- macOS:
brew install aspell
- Windows:
pacman -S mingw-w64-x86_64-aspell (MSYS2) or note Git Bash limitation
Step 5: Apply fixes (only if user approves)
If the user wants changes applied:
- Edit the threshold in
~/dev/claude-config/hooks/prompt-refine.sh (the source)
- Write/update
~/.aspell.en.pws if adding a personal wordlist (format below)
- Copy the updated hook to
~/.claude/hooks/prompt-refine.sh
- Confirm changes with a quick test: run the hook against a sample messy and clean prompt
aspell personal wordlist format (~/.aspell.en.pws):
personal_ws-1.1 en 0
dont
bc
im
api
cli
Rules
- Never delete or truncate the log without asking
- Show actual log entries when discussing false positives/negatives so the user can judge
- Always edit the source file in
~/dev/claude-config/ first, then copy to ~/.claude/
- When changing the threshold, explain the tradeoff (sensitivity vs noise)
- If the log is very large (>500 entries), summarize and show only the most recent 20 + any flagged issues
1---2name: prompt-refine3description: Audit the prompt refinement hook log — review trigger history, analyze false positives/negatives, and tune heuristics4---56# Prompt Refine Audit78Audit and tune the `UserPromptSubmit` prompt-refine hook.910## Files1112- **Log:** `~/.claude/logs/prompt-refine.log`13- **Hook:** `~/.claude/hooks/prompt-refine.sh` (active)14- **Source:** `~/dev/claude-config/hooks/prompt-refine.sh` (repo)1516## Workflow1718### Step 1: Read the log1920Read `~/.claude/logs/prompt-refine.log`. If it doesn't exist or is empty, tell the user no prompts have been evaluated yet.2122### Step 2: Summarize2324Present a summary table:2526| Metric | Value |27|--------|-------|28| Total prompts evaluated | count of `---` entries |29| Triggered (YES) | count |30| Not triggered (no) | count |31| Skipped (aspell missing) | count |32| Trigger rate | percentage |33| Current threshold | read from hook script (the `0.15` value on the `NEEDS_REFINE` line) |3435### Step 3: Flag issues3637Scan for these problems:3839**False positives** (triggered but shouldn't have):40- Triggered entries where most "misspelled" words are proper nouns, tech terms, or common informal words (e.g., `dont`, `bc`, `im`, `hm`, `ok`, `wanna`, `gonna`, `repo`, `env`, `config`, `api`, `cli`, `pr`, `ui`, `ux`, `db`, `ts`, `js`, `css`, `html`, `json`, `yaml`, `npm`, `pnpm`, `vps`, `ssh`, `mcp`, `lsp`)41- Entries where the prompt is clearly understandable despite typo count4243**False negatives** (didn't trigger but should have):44- Non-triggered entries where the prompt text has obvious garbled words or unclear intent4546**aspell gaps:**47- Any `SKIP (aspell not found)` entries — means aspell isn't installed on that machine4849### Step 4: Recommend fixes5051Based on the analysis, suggest concrete changes:5253- **Threshold too low (too many false positives):** suggest raising the `0.15` value54- **Threshold too high (missing messy prompts):** suggest lowering it55- **Tech terms triggering aspell:** suggest adding a personal wordlist at `~/.aspell.en.pws` with commonly flagged terms. Generate the wordlist content based on the most frequently flagged non-typo words in the log.56- **aspell missing:** provide install command for the detected platform:57 - Linux: `sudo apt install aspell aspell-en` or `sudo dnf install aspell aspell-en`58 - macOS: `brew install aspell`59 - Windows: `pacman -S mingw-w64-x86_64-aspell` (MSYS2) or note Git Bash limitation6061### Step 5: Apply fixes (only if user approves)6263If the user wants changes applied:64651. Edit the threshold in `~/dev/claude-config/hooks/prompt-refine.sh` (the source)662. Write/update `~/.aspell.en.pws` if adding a personal wordlist (format below)673. Copy the updated hook to `~/.claude/hooks/prompt-refine.sh`684. Confirm changes with a quick test: run the hook against a sample messy and clean prompt6970**aspell personal wordlist format** (`~/.aspell.en.pws`):71```72personal_ws-1.1 en 073dont74bc75im76api77cli78```7980## Rules8182- Never delete or truncate the log without asking83- Show actual log entries when discussing false positives/negatives so the user can judge84- Always edit the source file in `~/dev/claude-config/` first, then copy to `~/.claude/`85- When changing the threshold, explain the tradeoff (sensitivity vs noise)86- If the log is very large (>500 entries), summarize and show only the most recent 20 + any flagged issues
codywilliamson