Skill Trigger Logging System

Skill Trigger Logging System

Research document for testing skill invocation accuracy via Claude Code hooks.

Problem Statement

Skills are "model-invoked" - Claude decides internally when to activate them. There's no direct PreSkillLoad or SkillActivation hook event. This makes it difficult to:

• Verify correct skills activate for given prompts
• Detect trigger phrase regressions after updates
• Identify skill overlap/competition
• Measure skill invocation accuracy

Solution: Indirect Observation via Hooks

Claude Code hooks can observe file reads and user prompts. By correlating these, we can reconstruct which skills were invoked for which prompts.

Hook Configuration

Add to ~/.claude/settings.json:

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ)|PROMPT|$(pwd)|$(jq -r '.prompt' | tr '\\n' ' ' | cut -c1-200)\" >> ~/.claude/logs/skill-triggers.log"
          }
        ]
      }
    ],
    "PreToolUse": [
      {
        "matcher": "Read",
        "hooks": [
          {
            "type": "command",
            "command": "jq -r '.tool_input.file_path' | grep -E '(SKILL\\.md|/references/)' | while read f; do echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ)|READ|$(pwd)|$f\"; done >> ~/.claude/logs/skill-triggers.log 2>/dev/null || true"
          }
        ]
      }
    ]
  }
}

Log Format

TIMESTAMP|EVENT_TYPE|WORKING_DIR|CONTENT

Example output:

2024-12-21T14:30:00Z|PROMPT|~/Projects/myapp|debug this null pointer exception
2024-12-21T14:30:01Z|READ|~/Projects/myapp|~/Projects/claude-skills/skills/debugging-wizard/SKILL.md
2024-12-21T14:30:02Z|READ|~/Projects/myapp|~/Projects/claude-skills/skills/debugging-wizard/references/common-patterns.md

Setup Instructions

1. Create Log Directory

mkdir -p ~/.claude/logs

2. Add Hooks to Settings

# Edit or create settings file
nano ~/.claude/settings.json

Paste the hook configuration above.

3. Verify Hooks Are Active

# Start a new Claude Code session and check logs
tail -f ~/.claude/logs/skill-triggers.log

Analysis Scripts

View Recent Activity

# Last 20 skill-related events
tail -20 ~/.claude/logs/skill-triggers.log | column -t -s'|'

Correlate Prompts to Skills

#!/bin/bash
# ~/.claude/scripts/analyze-triggers.sh

LOG=~/.claude/logs/skill-triggers.log

echo "=== Prompt → Skill Mapping ==="
echo ""

# Group reads that follow each prompt
awk -F'|' '
  /\|PROMPT\|/ {
    if (prompt != "") print prompt " → " skills
    prompt = $4
    skills = ""
  }
  /\|READ\|.*SKILL\.md/ {
    match($4, /skills\/([^\/]+)\//, arr)
    if (arr[1] != "") skills = skills " " arr[1]
  }
  END { if (prompt != "") print prompt " → " skills }
' "$LOG"

Filter by Project

# Only show triggers from a specific project
grep "/Projects/claude-skills|" ~/.claude/logs/skill-triggers.log

Skill Invocation Frequency

# Count which skills are invoked most often
grep "|READ|" ~/.claude/logs/skill-triggers.log \
  | grep "SKILL.md" \
  | sed 's|.*/skills/||; s|/SKILL.md||' \
  | sort | uniq -c | sort -rn

Test Harness

Test Case Format

Create ~/.claude/tests/skill-trigger-tests.json:

[
  {
    "prompt": "debug this null pointer exception",
    "expected_skills": ["debugging-wizard"],
    "not_expected": ["test-master"]
  },
  {
    "prompt": "optimize this PostgreSQL query",
    "expected_skills": ["postgres-pro", "database-optimizer"],
    "not_expected": ["sql-pro"]
  },
  {
    "prompt": "review this NestJS controller for security issues",
    "expected_skills": ["nestjs-expert", "secure-code-guardian"],
    "not_expected": ["django-expert"]
  }
]

Run Tests and Compare

#!/bin/bash
# ~/.claude/scripts/test-skill-triggers.sh

TESTS=~/.claude/tests/skill-trigger-tests.json
LOG=~/.claude/logs/skill-triggers.log
RESULTS=~/.claude/logs/test-results-$(date +%Y%m%d).log

echo "Skill Trigger Test Results - $(date)" > "$RESULTS"
echo "=================================" >> "$RESULTS"

jq -c '.[]' "$TESTS" | while read test; do
  prompt=$(echo "$test" | jq -r '.prompt')
  expected=$(echo "$test" | jq -r '.expected_skills[]' | tr '\n' ' ')

  # Find skills invoked after this prompt
  actual=$(grep -A5 "$prompt" "$LOG" | grep "SKILL.md" | sed 's|.*/skills/||; s|/SKILL.md||' | tr '\n' ' ')

  if [[ "$actual" == *"$expected"* ]]; then
    echo "[PASS] $prompt → $actual" >> "$RESULTS"
  else
    echo "[FAIL] $prompt" >> "$RESULTS"
    echo "       Expected: $expected" >> "$RESULTS"
    echo "       Actual:   $actual" >> "$RESULTS"
  fi
done

cat "$RESULTS"

Insights From Working Directory

The pwd field enables:

1. Project-specific analysis - Which skills activate in which codebases?
2. Context correlation - Does the same prompt trigger different skills in different projects?
3. Skill relevance - Are certain skills only useful in certain project types?

Example: Project-Skill Heatmap

# Which skills are used in which projects?
awk -F'|' '/\|READ\|.*SKILL\.md/ {
  split($3, path, "/")
  project = path[5]  # Assumes /Users/user/Projects/name structure
  match($4, /skills\/([^\/]+)\//, skill)
  print project, skill[1]
}' ~/.claude/logs/skill-triggers.log | sort | uniq -c | sort -rn

Limitations

Future Enhancements

1. Structured logging - Output JSON for easier parsing
2. Dashboard - Visual display of skill invocation patterns
3. Regression CI - Run test suite on skill changes
4. Overlap detection - Alert when multiple skills compete for same prompt

References

• Claude Code hooks documentation
• obra/superpowers - Verification discipline patterns
• claude-code-plugins-plus-skills - Verification script patterns