Setup Checks

Set Up AI Checks

You are helping a developer set up AI Checks — markdown files that define code quality standards. Each check runs as a full AI agent on every PR — not just reading the diff, but able to read files, run commands, and use a browser. If it finds something, it fails the PR with a suggested fix. Otherwise, it passes silently.

Complete all steps below in order.

Step 1: Orient

Welcome the user. Keep it brief — one or two sentences:

We're going to set up AI checks for your repo. I'll explore your codebase, write checks tailored to your project, and get you set up to run them.

Then create a todo list for the session:

1. Explore codebase & write checks
2. Try a check
3. Set up CI (optional)
4. Commit & PR

Mark task 1 in progress.

Step 2: Git Setup

1. Verify this is a git repo: git rev-parse --git-dir 2>/dev/null — stop if not.
2. Check for uncommitted changes: git status --porcelain

If dirty, ask:

You have uncommitted changes. I'd like to stash them and create a new branch. OK?

If approved:

git rev-parse --abbrev-ref HEAD # save original branch
git stash push -m "Stashing before checks setup"
git checkout -b add-checks

Remember the original branch and whether you stashed — you'll restore later.

Step 3: Explore & Write Checks

Speed target: under 60 seconds. Stay shallow — surface patterns, don't read entire files.

Run these in parallel where possible:

Codebase scan

• Project structure, languages, frameworks, build system
• Linters, formatters, type checkers already in use
• Testing framework & patterns
• CI/CD workflows
• Contributing guidelines (CONTRIBUTING.md, CLAUDE.md, AGENTS.md)

PR review history (via gh CLI, skip if gh is not available)

gh pr list --state merged --limit 30 --json number,title
# For 3-5 recent PRs with review comments:
gh api repos/{owner}/{repo}/pulls/{number}/comments --jq '.[] | {body: .body, path: .path}'

Look for: what reviewers commonly flag, what linters don't catch, what areas get the most review attention.

Propose checks

Share 3-5 bullet findings, then propose exactly 3 checks via a multiselect question (all selected by default). Cardinal rule: never create a check for something already enforced by a deterministic tool (linter, type checker, CI step). Checks are for things that require judgment — slop, drifting conventions, security anti-patterns, poor taste. Each proposed check must be specific to this project and clearly valuable.

Write check files

Create .checks/ and write one .md file per check:

---
name: Human-Readable Check Name
description: One-sentence description.
---

# Check Name

## Context

WHY this check exists. Reference specific project patterns, configs, or review comments you found.

## What to Check

### 1. Category

Concrete instructions with specific file paths/patterns and GOOD/BAD code examples from this codebase.

### 2. Category

...

## Key Files

List the files most relevant to this check.

## Exclusions

Cases where this check does NOT apply.

Keep file names lowercase with hyphens (e.g., error-handling.md, api-conventions.md).

Update agent config

Add a section to the project's agent configuration file so checks become part of the workflow. Choose the right file based on what exists or what tool the user is using:

• CLAUDE.md — if using Claude Code
• AGENTS.md — if using a generic agent setup
• COPILOT.md — if using GitHub Copilot
• If none of these exist, create the most appropriate one.

Append a section like this:

## Checks

Before opening a pull request, run all checks in `.checks/` against your current diff and fix any failures before requesting review.

Mark task 1 complete.

Step 4: Try a Check

Mark task 2 in progress. Let the user see a check in action. Ask them which check they'd like to try:

Which check would you like to try? I'll run it against the codebase right now.

Present the list of checks as options. Once they pick one:

1. Read the selected check file from .checks/
2. Scan the codebase against the check's criteria — read the relevant source files, not just a diff
3. Evaluate what you find:
   • For each finding: state severity (Error / Warning / Info), reference the specific file and line, explain what's wrong and how to fix it
   • If everything looks good, report PASS
   • If there are findings, report FAIL and list them
4. Present the result — if it fails, offer to fix the issues

Mark task 2 complete.

Step 5: CI Setup (Optional)

Mark task 3 in progress. Ask the user:

Would you like to also run these checks automatically on every PR via GitHub Actions? I'll add the workflow file to the PR — you just need to add an API key as a repository secret.

If they say yes, create .github/workflows/checks.yml:

name: AI Checks

on:
  pull_request:
    types: [opened, synchronize, reopened]

concurrency:
  group: ai-checks-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  checks:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      checks: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/cache@v4
        with:
          path: ~/.npm
          key: claude-code-${{ runner.os }}

      - name: Install agent CLI
        run: npm install -g @anthropic-ai/claude-code

      - name: Get PR diff
        run: |
          gh pr diff ${{ github.event.pull_request.number }} > /tmp/pr.diff
          head -c 100000 /tmp/pr.diff > /tmp/pr-truncated.diff
        env:
          GH_TOKEN: ${{ github.token }}

      - name: Run all checks
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GH_TOKEN: ${{ github.token }}
        run: |
          REPORT_FILE=$(mktemp)
          echo "## AI Check Results" >> "$REPORT_FILE"
          echo "" >> "$REPORT_FILE"
          echo "| Check | Result |" >> "$REPORT_FILE"
          echo "|---|---|" >> "$REPORT_FILE"

          for check_file in .checks/*.md; do
            [ -f "$check_file" ] || continue
            name=$(basename "$check_file" .md)

            # Create check run in progress
            CHECK_RUN_ID=$(gh api repos/${{ github.repository }}/check-runs \
              -f name="check/$name" \
              -f head_sha=${{ github.event.pull_request.head.sha }} \
              -f status=in_progress \
              --jq '.id')

            # Run agent with the check prompt and PR diff
            OUTPUT=$(claude -p "$(cat <<PROMPT
          You are running a quality check on a pull request.

          ## Check
          $(cat "$check_file")

          ## PR Diff
          $(cat /tmp/pr-truncated.diff)

          ## Instructions
          Evaluate the changed files against this check.
          Only review changed lines. Do not flag pre-existing issues in unchanged code.
          Output valid JSON: {"verdict":"PASS" or "FAIL","reason":"...","suggestions":[{"file":"...","line":0,"fix":"..."}]}
          PROMPT
            )" --output-format json 2>&1) || true

            # Parse verdict — claude --output-format json wraps the response in an envelope
            # with the actual content in .result as a string (possibly markdown-fenced)
            INNER=$(echo "$OUTPUT" | jq -r '.result // empty' 2>/dev/null | sed 's/^```json//;s/^```//;s/```$//' | jq -r '.' 2>/dev/null || echo "$OUTPUT")
            VERDICT=$(echo "$INNER" | jq -r '.verdict // "ERROR"' 2>/dev/null || echo "ERROR")
            REASON=$(echo "$INNER" | jq -r '.reason // "No reason given"' 2>/dev/null || echo "Parse failure")

            # Extract suggestions for detail text
            SUGGESTIONS=$(echo "$INNER" | jq -r '.suggestions // [] | .[] | "- **\(.file):\(.line)** — \(.fix)"' 2>/dev/null || echo "")

            # Build detail markdown
            DETAIL="## $name\n\n**Verdict:** $VERDICT\n\n### Reason\n\n$REASON"
            if [ -n "$SUGGESTIONS" ]; then
              DETAIL="$DETAIL\n\n### Suggestions\n\n$SUGGESTIONS"
            fi

            # Complete the check run
            CONCLUSION=$([ "$VERDICT" = "PASS" ] && echo "success" || echo "failure")
            gh api repos/${{ github.repository }}/check-runs/$CHECK_RUN_ID \
              -X PATCH \
              -f status=completed \
              -f conclusion="$CONCLUSION" \
              -f "output[title]=$(echo "$REASON" | head -c 140)" \
              -f "output[summary]=$REASON" \
              -f "output[text]=$(printf "$DETAIL")" > /dev/null

            # Append to report
            ICON=$([ "$VERDICT" = "PASS" ] && echo "✅" || echo "❌")
            echo "| $ICON $name | $REASON |" >> "$REPORT_FILE"

            # Save full output for debugging
            echo "$OUTPUT" > "/tmp/check-$name.json"
          done

          # Post single consolidated comment
          gh pr comment ${{ github.event.pull_request.number }} --body "$(cat "$REPORT_FILE")"

      - name: Upload check outputs
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: check-results
          path: /tmp/check-*.json
          retention-days: 7

Tell the user they can add their API key as a repository secret by running this command in their terminal (do NOT run it yourself — it prompts for the secret value securely):

To activate the workflow after merging, run this in your terminal:

echo "YOUR_API_KEY" | gh secret set ANTHROPIC_API_KEY --repo <owner>/<repo>

Replace YOUR_API_KEY with your actual Anthropic API key.

Fill in the actual <owner>/<repo> for them.

Mark task 3 complete.

Step 6: Commit & PR

Mark task 4 in progress. Commit everything and open a PR:

git add .checks/
git add .github/workflows/checks.yml 2>/dev/null # if CI was set up
git add CLAUDE.md AGENTS.md COPILOT.md 2>/dev/null # whichever was modified
git commit -m "Add checks for automated PR review

Checks:
- [list each check name]"
git push origin HEAD
gh pr create --title "Add checks for automated PR review" --body "Adds AI-powered code review checks that run on every PR.

## Checks added

[list each check with one-line description]

## How it works

Each check is a markdown file in \`.checks/\`. They run as full agents on every PR — reading files, running commands, and providing suggested fixes when they find issues."

If you stashed changes earlier, restore them:

git checkout <original-branch>
git stash pop

Mark task 4 complete. Share the PR URL.