Pr Validator

PR Validator

Validate whether an agent-created PR satisfies its originating PBI's acceptance criteria and follows project conventions. This runs during the Monitor phase — after the coding agent creates a PR but before the human reviews it.

Purpose

Save human review time by catching obvious gaps:

• Missing acceptance criteria
• Missing tests
• Convention violations that the agent should have followed
• Scope creep (changes beyond what was requested)

This is NOT a full code review. It's a structured checklist that flags what to look at.

Inputs

• PR number and repo slug
• The PBI that originated the PR (AB# ID or from feature state)

Process

Step 1: Gather PR Data

gh pr view <prNumber> --repo "<slug>" --json title,body,files,additions,deletions,commits,reviews,statusCheckRollup

Also get the diff stat:

gh pr diff <prNumber> --repo "<slug>" --stat

Step 2: Gather PBI Data

Read the originating PBI's description from feature state or ADO:

$su = Join-Path $HOME ".feature-orchestrator" "state-utils.js"
node $su get-feature "<feature-name>"

Find the PBI that matches this PR (by repo + AB# reference in PR title/body). Extract:

• Acceptance Criteria — the checklist from the PBI description
• Files to Modify/Create — expected file paths
• Technical Requirements — specific implementation guidance
• Testing — expected test coverage

Step 3: Acceptance Criteria Check

For each acceptance criterion in the PBI:

1. Search the PR diff for evidence that it's addressed
2. Mark as: ✅ Addressed | ⚠️ Partially | ❌ Not found | ❓ Can't determine

How to check:

• If the criterion mentions a specific behavior → look for code implementing it
• If it mentions a specific file → check if that file is in the PR's changed files
• If it mentions tests → check if test files are included
• If it's too abstract to verify from diff alone → mark ❓

Step 4: File Coverage Check

Compare the PBI's "Files to Modify/Create" against the PR's actual changed files:

• Expected but not changed → flag as potential gap
• Changed but not expected → flag as potential scope creep (may be fine — dependencies, imports)
• New files created → check naming conventions match the repo's patterns

Step 5: Convention Check

Based on the repo's .github/copilot-instructions.md (which the agent should have followed), spot-check:

• Tests included? If the PBI specified tests and no test files are in the diff → flag
• Telemetry? If the PBI mentioned telemetry/spans and no span-related code is visible → flag
• Feature flag? If the PBI mentioned a feature flag and none is visible → flag
• License headers? If new files were created, check for headers (don't read every file — just note if new files exist)

Do NOT do a full code review. Don't check variable naming, code style, or logic correctness. The human reviewer does that. Focus only on structural completeness.

Step 6: CI Status Check

gh pr checks <prNumber> --repo "<slug>"

Report:

• All passing → ✅
• Some failing → list which checks failed
• Pending → note that CI is still running

Step 7: Present Report

## 🔍 PR Validation: #<prNumber> — <PR title>

**PBI**: AB#<id> — <title>
**Repo**: <slug>
**Changes**: +<additions> -<deletions> across <N> files

### Acceptance Criteria

| # | Criterion | Status | Evidence |
|---|-----------|--------|----------|
| 1 | [criterion text] | ✅ Addressed | [file or code reference] |
| 2 | [criterion text] | ⚠️ Partial | [what's missing] |
| 3 | [criterion text] | ❌ Not found | — |

### File Coverage

| Expected (from PBI) | In PR? | Notes |
|---------------------|--------|-------|
| path/to/File.java | ✅ | Modified |
| path/to/Test.java | ❌ | Not in diff — tests may be missing |

**Unexpected changes**: [list files changed that weren't in the PBI, if any]

### Convention Checks

| Check | Status |
|-------|--------|
| Tests included | ✅ / ❌ |
| Telemetry spans | ✅ / ❌ / N/A |
| Feature flag gating | ✅ / ❌ / N/A |
| CI status | ✅ All passing / ❌ [failures] |

### Summary

**Overall**: 🟢 Looks good / 🟡 Review these gaps / 🔴 Significant gaps

[1-2 sentence summary: what the human reviewer should focus on]

When to Run

• Automatically: When the Monitor phase detects a new PR from the coding agent
• Manually: When the user says "validate PR" or "check this PR"
• On refresh: When the dashboard refreshes PR status and a new open PR is found

Important Guidelines

• Speed over depth: This should take <30 seconds. Don't read every line of code.
• No false confidence: If you can't verify a criterion from the diff, say ❓ not ✅
• Actionable output: Every ❌ or ⚠️ should tell the human what to look for
• Don't block: This is informational. Even if gaps exist, the human decides whether to approve