Audit Review Decisions Skill
Mine merged PR review threads for agreed-but-deferred suggestions that were never
implemented. Identify review debt before it compounds.
When to Use
- User says "audit review decisions", "find deferred review items", "surface review
debt", "what did reviewers flag for later"
Arguments
$1 — Time period (e.g. 14d, 30d, 7d). Default: 14d.
$2 — Output path. Default:
${AUTOSKILLIT_TEMP}/audit-review-decisions/review_decisions_audit_$(date +%Y-%m-%d_%H%M%S).md
Critical Constraints
NEVER:
- Create files outside
${AUTOSKILLIT_TEMP}/audit-review-decisions/
- Have triage or validation subagents make GitHub API calls (local data only for Step 2)
- Post duplicate
[AUDIT] markers — check for existing marker before posting
- Run subagents in the background (
run_in_background: true is prohibited)
- Use
gh pr list without --limit to avoid pagination truncation
- Use
\| in Grep patterns — use | for alternation (ERE, not BRE)
ALWAYS:
- Save raw PR JSON to temp before any analysis (Step 1)
- Use GraphQL alias batching (~20 PRs per query) for data collection
- Include
rateLimit { cost remaining resetAt } in every GraphQL query
- Sleep 1s between consecutive mutating GitHub API calls (Step 5 watermark posts)
- Step 2 triage subagents read local JSON files only — zero API calls
- Step 3 validation subagents grep the actual current codebase
- Skip threads that already contain an
[AUDIT] comment
- Resolve owner/repo from
git remote get-url origin — never hardcode
- Use
/autoskillit: prefix when invoking any other skill
Workflow
Step 0: Watermark Resolution
Parse $1 for time period. Default 14d. Compute PERIOD_DAYS.
Resolve OWNER and REPO from git remote get-url origin.
Query the most recent [AUDIT] sentinel comment across recently merged PRs:
gh api graphql -f query='
query($owner:String!, $name:String!) {
rateLimit { cost remaining resetAt }
repository(owner:$owner, name:$name) {
pullRequests(first:500, states:MERGED, orderBy:{field:UPDATED_AT,direction:DESC}) {
nodes { number
reviewThreads(first:50) {
nodes { comments(first:10) { nodes { body createdAt } } }
}
}
}
}
}' -f owner="${OWNER}" -f name="${REPO}"
Extract the most recent createdAt from any comment whose body starts with
[AUDIT]. Store as LAST_AUDIT_TS (empty string if none — first run).
Compute SCAN_SINCE:
- If
LAST_AUDIT_TS is set: max(LAST_AUDIT_TS, date -d "now - PERIOD_DAYS days")
- Else:
date -d "now - PERIOD_DAYS days" --iso-8601=seconds
Log: Scan window: ${SCAN_SINCE} to now (${PERIOD_DAYS}d configured, last audit: ${LAST_AUDIT_TS:-none})
Step 1: Data Collection (GraphQL Batch)
List merged PRs in the scan window:
SCAN_DATE=$(echo "${SCAN_SINCE}" | cut -c1-10)
PR_NUMS=$(gh pr list --state merged \
--search "merged:>=${SCAN_DATE}" \
--json number --limit 500 | jq -r '.[].number')
Create temp directory:
mkdir -p "${AUTOSKILLIT_TEMP}/audit-review-decisions/raw"
Batch fetch in groups of 20 using GraphQL aliases. For each batch, build a query
with aliased pr${i}: pullRequest(number: ${NUM}) nodes. Each node fetches:
number title mergedAt
reviews(first: 100) {
nodes { author { login } body state submittedAt }
}
reviewThreads(first: 100) {
pageInfo { hasNextPage endCursor }
nodes {
isResolved
comments(first: 100) {
nodes { databaseId author { login } body path line createdAt }
}
}
}
Include rateLimit { cost remaining resetAt } at query root.
After the initial fetch, for each PR where reviewThreads.pageInfo.hasNextPage is
true, issue additional aliased queries with reviewThreads(first:100, after:$endCursor)
until hasNextPage is false. Merge the nodes arrays across pages before filtering.
For each PR in the batch response:
- Filter out threads whose
comments list contains any comment with body
starting with [AUDIT] (already watermarked — skip entirely).
- If the PR has zero remaining threads: skip saving.
- Otherwise: save filtered data to
${AUTOSKILLIT_TEMP}/audit-review-decisions/raw/pr_${number}.json
Step 2: Triage (Haiku — Broad Pass)
List all JSON files in raw/. Split into batches of ~5 files per agent.
Launch parallel Haiku subagents (one per batch, model: "haiku"). Each agent:
- Reads its assigned JSON files only (no API calls).
- Flags a thread if it matches any signal:
<!-- REVIEW-FLAG: tag present
- Body contains one of:
"Valid observation — flagged for design decision",
"out of scope for this fix cycle", "requires a dedicated cleanup commit",
"left open for human review", "future improvement", "beyond this PR's scope",
"requires team consensus"
- Thread
isResolved: false AND author acknowledged validity in a reply
- Review body
state: COMMENTED with no corresponding thread (needs_human indicator)
- Returns candidates as response text only — no file writes. Per-candidate format:
PR: {number}
thread_index: {N}
comment_id: {databaseId of first comment in thread}
path: {file path or empty}
line: {line number or empty}
signal: REVIEW-FLAG|KEYWORD|UNRESOLVED|NEEDS_HUMAN
severity: {from REVIEW-FLAG tag, or "unknown"}
dimension: {from REVIEW-FLAG tag, or "unknown"}
quote: {first 200 chars of flagged comment body}
- False positives are acceptable; false negatives are not.
Collect and parse candidate text from all agent responses.
Step 3: Validation (Sonnet — Deep Pass)
Group candidates into batches of ~10. Launch parallel Sonnet subagents
(model: "sonnet") per batch.
Each Sonnet agent receives its candidate batch and, for each candidate:
Collect and parse validated findings from all agent responses.
Step 4: Report Generation
- Collect all validated findings from Step 3 subagent responses.
- Sort findings: VALID first (by priority HIGH→MEDIUM→LOW), then RESOLVED, then STALE.
- Resolve the output path:
- Use
$2 if provided.
- Otherwise:
${AUTOSKILLIT_TEMP}/audit-review-decisions/review_decisions_audit_$(date +%Y-%m-%d_%H%M%S).md
- Create parent directory:
mkdir -p "$(dirname "${OUTPUT_PATH}")"
- Write the markdown report to
${OUTPUT_PATH}. Structure:
PR Review Decisions Audit — {PERIOD_DAYS}d window
Generated: {ISO timestamp}
Scan window: {SCAN_SINCE} to {now}
PRs scanned: {N} | Threads examined: {M} | Threads skipped (already audited): {K}
Summary
| Classification |
Count |
| VALID (ticket-worthy) |
{N} |
| RESOLVED (already fixed) |
{N} |
| STALE (no longer applicable) |
{N} |
Priority Triage
HIGH Priority
For each HIGH VALID finding, write a section:
### {suggested_title}
**PR:** #{number} | **File:** {path}:{line} | **Severity:** {severity} | **Dimension:** {dimension}
> {reviewer_quote}
**Current relevance:** VALID — {impact}
**Suggested issue title:** {suggested_title}
**Affected files:** {path}
MEDIUM Priority
{Same structure}
LOW Priority
{Same structure}
RESOLVED Findings
{List: PR, file, one-line description of what was fixed}
STALE Findings
{List: PR, file, one-line description of why no longer applicable}
Open PR Findings
{Findings from PRs that were open (not merged) at scan time — may still be addressed.
Same per-finding structure but labeled as pending.}
Pattern Analysis
Most common deferral phrases (by frequency):
{Table: phrase | count | % of all candidates}
Dimensions with highest VALID rate:
{Table: dimension | valid | resolved | stale | valid_rate}
Systemic escape hatches detected:
{Narrative: which phrases act as systematic blockers to tracking, with counts}
Recommendations:
{2–4 concrete process recommendations based on the pattern data}
- After writing the file, print a terminal summary:
audit-review-decisions complete
Output: {OUTPUT_PATH}
VALID: {N} | RESOLVED: {N} | STALE: {N}
Top finding: {first HIGH priority suggested_title, or "none"}
Step 5: Watermark (Thread Annotation)
For every finding processed in Steps 2–3 (all classifications — VALID, RESOLVED, STALE):
Re-check for existing audit marker (live): fetch the thread's current comments
directly from the GitHub API — do not use the Step 1 JSON cache, which already
filtered out [AUDIT]-marked threads and cannot detect markers posted after Step 1:
gh api "repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}/comments" \
--jq "[.[] | select(.id == ${COMMENT_ID} or .in_reply_to_id == ${COMMENT_ID}) | .body | startswith(\"[AUDIT]\")] | any"
If the result is true: skip this thread (idempotent — no duplicate post).
Determine marker body based on classification and ticket status:
| Classification |
Ticket created? |
Marker body |
| VALID |
Yes |
[AUDIT] — tracked in #{issue_number} |
| VALID |
No |
[AUDIT] — acknowledged, no action taken |
| RESOLVED |
— |
[AUDIT] — verified resolved in current codebase |
| STALE |
— |
[AUDIT] — no longer applicable |
Post reply comment:
gh api "repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}/comments/${COMMENT_ID}/replies" \
--method POST \
--field body="${MARKER_BODY}"
sleep 1
COMMENT_ID is the databaseId of the first comment in the thread (from Step 1 JSON).
Thread reply constraint: These calls cannot be batched via the reviews API — each
requires an individual POST. The 1s delay between calls is mandatory per GitHub API
discipline.
Log progress per finding: [AUDIT] Posted marker on PR #{number} thread {comment_id}: {marker_body}
1---2name: audit-review-decisions3categories: [audit]4description: >5 Audit merged PR review threads for agreed-but-deferred suggestions (design decisions,6 future work, out-of-scope items) that were never implemented. Mines REVIEW-FLAG markers7 from resolve-review and legacy keyword signals. Produces a structured markdown report8 with VALID/RESOLVED/STALE classifications and annotates processed threads with [AUDIT]9 markers to prevent re-identification on future runs.10hooks:11 PreToolUse:12 - matcher: "*"13 hooks:14 - type: command15 command: "echo 'Auditing PR review decisions...'"16 once: true17metadata:18 source: https://skillmd.com/skills/talont-org/audit-review-decisions19---2021# Audit Review Decisions Skill2223Mine merged PR review threads for agreed-but-deferred suggestions that were never24implemented. Identify review debt before it compounds.2526## When to Use2728- User says "audit review decisions", "find deferred review items", "surface review29 debt", "what did reviewers flag for later"3031## Arguments3233- `$1` — Time period (e.g. `14d`, `30d`, `7d`). Default: `14d`.34- `$2` — Output path. Default:35 `${AUTOSKILLIT_TEMP}/audit-review-decisions/review_decisions_audit_$(date +%Y-%m-%d_%H%M%S).md`3637## Critical Constraints3839**NEVER:**40- Create files outside `${AUTOSKILLIT_TEMP}/audit-review-decisions/`41- Have triage or validation subagents make GitHub API calls (local data only for Step 2)42- Post duplicate `[AUDIT]` markers — check for existing marker before posting43- Run subagents in the background (`run_in_background: true` is prohibited)44- Use `gh pr list` without `--limit` to avoid pagination truncation45- Use `\|` in Grep patterns — use `|` for alternation (ERE, not BRE)4647**ALWAYS:**48- Save raw PR JSON to temp before any analysis (Step 1)49- Use GraphQL alias batching (~20 PRs per query) for data collection50- Include `rateLimit { cost remaining resetAt }` in every GraphQL query51- Sleep 1s between consecutive mutating GitHub API calls (Step 5 watermark posts)52- Step 2 triage subagents read local JSON files only — zero API calls53- Step 3 validation subagents grep the actual current codebase54- Skip threads that already contain an `[AUDIT]` comment55- Resolve owner/repo from `git remote get-url origin` — never hardcode56- Use `/autoskillit:` prefix when invoking any other skill5758---5960## Workflow6162### Step 0: Watermark Resolution63641. Parse `$1` for time period. Default `14d`. Compute `PERIOD_DAYS`.65662. Resolve `OWNER` and `REPO` from `git remote get-url origin`.67683. Query the most recent `[AUDIT]` sentinel comment across recently merged PRs:69 ```bash70 gh api graphql -f query='71 query($owner:String!, $name:String!) {72 rateLimit { cost remaining resetAt }73 repository(owner:$owner, name:$name) {74 pullRequests(first:500, states:MERGED, orderBy:{field:UPDATED_AT,direction:DESC}) {75 nodes { number76 reviewThreads(first:50) {77 nodes { comments(first:10) { nodes { body createdAt } } }78 }79 }80 }81 }82 }' -f owner="${OWNER}" -f name="${REPO}"83 ```84 Extract the most recent `createdAt` from any comment whose `body` starts with85 `[AUDIT]`. Store as `LAST_AUDIT_TS` (empty string if none — first run).86874. Compute `SCAN_SINCE`:88 - If `LAST_AUDIT_TS` is set: `max(LAST_AUDIT_TS, date -d "now - PERIOD_DAYS days")`89 - Else: `date -d "now - PERIOD_DAYS days" --iso-8601=seconds`90915. Log: `Scan window: ${SCAN_SINCE} to now (${PERIOD_DAYS}d configured, last audit: ${LAST_AUDIT_TS:-none})`9293---9495### Step 1: Data Collection (GraphQL Batch)96971. List merged PRs in the scan window:98 ```bash99 SCAN_DATE=$(echo "${SCAN_SINCE}" | cut -c1-10)100 PR_NUMS=$(gh pr list --state merged \101 --search "merged:>=${SCAN_DATE}" \102 --json number --limit 500 | jq -r '.[].number')103 ```1041052. Create temp directory:106 ```bash107 mkdir -p "${AUTOSKILLIT_TEMP}/audit-review-decisions/raw"108 ```1091103. Batch fetch in groups of 20 using GraphQL aliases. For each batch, build a query111 with aliased `pr${i}: pullRequest(number: ${NUM})` nodes. Each node fetches:112 ```graphql113 number title mergedAt114 reviews(first: 100) {115 nodes { author { login } body state submittedAt }116 }117 reviewThreads(first: 100) {118 pageInfo { hasNextPage endCursor }119 nodes {120 isResolved121 comments(first: 100) {122 nodes { databaseId author { login } body path line createdAt }123 }124 }125 }126 ```127 Include `rateLimit { cost remaining resetAt }` at query root.128 After the initial fetch, for each PR where `reviewThreads.pageInfo.hasNextPage` is129 `true`, issue additional aliased queries with `reviewThreads(first:100, after:$endCursor)`130 until `hasNextPage` is `false`. Merge the `nodes` arrays across pages before filtering.1311324. For each PR in the batch response:133 - Filter out threads whose `comments` list contains any comment with `body`134 starting with `[AUDIT]` (already watermarked — skip entirely).135 - If the PR has zero remaining threads: skip saving.136 - Otherwise: save filtered data to137 `${AUTOSKILLIT_TEMP}/audit-review-decisions/raw/pr_${number}.json`138139---140141### Step 2: Triage (Haiku — Broad Pass)1421431. List all JSON files in `raw/`. Split into batches of ~5 files per agent.1441452. Launch **parallel Haiku subagents** (one per batch, `model: "haiku"`). Each agent:146 - Reads its assigned JSON files only (no API calls).147 - Flags a thread if it matches any signal:148 - `<!-- REVIEW-FLAG:` tag present149 - Body contains one of: `"Valid observation — flagged for design decision"`,150 `"out of scope for this fix cycle"`, `"requires a dedicated cleanup commit"`,151 `"left open for human review"`, `"future improvement"`, `"beyond this PR's scope"`,152 `"requires team consensus"`153 - Thread `isResolved: false` AND author acknowledged validity in a reply154 - Review body `state: COMMENTED` with no corresponding thread (needs_human indicator)155 - Returns candidates as **response text only — no file writes**. Per-candidate format:156 ```157 PR: {number}158 thread_index: {N}159 comment_id: {databaseId of first comment in thread}160 path: {file path or empty}161 line: {line number or empty}162 signal: REVIEW-FLAG|KEYWORD|UNRESOLVED|NEEDS_HUMAN163 severity: {from REVIEW-FLAG tag, or "unknown"}164 dimension: {from REVIEW-FLAG tag, or "unknown"}165 quote: {first 200 chars of flagged comment body}166 ```167 - False positives are acceptable; false negatives are not.1681693. Collect and parse candidate text from all agent responses.170171---172173### Step 3: Validation (Sonnet — Deep Pass)1741751. Group candidates into batches of ~10. Launch **parallel Sonnet subagents**176 (`model: "sonnet"`) per batch.1771782. Each Sonnet agent receives its candidate batch and, for each candidate:179 - If `path` is set: reads the file and surrounding context.180 - Greps the current codebase for the core concern from `quote` (judgment-based181 pattern, not a literal string match).182 - Classifies:183 - `VALID` — issue still present, impactful, ticket-worthy184 - `RESOLVED` — code changed; concern no longer applies185 - `STALE` — code deleted/refactored; finding irrelevant186 - Returns findings as **response text only — no file writes**. Per-finding format:187 ```188 PR: {number}189 comment_id: {databaseId}190 classification: VALID|RESOLVED|STALE191 path: {file:line or empty}192 severity: {critical|warning|info|unknown}193 dimension: {arch|bugs|defense|tests|cohesion|slop|unknown}194 priority: HIGH|MEDIUM|LOW195 impact: {one sentence}196 suggested_title: {short GitHub issue title}197 reviewer_quote: {verbatim first 300 chars}198 ```199 - Priority assignment: `HIGH` = severity=critical OR dimension in (bugs, arch);200 `MEDIUM` = severity=warning; `LOW` = severity=info or unknown.2012023. Collect and parse validated findings from all agent responses.203204---205206### Step 4: Report Generation2072081. Collect all validated findings from Step 3 subagent responses.2092. Sort findings: VALID first (by priority HIGH→MEDIUM→LOW), then RESOLVED, then STALE.2103. Resolve the output path:211 - Use `$2` if provided.212 - Otherwise: `${AUTOSKILLIT_TEMP}/audit-review-decisions/review_decisions_audit_$(date +%Y-%m-%d_%H%M%S).md`2134. Create parent directory: `mkdir -p "$(dirname "${OUTPUT_PATH}")"`2145. Write the markdown report to `${OUTPUT_PATH}`. Structure:215216---217218# PR Review Decisions Audit — {PERIOD_DAYS}d window219220**Generated:** {ISO timestamp}221**Scan window:** {SCAN_SINCE} to {now}222**PRs scanned:** {N} | **Threads examined:** {M} | **Threads skipped (already audited):** {K}223224## Summary225226| Classification | Count |227|---|---|228| VALID (ticket-worthy) | {N} |229| RESOLVED (already fixed) | {N} |230| STALE (no longer applicable) | {N} |231232## Priority Triage233234### HIGH Priority235236For each HIGH VALID finding, write a section:237238```239### {suggested_title}240241**PR:** #{number} | **File:** {path}:{line} | **Severity:** {severity} | **Dimension:** {dimension}242243> {reviewer_quote}244245**Current relevance:** VALID — {impact}246247**Suggested issue title:** {suggested_title}248**Affected files:** {path}249```250251### MEDIUM Priority252{Same structure}253254### LOW Priority255{Same structure}256257## RESOLVED Findings258259{List: PR, file, one-line description of what was fixed}260261## STALE Findings262263{List: PR, file, one-line description of why no longer applicable}264265## Open PR Findings266267{Findings from PRs that were open (not merged) at scan time — may still be addressed.268Same per-finding structure but labeled as pending.}269270## Pattern Analysis271272**Most common deferral phrases (by frequency):**273{Table: phrase | count | % of all candidates}274275**Dimensions with highest VALID rate:**276{Table: dimension | valid | resolved | stale | valid_rate}277278**Systemic escape hatches detected:**279{Narrative: which phrases act as systematic blockers to tracking, with counts}280281**Recommendations:**282{2–4 concrete process recommendations based on the pattern data}283284---2856. After writing the file, print a terminal summary:286 ```287 audit-review-decisions complete288 Output: {OUTPUT_PATH}289 VALID: {N} | RESOLVED: {N} | STALE: {N}290 Top finding: {first HIGH priority suggested_title, or "none"}291 ```292293---294295### Step 5: Watermark (Thread Annotation)296297For every finding processed in Steps 2–3 (all classifications — VALID, RESOLVED, STALE):2982991. **Re-check for existing audit marker (live)**: fetch the thread's current comments300 directly from the GitHub API — do not use the Step 1 JSON cache, which already301 filtered out `[AUDIT]`-marked threads and cannot detect markers posted after Step 1:302 ```bash303 gh api "repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}/comments" \304 --jq "[.[] | select(.id == ${COMMENT_ID} or .in_reply_to_id == ${COMMENT_ID}) | .body | startswith(\"[AUDIT]\")] | any"305 ```306 If the result is `true`: skip this thread (idempotent — no duplicate post).3073082. **Determine marker body** based on classification and ticket status:309310 | Classification | Ticket created? | Marker body |311 |---|---|---|312 | VALID | Yes | `[AUDIT] — tracked in #{issue_number}` |313 | VALID | No | `[AUDIT] — acknowledged, no action taken` |314 | RESOLVED | — | `[AUDIT] — verified resolved in current codebase` |315 | STALE | — | `[AUDIT] — no longer applicable` |3163173. **Post reply comment**:318 ```bash319 gh api "repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}/comments/${COMMENT_ID}/replies" \320 --method POST \321 --field body="${MARKER_BODY}"322 sleep 1323 ```324 `COMMENT_ID` is the `databaseId` of the first comment in the thread (from Step 1 JSON).3253264. **Thread reply constraint**: These calls cannot be batched via the reviews API — each327 requires an individual POST. The 1s delay between calls is mandatory per GitHub API328 discipline.3293305. Log progress per finding: `[AUDIT] Posted marker on PR #{number} thread {comment_id}: {marker_body}`