stickerdaniel Verify a Candidate PR Actually Fixes the Issue
Goal: take a PR number, check it out cleanly, re-run the same tool call that failed on main, compare the outputs, and produce a verdict — "fixes", "fixes but with concerns", "does not fix", or "cannot run". Pair with /2-repro-issue (which produces the baseline). No edits, no merges, no pushes.
Phase 1 — Resolve the PR and its linked issue
# Accept "498", "#498", or "https://github.com/.../pull/498" — extract the digits only
PR=$(echo "$ARGUMENTS" | sed -E 's|.*/||; s|#||g' | grep -oE '^[0-9]+' | head -1)
[ -z "$PR" ] && { echo "Invalid input: '$ARGUMENTS'. Pass a PR number or URL." >&2; exit 1; }
REPO=stickerdaniel/linkedin-mcp-server
gh pr view $PR --repo $REPO --json title,body,baseRefName,headRefName,headRepositoryOwner,mergeable,mergeStateStatus,additions,deletions,changedFiles,maintainerCanModify,statusCheckRollup
Extract the linked issue number from the PR body (Closes #, Fixes #, Resolves #, or plain #N). Call it ISSUE. If multiple, ask the user which one is the verification target.
Phase 2 — Pick up the baseline + request metadata
/2-repro-issue writes two files for each issue:
/tmp/repro-issue-<ISSUE>-main.json— the on-main response baseline/tmp/repro-issue-<ISSUE>-meta.json— the{tool, arguments}used to call it
Both must exist. The meta file is how this skill replays the exact same call instead of guessing tool and args from the response body.
ls -la /tmp/repro-issue-$ISSUE-main.json /tmp/repro-issue-$ISSUE-meta.json 2>&1
# Both files must exist AND be non-empty. ls alone never fails the script, so
# guard each path before we touch the PR branch or LinkedIn.
[ -s /tmp/repro-issue-$ISSUE-main.json ] || { echo "Missing or empty response baseline. Re-run /2-repro-issue $ISSUE." >&2; exit 1; }
[ -s /tmp/repro-issue-$ISSUE-meta.json ] || { echo "Missing or empty meta file. Re-run /2-repro-issue $ISSUE." >&2; exit 1; }
TOOL=$(jq -r .tool /tmp/repro-issue-$ISSUE-meta.json)
ARGS_JSON=$(jq -c .arguments /tmp/repro-issue-$ISSUE-meta.json)
echo "Replaying: $TOOL($ARGS_JSON)"
If either file is missing, stop and tell the user: "No baseline for #$ISSUE. Run /2-repro-issue $ISSUE first so we have an on-main reference to diff against." Do not silently re-run the reproduction, /2-repro-issue is the canonical source of "what's the call, what's the failure mode".
If the baseline is older than 24 h, warn, LinkedIn DOM/data may have shifted. Offer to re-run /2-repro-issue first.
Phase 3 — Check out the PR
git status --porcelain | head -5 # must be clean
git stash list | head -3 # warn if there are stashes the user may forget
# Remember where to return. `--abbrev-ref HEAD` returns the literal string "HEAD"
# in detached state (CI worktrees, prior PR checkouts), so fall back to the SHA.
CURRENT_REF=$(git symbolic-ref -q --short HEAD || git rev-parse HEAD)
# Fetch the PR head into FETCH_HEAD only and check it out as a detached HEAD.
# Skipping a named local branch avoids clobbering a maintainer's existing
# `pr-$PR` work and removes the cleanup-time `git branch -D` foot-gun.
# Guard the fetch: without it a network/auth failure or deleted pull ref would
# leave a stale FETCH_HEAD and the verdict would run against the wrong commit.
git fetch origin "pull/$PR/head" || { echo "git fetch for PR #$PR failed, aborting before checkout." >&2; exit 1; }
PR_SHA=$(git rev-parse FETCH_HEAD)
git checkout --detach "$PR_SHA"
# Any early exit after this point must still return to $CURRENT_REF + clear
# /tmp scratch files. Install a single trap so the fail-fast guards in Phase 4
# do not strand the user on the PR commit.
cleanup_verify() {
rc=$?
trap - EXIT INT TERM
kill $SERVER_PID 2>/dev/null
wait $SERVER_PID 2>/dev/null
git checkout "$CURRENT_REF" 2>/dev/null
rm -f /tmp/verify-pr-$PR.json /tmp/verify-pr-$PR-headers /tmp/verify-pr-$PR.log /tmp/pr-$PR-meta.json
exit $rc
}
trap cleanup_verify EXIT INT TERM
# Scope sanity check
gh pr view $PR --repo $REPO --json files --jq '.files[].path' | head -20
git diff --stat $(git merge-base "$PR_SHA" origin/main)..$PR_SHA | tail -10
If the PR has merge conflicts with main (mergeStateStatus: DIRTY), continue anyway — local checkout still works — but note it in the verdict.
If maintainerCanModify: true, mention that to the user; it unlocks the "take over and push a small follow-up commit" path later.
Phase 4 — Re-run the same MCP call on the PR branch
$TOOL and $ARGS_JSON came from the meta file in Phase 2. Restart the server because old workers hold the old code. Probe a free port (default 8000 is commonly taken):
PORT=8765
while lsof -nP -iTCP:$PORT -sTCP:LISTEN >/dev/null 2>&1; do PORT=$((PORT+1)); done
uv run -m linkedin_mcp_server --transport streamable-http --port $PORT --log-level INFO > /tmp/verify-pr-$PR.log 2>&1 &
SERVER_PID=$!
# Wait for the port to actually start LISTENing (cap 30s) — a blind sleep would
# let a startup crash silently become a "does not fix" verdict.
for i in $(seq 1 30); do
lsof -nP -iTCP:$PORT -sTCP:LISTEN >/dev/null 2>&1 && break
kill -0 $SERVER_PID 2>/dev/null || { echo "Server died during startup. Tail of /tmp/verify-pr-$PR.log:" >&2; tail -20 /tmp/verify-pr-$PR.log >&2; exit 1; }
sleep 1
done
lsof -nP -iTCP:$PORT -sTCP:LISTEN >/dev/null 2>&1 || { echo "Server never bound port $PORT after 30s" >&2; tail -20 /tmp/verify-pr-$PR.log >&2; exit 1; }
curl -s -D /tmp/verify-pr-$PR-headers -X POST http://127.0.0.1:$PORT/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"verify-pr","version":"1.0"}}}' > /dev/null
SESSION_ID=$(grep -i 'Mcp-Session-Id' /tmp/verify-pr-$PR-headers | awk '{print $2}' | tr -d '\r')
[ -z "$SESSION_ID" ] && { echo "MCP initialize returned no Mcp-Session-Id. Tail of /tmp/verify-pr-$PR.log:" >&2; tail -20 /tmp/verify-pr-$PR.log >&2; kill $SERVER_PID 2>/dev/null; exit 1; }
# notifications/initialized often replies with a pydantic validation error.
# Harmless, same session ID still works for tools/call.
curl -s -X POST http://127.0.0.1:$PORT/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "Mcp-Session-Id: $SESSION_ID" \
-d '{"jsonrpc":"2.0","id":2,"method":"notifications/initialized","params":{}}' > /dev/null
# Same tool + args as baseline, pulled verbatim from /tmp/repro-issue-$ISSUE-meta.json:
curl -s -X POST http://127.0.0.1:$PORT/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "Mcp-Session-Id: $SESSION_ID" \
-d "{\"jsonrpc\":\"2.0\",\"id\":3,\"method\":\"tools/call\",\"params\":{\"name\":\"$TOOL\",\"arguments\":$ARGS_JSON}}" \
| tee /tmp/verify-pr-$PR.json | head -200
kill $SERVER_PID 2>/dev/null; wait $SERVER_PID 2>/dev/null
Phase 5 — Diff and classify
diff -u /tmp/repro-issue-$ISSUE-main.json /tmp/verify-pr-$PR.json | head -200
Classify the PR outcome:
- Fixes cleanly ✓ — the failing field/section/error from the baseline is gone, no new failures visible, output structure matches
CLAUDE.md → Tool Return Format({url, sections, ...}). - Fixes with concerns ⚠ — works on the issue's target but: (a) introduces new keys/sections not in the documented return format, (b) fails on a second target (different locale, edge case), (c) regresses an unrelated tool, (d) scope creep into unrelated files.
- Does not fix ✗ — same failure mode persists.
- Cannot run — merge conflicts that affect runtime, dependency drift (e.g. PR pins a Python version we don't have), or unrelated breakage.
- Cannot verify fix (skill limitation) —
/2-repro-issueproduced a success-baseline, not a failure-baseline (the bug only manifests under crash conditions, stdio transport, specific environments, or was already shipped-fixed). The diff therefore cannot show "before/after fix". Fall back to: (a) regression check — same baseline call must still succeed on PR branch, (b) code audit against the rules in Phase 6. Report this explicitly as a skill limitation, not as a PR weakness.
For locale-sensitive bugs, run one extra call against a deliberately non-English target before declaring "fixes cleanly". The detection logic must not regress on a German/RTL profile.
Phase 6 — Audit the diff against CLAUDE.md scraping rules
Read the actual diff, not just the file list:
git diff $(git merge-base "$PR_SHA" origin/main)..$PR_SHA
Hard flags (any from ✓ to ⚠):
- Locale-dependent detection: any
== "Connect",in ["Pending", "Follow"],contains("1st"),aria-label="..."with translated text. The verb is locale-dependent; attribute presence is not. SeeCLAUDE.md → detection must be locale-independent. - LinkedIn-class-name selectors:
.entity-result__item,.artdeco-button__text, etc. Only minimal generic selectors are acceptable (a[href*="/jobs/view/"]). - Multiple navigations behind one section: any new entry in
PERSON_SECTIONS/COMPANY_SECTIONS(scraping/fields.py) must map to exactly one URL. - Missing tests: bug fixes should add or update a test in
tests/test_scraping.py. Pure-DOM fixes without test coverage are a yellow flag.
Phase 7 — Report
**PR #<PR>** (fixes #<ISSUE>) — <one-line PR title>
**Mergeable:** <CLEAN | DIRTY conflicts | BLOCKED — reason>
**Scope:** <+X/-Y, N files; flag if unrelated>
**Verdict:** <Fixes ✓ | Fixes with concerns ⚠ | Does not fix ✗ | Cannot run | Cannot verify fix (skill limitation)>
**Evidence:** 2–4 lines of the diff between baseline and PR-branch output that prove the verdict.
**Audit flags:** <locale-dependent | DOM-class selectors | section-mapping violation | missing tests | none>
**Cleaner alternative:** <other PR # | refactor sketch | none — PR is good as-is>
**Recommended next step:** <merge | request changes citing flag X | take over via maintainer-edits (maintainerCanModify=true) | close as not-needed>
Then one short paragraph: what the PR actually changes, why it does/doesn't address root cause, and what the locale-independent / DOM-minimal version would look like if the verdict is ⚠ or ✗.
Phase 8 — Cleanup
# The EXIT trap installed in Phase 3 runs automatically — it returns to
# $CURRENT_REF and cleans up /tmp/verify-pr-$PR.* on any exit (success, error,
# Ctrl-C). No local branch was ever created (Phase 3 uses a detached HEAD on
# FETCH_HEAD), so nothing needs deleting either.
# Keep /tmp/repro-issue-$ISSUE-main.json + meta.json so the user can re-verify against another PR later.
Leave the user on the branch they started on, with a clean working tree.
Non-negotiables
- Real LinkedIn, same profile, same call as the baseline. Anything else isn't a verification.
- Restart the server after
git checkout— running workers hold stale code. - Do not push, do not edit the PR, do not merge. This skill verifies and reports.
- Locale and DOM-class flags are non-overridable: if they're present, the verdict is ⚠ at best, regardless of whether the diff "works on the target".
- If the baseline never captured a failure (e.g. crash-only bug, stdio-only bug, env-specific bug), name it "Cannot verify fix (skill limitation)" and lean on code-audit + regression check. Never escalate a missing failure into a PR criticism.