How do I install the Repo Verify skill?

Run npx skillmds@latest add benchflow-ai/repo-verify in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Repo Verify safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Repo Verify?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Repo Verify free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Repo Verify?

benchflow-ai (@benchflow-ai) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Repo Verify by benchflow-ai — Agent Skill

Name: Repo Verify
Author: benchflow-ai

Repo Verify

Execute a base→gold fail→pass check on a fix pull request to confirm it contains a genuine regression test. Checks out the pre-PR commit in a throwaway git worktree, installs dependencies, applies only the PR's test changes (expects FAIL), then applies the source fix (expects PASS). A real FAIL→PASS transition returns verified:true; a build failure, an already-passing test, or a suite needing a live server/database returns verified:false with the reason. Use to confirm a PR's tests actually reproduce the bug it fixes, check whether a fix is covered by a regression test, or validate that a repository's fixes are reproducible. Trigger phrases: 'does this PR have a real regression test', 'verify this fix', 'run the fail to pass', 'check if this PR's tests reproduce the bug'.

benchflow-ai 08f95bd 3 files · 24.0 KB Updated Sep 17, 2026 6 repo stars

File contents

Repo verify

Confirm — by running it — that a fix PR contains a test that genuinely fails before the fix and passes after. Static signals like "the PR touches a test file" don't establish that; only executing it does. Runs in a throwaway git worktree, so the working checkout is never touched.

Workflow

Pick a fix PR (or base/head SHAs):

python3 scripts/verify_env.py /path/to/repo --pr 22
# or: --base <sha> --head <sha>

Read the verdict:
- verified: true, reason: "real fail→pass" — the PR's tests reproduce the bug and the fix resolves them.
- verified: false — the reason says why: the PR changes no tests; the tests already pass at base (so nothing was reproduced); they still fail after the fix (build or environment problem); or the suite is e2e/database-backed and can't run hermetically.
- a warning about e2e/integration means reproduction likely needs a live server or database.
To sweep a repository, loop over its fix PRs and keep the ones returning verified: true.

Limits

A build that won't reproduce is a legitimate verified: false — record it rather than retrying until it passes.
The harness runs the PR's own tests. If the test was written alongside the fix, it shows the fix works, not that the bug pre-existed independently.
Dependency install is best-effort and timeboxed; verified: false with an install error means the repo needs a proper build environment, not that the fix is bad.
It runs only the PR's changed test files, not the full suite — so it checks the fail→pass precondition, not that the rest of the suite still passes.

---name: repo-verifydescription: Execute a base→gold fail→pass check on a fix pull request to confirm it contains a genuine regression test. Checks out the pre-PR commit in a throwaway git worktree, installs dependencies, applies only the PR's test changes (expects FAIL), then applies the source fix (expects PASS). A real FAIL→PASS transition returns verified:true; a build failure, an already-passing test, or a suite needing a live server/database returns verified:false with the reason. Use to confirm a PR's tests actually reproduce the bug it fixes, check whether a fix is covered by a regression test, or validate that a repository's fixes are reproducible. Trigger phrases: 'does this PR have a real regression test', 'verify this fix', 'run the fail to pass', 'check if this PR's tests reproduce the bug'.---# Repo verifyConfirm — by running it — that a fix PR contains a test that genuinely fails before the fix andpasses after. Static signals like "the PR touches a test file" don't establish that; onlyexecuting it does. Runs in a throwaway `git worktree`, so the working checkout is never touched.## Workflow1. Pick a fix PR (or base/head SHAs): ```bash python3 scripts/verify_env.py /path/to/repo --pr 22 # or: --base <sha> --head <sha> ```2. Read the verdict: - `verified: true`, `reason: "real fail→pass"` — the PR's tests reproduce the bug and the fix resolves them. - `verified: false` — the reason says why: the PR changes no tests; the tests already pass at base (so nothing was reproduced); they still fail after the fix (build or environment problem); or the suite is e2e/database-backed and can't run hermetically. - a `warning` about e2e/integration means reproduction likely needs a live server or database.3. To sweep a repository, loop over its fix PRs and keep the ones returning `verified: true`.## Limits- A build that won't reproduce is a legitimate `verified: false` — record it rather than retrying until it passes.- The harness runs the PR's **own** tests. If the test was written alongside the fix, it shows the fix works, not that the bug pre-existed independently.- Dependency install is best-effort and timeboxed; `verified: false` with an install error means the repo needs a proper build environment, not that the fix is bad.- It runs only the PR's changed test files, not the full suite — so it checks the fail→pass precondition, not that the rest of the suite still passes.

Frequently asked questions