# Where AI Coding Agents > Pre-flight checker that prevents AI coding agent PRs from failing, based on empirical analysis of 33k agent-authored PRs on GitHub. Applies the rejection taxonomy from Ehsani et al. (MSR 2026) to catch the top causes of PR rejection before submission. Use when: 'check this PR before I submit it', 'why might this PR get rejected', 'review my agent-generated PR', 'audit this pull request for common agent mistakes', 'pre-flight check my changes', 'validate this PR against agent failure patterns'. - Skill: `ndpvt-web/where-ai-coding-agents` (Agent Skill) - Install (CLI): `npx skillmds@latest add ndpvt-web/where-ai-coding-agents` - Raw SKILL.md: https://api.skillmd.com/api/skills/ndpvt-web/where-ai-coding-agents/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: ndpvt-web (https://skillmd.com/u/ndpvt-web) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/ndpvt-web/where-ai-coding-agents --- # Where AI Coding Agents Fail: Pre-Flight PR Checker This skill applies findings from a large-scale empirical study of 33,000 agent-authored pull requests across GitHub (Ehsani et al., MSR 2026) to systematically audit PRs before submission. The study identified that 28.5% of agent PRs fail to merge, with specific, predictable failure patterns. By checking a PR against the paper's hierarchical rejection taxonomy — covering reviewer-level, PR-level, code-level, and agentic-level failures — this skill catches the exact issues that cause real-world rejections before they happen. ## When to Use - When preparing to submit an AI-generated or AI-assisted pull request to an open-source or team repository - When reviewing a batch of changes and wanting to predict merge likelihood - When an agent-generated PR has been rejected and you need to diagnose why - When deciding whether a task (bug fix, feature, refactoring) is suitable for agentic contribution - When you want to decompose a large PR into smaller, more mergeable units - When auditing CI/CD readiness of changes before pushing ## Key Technique The paper's core contribution is a **hierarchical rejection taxonomy** derived from qualitative analysis of 600 rejected PRs, organized into four levels: Reviewer (38% of rejections — PR simply abandoned with no review), PR-level (30% — duplicates, unwanted features, wrong branch), Code-level (22% — CI failures, incorrect/incomplete implementations), and Agentic-level (2% — misalignment with reviewer instructions, license issues). The critical insight is that **the majority of agent PR failures are not code quality problems** — they are contextual and social failures. The single largest rejection category is "Abandoned/Not Reviewed" (38%), meaning the PR never received meaningful human engagement. Quantitatively, the study found three statistically significant predictors of rejection: **(1) PR size** — each increase in lines changed reduces merge odds (Cliff's delta = -0.17); **(2) file spread** — touching more files correlates with rejection (delta = -0.10); and **(3) CI failures** — each additional failed CI check reduces merge odds by ~15% (delta = -0.24). Review comment count and revision count were not significant predictors. Task type matters enormously: documentation PRs merge at 84%, CI/build PRs at 74-79%, but bug fixes merge at only 64% and performance PRs at 55%. The actionable framework is therefore: minimize PR size, ensure CI passes locally before submission, verify the work isn't duplicated, confirm the change is wanted, and target task types where agents empirically succeed. ## Step-by-Step Workflow 1. **Classify the task type.** Determine which of the 11 categories the PR falls into: feature, fix, performance, refactoring, style, documentation, test, chore, build, CI, or other. Flag high-risk categories (performance: 55% merge rate, fix: 64%) and note low-risk ones (documentation: 84%, CI: 79%). 2. **Check for duplicates.** Search the repository's open PRs, recent closed PRs, and issue tracker for overlapping work. Duplicate PRs account for 23% of all rejections. Run: `gh pr list --state all --search ""` and `gh issue list --search ""`. 3. **Verify the change is wanted.** Confirm there is an open issue, maintainer request, or explicit acceptance of the contribution type. Unwanted features account for 4% of rejections. Check CONTRIBUTING.md, recent maintainer comments, and project roadmap. 4. **Measure PR size and scope.** Count total lines changed and files touched. The study shows not-merged PRs are ~17% larger in LOC and touch ~10% more files. If the diff exceeds 300 lines or touches more than 5 files, consider decomposing into smaller PRs. 5. **Run CI/CD locally before submission.** Execute the project's test suite, linter, and build pipeline. CI failure is the strongest code-level rejection signal (delta = -0.24), with each failed check cutting merge odds by ~15%. Run: `npm test`, `make check`, `pytest`, or whatever the project uses. 6. **Validate branch targeting.** Confirm the PR targets the correct branch (main, develop, release). Wrong-branch PRs are a small but entirely preventable rejection category. Check the project's branching strategy in CONTRIBUTING.md or recent merged PRs. 7. **Audit the PR description.** Ensure the title and body accurately describe the actual changes. Wrong/misleading task descriptions account for 1% of rejections. Cross-reference the diff against the PR description line by line. 8. **Check for completeness.** Verify all required logic is present — no TODO stubs, no placeholder implementations, no missing error handling that the issue explicitly requires. Incomplete implementations account for 2% of rejections. 9. **Verify licensing and CLA requirements.** Check if the project requires a Contributor License Agreement or has specific licensing constraints. License issues block PRs entirely. 10. **Generate a merge-likelihood assessment.** Produce a structured report scoring the PR across all taxonomy levels with a final recommendation: submit as-is, revise, or split. ## Concrete Examples **Example 1: Pre-flight check on a bug fix PR** User: "Check this PR before I submit it — it fixes a race condition in the connection pool" Approach: 1. Classify: This is a "fix" task (64% base merge rate — moderate risk) 2. Run `gh pr list --state all --search "race condition connection pool"` to check for duplicates 3. Measure the diff: `git diff --stat main` — if it touches 12 files and changes 847 lines, flag as oversized 4. Run the project's test suite and CI checks locally 5. Verify an open issue exists requesting this fix Output: ``` ## PR Pre-Flight Report Task type: bug fix (historical merge rate: 64% — moderate risk) ### Rejection Risk Assessment | Check | Status | Detail | |------------------------|---------|---------------------------------------------| | Duplicate PR | PASS | No overlapping PRs found | | Change is wanted | PASS | Linked to issue #342 | | PR size | WARNING | 847 LOC across 12 files — exceeds threshold | | CI/CD | FAIL | 2 test failures in pool_test.go | | Branch target | PASS | Targets main (correct per CONTRIBUTING.md) | | Description accuracy | PASS | Description matches diff | | Implementation complete| PASS | No TODOs or stubs found | | License/CLA | PASS | CLA already signed | ### Recommendations 1. CRITICAL: Fix the 2 failing tests before submitting. Each CI failure cuts merge odds by ~15%. 2. STRONGLY RECOMMENDED: Split this PR. Consider separating the lock refactoring (files 1-4) from the actual race condition fix (files 5-8) and the test updates (files 9-12). Smaller PRs merge at significantly higher rates. 3. Estimated merge likelihood: 40% as-is → ~70% after fixes above. ``` **Example 2: Diagnosing a rejected PR** User: "My Copilot-generated PR was closed without comment, what happened?" Approach: 1. Fetch PR details with `gh pr view ` 2. Check comment/review history — if zero human comments, classify as "Abandoned/Not Reviewed" (38% of rejections) 3. Check if the repo has a history of ignoring bot PRs 4. Verify whether the contribution was solicited Output: ``` ## Rejection Diagnosis Category: Abandoned/Not Reviewed (Reviewer Level) This is the #1 rejection pattern for agent PRs, accounting for 38% of all rejections in the MSR 2026 study. The PR was closed without any meaningful human interaction. ### Likely causes: - Repository may not accept unsolicited agent contributions - No linked issue or maintainer request prompted the change - PR may have been auto-closed by a stale-bot after inactivity ### Remediation: 1. Before resubmitting, open an issue proposing the change and wait for maintainer feedback 2. Check if the project has a policy on AI-generated contributions (some repos explicitly ban them) 3. Consider contributing to repositories that actively use AI agents (check their merged PR history) ``` **Example 3: Choosing the right task for agentic contribution** User: "I want to use an AI agent to contribute to this repo. What tasks should I target?" Approach: 1. Fetch recent issues with `gh issue list --label "good first issue" --limit 20` 2. Categorize available issues by task type 3. Rank by empirical merge rates from the study 4. Cross-reference with the specific agent being used Output: ``` ## Task Selection Guide (by merge likelihood) ### High confidence (>75% merge rate): - Documentation updates, typo fixes, README improvements - CI/CD pipeline improvements, GitHub Actions updates - Build configuration, dependency bumps ### Moderate confidence (65-75%): - Code style fixes, linting cleanups - Chore tasks (dependency updates, config changes) - Adding test coverage for existing code - Refactoring with clear scope ### Lower confidence (<65%): - Bug fixes (especially complex multi-file bugs) - New feature implementations - Performance optimizations (lowest at 55%) ### Recommended issues from this repo: 1. #201 "Update CI to Node 20" — CI task, high merge rate 2. #189 "Fix typo in API docs" — Documentation, highest merge rate 3. #156 "Add missing unit tests for auth module" — Test, moderate rate Avoid: #178 "Optimize database query performance" — Performance task with lowest empirical merge rate (55%). ``` ## Best Practices - **Do:** Keep PRs small and focused. The study shows a clear negative correlation between PR size and merge likelihood. Target under 300 LOC and under 5 files. - **Do:** Run the full CI pipeline locally before submitting. CI failure is the single strongest code-level predictor of rejection (each failure = -15% merge odds). - **Do:** Check for duplicate work before starting. 23% of rejected PRs were duplicates of existing work. Search open PRs, closed PRs, and issues. - **Do:** Link to an existing issue or maintainer request. Unsolicited PRs are far more likely to be abandoned without review. - **Avoid:** Submitting performance optimization PRs without benchmarks and maintainer buy-in. Performance tasks have the lowest merge rate (55%). - **Avoid:** Large refactoring PRs that touch many files. Decompose into a series of small, independently reviewable changes. - **Avoid:** Resubmitting a rejected PR without addressing the specific rejection reason. Diagnose the failure category first using the taxonomy. ## Error Handling - **CI check data unavailable:** If the repository doesn't expose CI status via the API, check for a `.github/workflows/` directory and attempt to run workflows locally with `act` or the equivalent test commands. - **No PR history to check for duplicates:** Broaden the search to include commit messages (`git log --oneline --grep=""`), branch names, and issue comments. - **Repository has no CONTRIBUTING.md:** Infer contribution norms from recent merged PRs — look at PR size, description format, branch naming, and whether a CLA bot is active. - **Agent misalignment after reviewer feedback:** If a reviewer requests changes and the agent's follow-up doesn't address them (the "Misalignment" category), intervene manually. Do not let the agent iterate autonomously on reviewer feedback without human verification. ## Limitations - The study covers five specific agents (Codex, Copilot, Devin, Cursor, Claude Code) as of early 2026. Merge rates for newer agents or updated versions may differ. - Codex dominates the dataset (65% of PRs) with an unusually high 82.6% merge rate, which skews overall statistics. Agent-specific rates are more reliable. - The "Abandoned/Not Reviewed" category (38%) may reflect repository-level policies against AI PRs rather than PR quality issues — this is not diagnosable from PR content alone. - Merge rate is an imperfect proxy for quality. Some merged PRs may introduce technical debt; some rejected PRs may be technically sound but socially unwelcome. - The taxonomy was derived from open-source GitHub repositories. Internal/enterprise PR dynamics may differ significantly. ## Reference **Paper:** Ehsani, R., Pathak, S., Rawal, S., Al Mujahid, A., & Imran, M. M. (2026). "Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub." *International Mining Software Repositories Conference (MSR 2026).* **Link:** https://arxiv.org/abs/2601.15195v1 **Key takeaway:** Look at Table 1 (logistic regression coefficients), Figure 4 (rejection taxonomy), and Tables 2-3 (task-type merge rates by agent) for the core quantitative and qualitative frameworks.