# Understanding Dominant Themes Reviewing

> Analyze code review comments on AI-authored PRs to identify dominant review themes using a 12-category taxonomy derived from topic modeling and LLM-assisted semantic clustering. Classifies inline review comments into themes like security, testing, documentation, refactoring, styling, and more to surface what human reviewers focus on in agent-generated code. Trigger phrases: - "Analyze these code review comments for dominant themes" - "What are reviewers focusing on in these AI-generated PRs?" - "Classify review feedback into categories" - "What review themes dominate this pull request?" - "Audit the review comments on these agent-authored changes" - "Identify gaps in AI-generated code from review data"

- Skill: `ndpvt-web/understanding-dominant-themes-reviewing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ndpvt-web/understanding-dominant-themes-reviewing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ndpvt-web/understanding-dominant-themes-reviewing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ndpvt-web (https://skillmd.com/u/ndpvt-web)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ndpvt-web/understanding-dominant-themes-reviewing

---


# Understanding Dominant Themes in Reviewing AI-Authored Code

This skill enables Claude to classify inline code review comments on AI-authored pull requests into a rigorous 12-theme taxonomy, then determine the dominant review themes at both the comment level and the PR level. The technique comes from Haider & Zimmermann's empirical study of 19,450 review comments across 3,177 agent-authored PRs, which showed that beyond functional correctness, human reviewers overwhelmingly flag documentation gaps, refactoring needs, styling issues, and under-addressed testing and security concerns. By applying this taxonomy, Claude can give developers and teams a structured view of what human oversight actually catches in AI-generated code.

## When to Use

- When a user provides a set of code review comments (from GitHub, GitLab, or any review tool) and wants to understand what themes dominate the feedback.
- When analyzing whether an AI coding agent's output has systematic blind spots (e.g., consistently missing docs or tests).
- When a team wants to quantify the distribution of review effort across categories for AI-authored PRs versus human-authored PRs.
- When triaging a large batch of review comments to prioritize which feedback categories to address first.
- When building or evaluating a code review bot and needing a ground-truth taxonomy to classify comments.
- When auditing a codebase for patterns in reviewer concern areas after adopting AI coding assistants.

## Key Technique

The core method is a two-stage pipeline: **taxonomy derivation** followed by **zero-shot LLM annotation**.

**Stage 1 -- Taxonomy derivation.** The authors applied transformer-based BERTopic modeling to 19,450 review comments, producing 49 initial clusters. These were then refined through LLM-assisted semantic clustering (using GPT-5/ChatGPT to understand each cluster's underlying semantics and group related clusters), consolidating into 42 fine-grained topics and ultimately 12 distinct thematic categories. Each theme has a short tag (e.g., `secu`, `test`, `docs`), a human-readable name, and a definition grounded in the keyword distributions from the topic model.

**Stage 2 -- Zero-shot annotation.** Individual review comments are classified into the 12-theme taxonomy using a zero-shot prompt to an LLM. The prompt provides the full taxonomy (tag, name, description) and asks the model to select the single most fitting tag for each comment. No few-shot examples are needed. Evaluation against human annotations on 571 comments showed 78.63% exact match, 0.78 macro F1, and Cohen's kappa of 0.73 (substantial agreement). At the PR level, dominant theme identification achieves 78% Top-1 accuracy and 0.76 Jaccard similarity. This means the technique is reliable enough for automated triage and trend analysis, though not infallible for individual edge-case comments.

**What makes this different:** Prior work studied AI code generation quality, but this is the first large-scale empirical taxonomy of what *reviewers actually say* about AI-authored code. The finding that `feat` (core implementation) accounts for only ~39% of comments -- while `refactor` (14%), `docs` (11%), `style` (10%), and `undo` (10%) collectively dominate -- reveals that AI agents produce functionally adequate but poorly polished code. Security (`secu`) and testing (`test`) concerns, while less frequent overall, are significantly overrepresented in *rejected* PRs, making them high-signal indicators.

## The 12-Theme Taxonomy

| Tag | Theme | Description |
|---|---|---|
| `feat` | Core Implementation & Feature Development | New features, logic implementation, core functionality |
| `refactor` | Code Restructure & Simplification | Dead code removal, simplification, maintainability |
| `docs` | Documentation, Logs & Developer Guidance | API docs, changelogs, markdown, developer notes |
| `style` | Styling & Formatting Adjustments | Readability, indentation, linting, naming consistency |
| `undo` | Reverts, Rollbacks & Change Rejection | Reversions, rollbacks, restoring previous state |
| `test` | Testing, Assertions & Validation | Test coverage, assertions, validation checks |
| `chore` | Dependency, Module & Import Management | Package deps, imports, exports, SDK management |
| `secu` | Security, Safety & Reliability | Vulnerabilities, tokens, permissions, defenses |
| `build` | Build, Configuration & Default Behavior | Build systems, config files, default settings |
| `perf` | Performance Improvement | Memory, execution speed, latency optimization |
| `ci` | CI/CD Workflow Management | GitHub Actions, deployment, version workflows |
| `cmd` | Command-Line Tools & Developer Utilities | CLI tools, batch processing, dev utilities |

## Step-by-Step Workflow

1. **Collect review comments.** Gather all inline review comments for the target PR(s). For GitHub, use `gh api repos/{owner}/{repo}/pulls/{number}/comments` or parse exported review data. Each comment needs at minimum its body text; file path and diff context are useful but optional.

2. **Normalize comment text.** Strip markdown formatting artifacts, quoted code blocks (retain only the reviewer's prose), and bot-generated boilerplate (e.g., automated linting output). Remove comments shorter than 10 characters or that are purely emoji reactions -- these carry no thematic signal.

3. **Classify each comment into the taxonomy.** For each comment, apply the following zero-shot prompt pattern:

   ```
   You are a code review analyst. Given the taxonomy below, assign the single most
   appropriate tag to this review comment.

   Taxonomy:
   - feat: Core Implementation & Feature Development (new features, logic, functionality)
   - refactor: Code Restructure & Simplification (dead code, simplification, maintainability)
   - docs: Documentation, Logs & Developer Guidance (API docs, changelogs, developer notes)
   - style: Styling & Formatting Adjustments (readability, indentation, linting, naming)
   - undo: Reverts, Rollbacks & Change Rejection (reversions, rollbacks, restore previous state)
   - test: Testing, Assertions & Validation (test coverage, assertions, validation checks)
   - chore: Dependency, Module & Import Management (package deps, imports, exports)
   - secu: Security, Safety & Reliability (vulnerabilities, tokens, permissions, defenses)
   - build: Build, Configuration & Default Behavior (build systems, config, defaults)
   - perf: Performance Improvement (memory, speed, latency optimization)
   - ci: CI/CD Workflow Management (GitHub Actions, deployment, workflows)
   - cmd: Command-Line Tools & Developer Utilities (CLI tools, batch processing)

   Review comment: "{comment_text}"

   Respond with ONLY the tag (e.g., "docs").
   ```

4. **Aggregate tags at the PR level.** Count tag frequencies across all classified comments for each PR. Retain the top 3 most frequent tags as the PR's dominant themes. The single most frequent tag is the PR's primary theme.

5. **Compute distribution statistics.** Calculate percentages for each theme across the full comment set and across PRs. Compare against the baseline distribution from the paper: `feat` ~39%, `refactor` ~14%, `docs` ~11%, `style` ~10%, `undo` ~10%, `test` ~5%, `chore` ~3%, `secu` ~3%, `build` ~2.5%, `perf` ~1%, `ci` ~0.6%.

6. **Flag high-risk deviations.** If `secu` or `test` themes are overrepresented compared to the baseline (especially if the PR is pending merge), flag this as a rejection risk indicator. The paper found security concerns at 5.59% in rejected PRs vs. 3.05% in accepted ones, and testing at 6.53% vs. 5.44%.

7. **Generate a structured report.** Produce a summary showing: (a) per-PR dominant themes, (b) overall theme distribution with comparison to baseline, (c) flagged risk areas, and (d) actionable recommendations (e.g., "Add missing docstrings" for docs-heavy feedback, "Increase test coverage" for test-heavy feedback).

8. **Validate uncertain classifications.** For comments where the theme is ambiguous (e.g., a comment about renaming a test function could be `style` or `test`), prefer the theme that matches the file context -- test files bias toward `test`, config files toward `build`.

## Concrete Examples

**Example 1: Classifying a batch of GitHub review comments**

User: "Here are 5 review comments from an AI-authored PR. What themes dominate?"

```
Comment 1: "This function is missing a docstring explaining the parameters and return value."
Comment 2: "Nit: inconsistent indentation on lines 42-45, should use 4 spaces."
Comment 3: "The API key is hardcoded here. This should be read from environment variables."
Comment 4: "Can you add a unit test for the edge case where input is an empty list?"
Comment 5: "This helper function duplicates logic already in utils.py -- reuse that instead."
```

Approach:
1. Classify each comment: `docs`, `style`, `secu`, `test`, `refactor`
2. Compute distribution: 20% each across 5 themes
3. No single dominant theme -- this PR has broad review surface area

Output:
```
Theme Classification:
  Comment 1 → docs  (missing docstring)
  Comment 2 → style (indentation inconsistency)
  Comment 3 → secu  (hardcoded API key)
  Comment 4 → test  (missing edge-case test)
  Comment 5 → refactor (duplicated logic)

PR Dominant Themes: No single dominant — evenly distributed across 5 themes.

Risk Flags:
  ⚠ secu: Hardcoded API key detected in review (Comment 3).
    Security themes are 1.8x more common in rejected PRs.
    → Action: Move secret to environment variable before merge.

  ⚠ test: Missing test coverage flagged (Comment 4).
    Testing gaps correlate with PR rejection.
    → Action: Add unit test for empty-list edge case.
```

**Example 2: Analyzing theme distribution across multiple AI-authored PRs**

User: "I have review data from 50 AI-authored PRs. Summarize the dominant themes."

Approach:
1. Classify all comments across 50 PRs using the taxonomy
2. Compute per-PR dominant themes and aggregate distribution
3. Compare against baseline from paper

Output:
```
Theme Distribution (312 comments across 50 PRs):

  feat:      28.2%  (baseline: 38.5%)  ↓ Below expected
  refactor:  19.9%  (baseline: 14.0%)  ↑ Above expected
  docs:      16.3%  (baseline: 11.4%)  ↑ Above expected
  style:     13.1%  (baseline: 10.3%)  ↑ Above expected
  test:       8.0%  (baseline:  5.0%)  ↑ Above expected
  undo:       5.4%  (baseline: 10.0%)  ↓ Below expected
  secu:       4.5%  (baseline:  2.9%)  ↑ Above expected
  chore:      2.6%  (baseline:  3.3%)  ~ Normal
  build:      1.3%  (baseline:  2.5%)  ~ Normal
  perf:       0.6%  (baseline:  1.0%)  ~ Normal

Key Findings:
  - Refactoring and documentation feedback is elevated, suggesting the AI agent
    produces functional but poorly structured/documented code.
  - Security concerns are 1.5x the baseline — review these PRs carefully.
  - Lower 'feat' share means reviewers spend proportionally less time on
    correctness and more on code quality gaps.

Recommendations:
  1. Add documentation generation to the AI agent's post-processing pipeline.
  2. Enable linter enforcement before PR submission to reduce style feedback.
  3. Require the AI agent to generate tests alongside implementation code.
```

**Example 3: Identifying why AI-authored PRs get rejected**

User: "These 3 PRs were rejected. What review themes explain the rejections?"

Approach:
1. Classify all comments in the 3 rejected PRs
2. Identify dominant themes per PR
3. Cross-reference with paper's rejection indicators

Output:
```
PR #142: Dominant themes → secu (35%), feat (25%), test (20%)
  → Security and testing concerns dominate. Both are strong rejection signals.
  → 3 comments flag exposed credentials; 2 flag missing validation tests.

PR #207: Dominant themes → undo (45%), refactor (30%)
  → Reviewers want significant portions reverted or restructured.
  → The AI agent introduced changes outside the intended scope.

PR #389: Dominant themes → build (40%), ci (25%), chore (20%)
  → Configuration and build issues. The agent broke the CI pipeline.
  → All 3 themes relate to infrastructure the agent shouldn't have modified.
```

## Best Practices

- **Do:** Provide the full 12-theme taxonomy in every classification prompt. Zero-shot performance depends on the model having all options visible.
- **Do:** Use file path context to disambiguate borderline comments. A comment about "naming" in a test file is more likely `test` than `style`.
- **Do:** Retain the top-3 tags per PR rather than just top-1. The paper's Jaccard similarity metric validates that multi-label PR characterization is more accurate than single-label.
- **Do:** Compare against the baseline distribution from the paper to identify systematic gaps in the AI agent's output.
- **Avoid:** Classifying bot-generated comments (linter output, CI status messages). These inflate `style` and `ci` counts and don't reflect human reviewer judgment.
- **Avoid:** Treating this taxonomy as exhaustive for human-authored PRs. It was derived specifically from AI-authored code reviews and may not capture themes unique to human code (e.g., design discussion, architectural debate).

## Error Handling

- **Ambiguous comments:** If a comment spans multiple themes (e.g., "Add a test for this security check"), prefer the theme of the *subject* being discussed (`test` in this case — the reviewer wants a test added), not the domain context.
- **Very short comments:** Comments like "nit" or "+1" or "LGTM" lack enough signal. Tag these as `style` if they appear inline on code, or skip them from classification entirely.
- **Non-English comments:** The taxonomy was derived from English-language reviews. For non-English comments, translate first or skip — misclassification rates will be higher.
- **Empty or deleted comments:** Exclude from analysis. Do not attempt to classify placeholder text.
- **Conflicting dominant themes:** When two themes tie for top frequency in a PR, report both as co-dominant rather than arbitrarily picking one.

## Limitations

- The taxonomy was derived from GitHub PRs authored by AI coding agents (Copilot, Devin, etc.) and may not generalize to review comments on human-authored code or code from non-agentic AI tools.
- Zero-shot classification achieves ~78% accuracy. For high-stakes decisions (e.g., automated merge gating), validate a sample against human judgment before relying on automated classification.
- The `cmd` theme is rare (<1% of comments) and may be unreliably classified due to sparse training signal in the topic model.
- The paper's baseline distribution reflects a specific snapshot of open-source GitHub repositories. Internal/enterprise codebases may have different distributions.
- Comments that discuss multiple themes simultaneously will be forced into a single category, losing nuance. The single-label design is a known simplification.

## Reference

Haider, M.A. & Zimmermann, T. (2026). "Understanding Dominant Themes in Reviewing Agentic AI-authored Code." MSR '26. [arXiv:2601.19287](https://arxiv.org/abs/2601.19287v1). Look for: Table 1 (full taxonomy), Section 4 (annotation pipeline), Table 3 (theme distributions), and Section 5.2 (accepted vs. rejected PR analysis).