Review CI

CI Review Skill

Review CI results for a branch, commit, or PR using the Datadog MCP. Identifies failed jobs, failed tests, and known flaky tests, then produces actionable fix instructions for an AI coding agent or developer.

Prerequisite: Datadog MCP

This skill requires the Datadog MCP. Check if mcp__datadog-mcp__* tools are available in your tool list before proceeding. If they are NOT available, stop and instruct the user:

claude mcp add --transport http datadog-mcp \
  "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=core,software-delivery"

See https://docs.datadoghq.com/bits_ai/mcp_server/ for setup details. Authentication uses the user's existing Datadog credentials — run claude mcp to verify the connection is authenticated.

Repository constants

REPO_URL = https://github.com/DataDog/dd-trace-py
REPO_ID  = github.com/datadog/dd-trace-py   # lowercase, no scheme — for coverage tools

Step 0: Resolve the target

If the user did not supply a branch, commit SHA, or PR number, resolve from git:

git rev-parse --abbrev-ref HEAD   # branch
git rev-parse HEAD                # full commit SHA

Save both values — they are used in every subsequent step.

Step 1: Find the PR (if not already known)

If you have a branch but no PR number, call get_prs_by_head_branch:

repo_url    = https://github.com/DataDog/dd-trace-py
head_branch = <branch>

Pick the most recent open PR. A PR number is needed for search_pr_insights.

Step 1.5: Check Merge Queue status (if PR number is known)

The merge queue is an internal system (not GitHub's built-in merge queue). When a PR is queued, a working branch like mq-working-branch-main-4839cf6 is created and CI runs on that branch — not on the PR's head branch. This step finds working branch details so Steps 2–4 can also check for failures there.

Query search_datadog_logs for MQ worker logs:

query   = @pr_number:<pr_number> service:mergequeue-worker -"WaitingCron" -"not mergeable yet"
from    = now-7d
sort    = timestamp   (oldest first, to see full attempt history)

Key fields to extract from the log attributes (extra_fields if needed):

• status — queued, in_progress, rejected, merged
• workingBranch.name — e.g. mq-working-branch-main-4839cf6
• WorkflowType — look for BuildWorkingBranch (attempt start) and UpdateMergeRequestStatus (outcome)
• head_sha / working branch SHA — for CI queries

Important: The gitlab_pipeline_id / @ci.pipeline.id from MQ logs does NOT match Datadog CI pipeline events. Always use @git.branch:<working_branch_name> or @git.commit.sha:<working_branch_sha> to find CI results for a working branch.

If MQ logs are found:

1. Note all working branch names from past/current attempts
2. In Steps 2–4, query for failures on the working branch in addition to the PR's head commit:
   • @git.branch:<working_branch_name> @ci.status:error
   • @git.commit.sha:<working_branch_sha> @test.status:fail
3. For GitHub Actions on the working branch: gh run list --branch <working_branch_name>

If no MQ logs are found, the PR has not been queued — skip this step and proceed.

Step 2: Get failed CI jobs

Call search_datadog_ci_pipeline_events with ci_level=job to find failed jobs.

For the PR's head commit:

query    = @git.commit.sha:<full_sha> @ci.status:error
ci_level = job
from     = now-7d
sort     = -timestamp

If a working branch was found in Step 1.5, also query by branch:

query    = @git.branch:<working_branch_name> @ci.status:error
ci_level = job
from     = now-7d
sort     = -timestamp

Note the pipeline_name and job_name for each failure:

• Pipeline names like DataDog/apm-reliability/dd-trace-py → GitLab CI
• Workflow names like Changelog, Build, System Tests → GitHub Actions

The provider tells you where to get logs (Step 3).

Step 3: Get detailed logs for each failed job

GitLab CI jobs

Call search_datadog_logs with @ci.job.id:<job_id>, time window from job start to end+1h:

query = @ci.job.id:<job_id>
from  = <job start timestamp>
to    = <job end timestamp + 1 hour>

Parse logs for: compiler errors, formatting diffs, test assertion failures, specific file paths and line numbers. Sort oldest-first (sort=timestamp) to read the failure at the end.

If search_datadog_logs returns 0 results, retry with storage_tier=flex_and_indexes.

GitHub Actions jobs

Logs for GitHub Actions are NOT indexed in Datadog. Use gh CLI instead:

# Get run IDs for the PR's head commit
gh run list --commit <sha> --json databaseId,name,status,conclusion

# If a working branch was found in Step 1.5, also check it
gh run list --branch <working_branch_name> --json databaseId,name,status,conclusion

# Get logs for a specific failed run
gh run view <databaseId> --log-failed

Match failed runs by name to the job names from Step 2.

Step 4: Get failed test events (always run — even if all jobs passed)

Call search_datadog_test_events filtered by commit SHA. If a working branch was found in Step 1.5, run both queries:

# PR's head commit
query      = @git.commit.sha:<head_sha> @test.status:fail
test_level = test
from       = now-7d
page_limit = 10

# Working branch (if found in Step 1.5)
query      = @git.commit.sha:<working_branch_sha> @test.status:fail
test_level = test
from       = now-7d
page_limit = 10

Important: Tests can fail without failing the CI job (known-flaky tests are suppressed). Always run this step regardless of pipeline status.

Key fields to extract from each failing test:

• test.name, test.suite, test.module — what failed
• test.source.file + test.source.start/end — exact location
• error.message + error.stack — the actual failure
• test.is_known_flaky / test.is_flaky — if true, this is pre-existing noise
• ci.provider.name, ci.job.name — which CI job ran it

Step 5: PR insights (supplement)

If a PR number is known, call search_pr_insights:

repo_url  = https://github.com/DataDog/dd-trace-py
pr_number = <pr_number>

This supplements Step 2–4 but only lights up for test-level failures — it is silent for build/format/changelog failures.

Step 6: Present the actionable summary

Structure the output as follows:

Merge Queue status (if applicable)

If MQ logs were found in Step 1.5, show a brief history at the top:

**Merge Queue**: Attempt #1 rejected at 14:07 (checks failed). Attempt #2 in progress since 14:14.
Working branch: mq-working-branch-main-4839cf6

If the PR has not been queued, omit this section entirely.

Overall status

One line: "X job failures, Y test failures (Z known-flaky), A GitHub Actions failures." Or: "All CI passed. N known-flaky test failures — no action needed." If failures are on the working branch (not the PR head), note: "(on MQ working branch)".

Failed jobs (new/actionable)

For each non-flaky job failure:

**[Job name]** (GitLab/GitHub, <duration>s)
Root cause: <extracted error from logs>
Files affected: <file paths and line numbers>
Fix: <concrete command or action>

Examples of good fix instructions:

• "Run cargo fmt in src/native/ to fix Rust formatting"
• "Run ruff format ddtrace/internal/native/_native.pyi"
• "Run reno new <slug> to add a release note, or apply changelog/no-changelog label"
• "Fix assertion at tests/foo/test_bar.py:42: expected X, got Y"

Known-flaky test failures (noise)

List them briefly so the user is aware, but mark clearly as pre-existing:

Known-flaky (no action needed):
- test_foo_bar[py3.12] in tests/internal/test_foo.py:38 — assert 'x' == 'y'

Coverage (if requested)

Only include if the user asked about coverage or if patch coverage dropped significantly. Use get_datadog_code_coverage_commit_summary or get_datadog_code_coverage_branch_summary.

Quick reference: when to use each tool