Review Runs

Review Runs

Analyze the previous night's tend CI runs in this repository. Identify behavioral problems, skill gaps, and workflow issues — then propose improvements to the repo's local skills and workflows.

This skill runs in the adopter repo, not in tend. Improvements target .claude/skills/ and .config/tend.yaml in this repository.

First steps

Load /tend-ci-runner:running-in-ci first — it contains CI security rules, PR/comment formatting (line wrapping, heredoc hazards), and polling conventions. This skill opens PRs and issue comments, so those rules apply.

ls .claude/skills/

Load any repo-specific skill overlay before proceeding.

@review-gates.md

Evidence accumulation

Each run contributes to a monthly review-runs-tracking issue. Prepare the current tracker and read the current and previous month's evidence:

uv run --script \
  "${CLAUDE_PLUGIN_ROOT}/scripts/review_runs.py" prepare-evidence

The command creates this month's tracker when needed, closes older open trackers, persists the current issue id, and prints both evidence windows.

After analysis, write the new findings in the format from @review-gates.md to $TMPDIR/findings.md. Include a literal ## Run $GITHUB_RUN_ID heading. Then append them:

uv run --script \
  "${CLAUDE_PLUGIN_ROOT}/scripts/review_runs.py" append-evidence

The command refuses a findings file that does not name this run. It appends to the latest bot evidence comment while the combined body remains below 60 KB, then starts a new comment. Prior entries are never replaced.

Step 1: Find recent runs

List tend CI runs that completed since the previous review-runs run (nominally 24 hours — the cron runs daily):

uv run --script \
  "${CLAUDE_PLUGIN_ROOT}/scripts/list_recent_runs.py" review-runs

If no runs are found, report "no runs to review", complete Reconcile live work below, then exit.

Report the run census as the count this returns. .total_count counts the wider FETCH_FROM fetch, so it bounds the census from above rather than matching it — but a census that lands on a round page boundary (30, 100) is still the signature of a page that was never followed, so check that one against .total_count before trusting it.

Then, for each run ID from above, pull its jobs and classify them:

• Long-running (>30 min): Tend runs typically finish in single-digit minutes. Anything over 30 is worth a look — download session logs in Step 3 and diagnose where the time went (long background waits, push-wait-fix cycles, a stuck tool call).
• Near-timeout (within 90% of the cap): A job that consumed most of its timeout budget is one slow external check away from being killed. Structural, but classify the cost per Gate 3 by what the kill left on the record: usually waste-class (a cron-driven run a later tick retries), except where the killed session had already taken an outward action it was still gated on — a tend-review job killed mid-poll leaves its approval standing over red CI. Waste-class gets only a remedy that passes Gate 3, or nothing.

To determine the timeout cap for a workflow, read timeout-minutes from that workflow's own file under .github/workflows/ — the census admits workflows named outside the tend- prefix, so don't glob for one. Tend's generated workflows do not set timeout-minutes, so GitHub's 360-minute default applies unless the adopter has overridden it via workflows.<name>.jobs.<job>.timeout-minutes in .config/tend.yaml.

# Flag long-running and near-timeout jobs
gh api "repos/$GITHUB_REPOSITORY/actions/runs/$RUN_ID/jobs" \
  --jq '.jobs[]
    | ((.completed_at | fromdateiso8601) - (.started_at | fromdateiso8601)) as $dur
    | select($dur >= 1800)   # 30 min
    | {name, conclusion, duration_min: ($dur / 60 | floor), url: .html_url}'

After retrieving the timeout cap from the workflow file, flag any job whose duration exceeded 90% of it as a near-timeout. For the default 360-min cap, that threshold is 324 min.

Reconcile live work

The failed-run census and the tend-outage issue diagnose availability. Do not replay historical workflow runs to recover their event payloads: issue and PR recovery belongs to the unread notification queue, which applies the current workflow and current repository state.

As a daily backstop for delayed notifications, retention, edited activity, and repaired subscriptions, inspect the live repository for:

• an open issue with no bot response to the latest human activity;

• an open PR whose live head has no bot review, or whose latest comment, review, or inline review comment directed at the bot has no response; this includes replies to the bot's review on a fork PR;

• failing default-branch CI with no bot fix in progress. A live-state check like the two above it: scoped neither to ci-fix's watched workflows — Dependabot security updates, cron releases and doc builds fail there with no PR attached, and nothing else looks for them — nor to this run's window.

  DEFAULT_BRANCH=$(gh repo view --json defaultBranchRef --jq '.defaultBranchRef.name')
  # One server-side filter per red conclusion, to reach past a busy repo's green
  # runs. `cancelled` is left out: concurrency cancels dominate it.
  # Each filter is re-read until two consecutive pages agree, then unioned: a
  # single read can serve a coherent older page that omits the newest rows, and
  # nothing below recovers a row never returned.
  for status in failure startup_failure timed_out; do
    prev=__unread__  # no page equals this, so an empty answer is re-read too
    for _ in 1 2 3 4; do
      page=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs?branch=$DEFAULT_BRANCH&status=$status&per_page=50" \
        --jq '.workflow_runs[] | {id, name, path, event, conclusion, created_at}')
      [ -n "$page" ] && printf '%s\n' "$page"
      [ "$page" = "$prev" ] && break
      prev=$page
    done
  done | sort -ru
  # Closure: for each distinct `path` above that is a real workflow file, its
  # latest green default-branch run closes that path's older red rows.
  # `dynamic/dependabot/...` paths are not files and 404 here.
  # Can serve a stale row; re-query once before reporting a path as still red.
  gh api "repos/$GITHUB_REPOSITORY/actions/workflows/<basename of path>/runs?branch=$DEFAULT_BRANCH&status=success&per_page=1" \
    --jq '.workflow_runs[0] | {name, conclusion, created_at}'
  

  Both calls sometimes serve an older page, and staleness moves each answer backwards — in opposite directions. On the closure read a too-old green fails to close a path that is already fixed, so the sweep over-reports: re-run that call once and take the newer answer before reporting any path as still failing. On the red listing the missing rows are the newest ones, so the sweep under-reports — a stale page supports "main is green" while a failure stands on it. Consecutive reads of the same URL have returned different snapshots, and a stale page is internally coherent, so agreement between two of them is the only convergence signal the rows carry; that is what the loop above waits for. Rows come out newest-first within each conclusion.

  Unwindowed on purpose: a failure nobody fixed is still live on the nights after it ran, so anchoring on $TMPDIR/review-runs-since would surface each one the night it happened and read as an all-clear afterwards. The page reaches back weeks, so most rows are already fixed and the closure call is what separates them. What it cannot close stays live: Dependabot's security updates have no workflow file, and each run's name carries a per-update ID that never recurs, so those rows close only through a fix PR or a tracker. Step 1's census reaches back 49h at most, so skip only the tend rows inside its window — a tend workflow red for longer than that, with no green since, is news here like any other row. Report the scope the claim rests on — "main is green" is read later as covering every workflow — naming the workflows checked and how far back the page reached.

• an open Dependabot security alert with no PR or tracker proposing its fix — same closure as the red rows above, so an alert whose fix needs a maintainer decision stops re-surfacing once it is tracked. Dependabot opens that PR itself for most alerts, so the ones that reach this sweep are the ones where it could not — and nothing else in tend looks: weekly reviews the dependency PRs that exist, and the defining property here is that none was created.

  gh api "repos/$GITHUB_REPOSITORY/dependabot/alerts?state=open&per_page=50" \
    --jq '.[] | {number, dep: .dependency.package.name, manifest: .dependency.manifest_path,
                 sev: .security_advisory.severity, created_at,
                 fix: .security_vulnerability.first_patched_version.identifier}'
  

  The PAT's repo scope covers this. Where alerts are disabled the call fails with 403 Dependabot alerts are disabled for this repository. — that is the check not applying, not a missing scope; only an empty result means no open alerts. Otherwise run it unconditionally. An alert open for more than a few days with no PR naming its package is live work in the same sense as a red default-branch run. A red dynamic/dependabot/... row above naming that package is the mechanism: Dependabot is erroring, so waiting will not produce a PR and the manifest or lockfile has to be bumped directly. gh run view <id> --log-failed ends with an error table naming the dependency and the error type; security_update_not_possible also reports the lowest non-vulnerable version beside the highest the dependency tree currently resolves, which is the constraint to relax.

Handle live work through the normal triage, review, or CI-fix guidance. Keep failed runs in the report as diagnostic evidence.

After the exhaustive live scan, find the canonical current outage tracker and read every row. The issue body holds the first row and later rows are comments. Fail the sweep if the lookup fails; that is different from finding no open tracker:

if ! gh issue list --state open --label tend-outage --author @me \
  --limit 100 --json number,title \
  --jq '[.[] | select(.title == "Bot temporarily unavailable") | .number]
    | sort | .[0] // empty' > "$TMPDIR/review-runs-outage-number"; then
  echo "Could not read the outage tracker" >&2
  exit 1
fi
OUTAGE=$(cat "$TMPDIR/review-runs-outage-number")
if [ -n "$OUTAGE" ]; then
  gh issue view "$OUTAGE" --json body,comments --jq '.body, .comments[].body'
fi

Use every row to identify what the failed run may have missed. Diagnose it and handle any applicable current work. If a tracker was found, close the exact issue number returned above:

OUTAGE=$(cat "$TMPDIR/review-runs-outage-number")
[ -n "$OUTAGE" ] && gh issue close "$OUTAGE" --reason completed

Step 2: Token usage report

Run the token report script to get per-run token counts:

# Whole hours back to Step 1's anchor, rounded up so the whole band is priced.
# A literal `24` reopens the gap Step 1 closed. The `cat` isn't optional: an
# unset `$SINCE` makes `date -d ""` today's midnight, not an error.
SINCE=$(cat "$TMPDIR/review-runs-since")
HOURS=$(( ( $(date -u +%s) - $(date -u -d "$SINCE" +%s) + 3599 ) / 3600 ))
uv run --script \
  "${CLAUDE_PLUGIN_ROOT}/scripts/token_report.py" "$HOURS" \
  > "$TMPDIR/token-report.json"

Pass the same extra prefixes Step 1 censuses (after $HOURS, which the script reads as its first positional arg), so the two steps agree on what the fleet is — the repo's running-tend skill is the source for both (e.g. review- for a review-reviewers workflow that uses the tend action but isn't named tend-*).

Include the total cost and the per-workflow breakdown in the summary (Step 7). Escalate outliers to Step 3 — for example a run far above its workflow's usual cost, or a subject the subject table shows several runs against.

Step 3: Download and analyze session logs

Load /install-tend:debug-tend-run for download commands and JSONL parsing queries.

Skip runs without artifacts. Trace decision chains: what did tend decide, what evidence did it use, what was the outcome?

Step 4: Cross-check outcomes

For each analyzed run, compare what the bot did against what happened next. The same "did it stick?" question applies to every tend workflow — ask it of whatever ran. For example:

• Review: did subsequent commits undo something the bot approved? Did human reviewers flag issues it missed?
• Triage: was the classification correct? Did the issue get relabeled?
• Nightly: did the bot's PRs merge, or get closed as unhelpful?
• CI-fix: did the fix actually resolve the failure?

mention, notifications, weekly, and review-reviewers runs get the same treatment: find the bot's output and check whether it was accepted.

Dispositions — merged, closed, relabeled, reverted — are only half the signal. A maintainer replying in-thread that a bot claim was wrong, or requesting changes on a bot PR, leaves labels and state untouched and is equally a correction; where the bot authors most of the PRs, a review body is the first place a maintainer writes. The script collects all three — dispositions, thread comments, review bodies — for the window:

uv run --script \
  "${CLAUDE_PLUGIN_ROOT}/scripts/review_runs_corrections.py" \
  "$(cat "$TMPDIR/review-runs-since")"

Read every row: a correction is a maintainer contradicting a bot claim, not merely replying. Comment rows carry both timestamps because the window filters on updated_at — a created before the anchor is an older comment edited inside the window, a real hit rather than a broken filter. Empty dispositions, comments, and reviews is the all-clear.

Write "no maintainer corrections" into the tracking issue only after the script ran and returned empty — future runs read the phrase as ground truth when counting occurrences under Gate 1, so an unchecked all-clear suppresses the evidence it exists to accumulate. The script exits non-zero rather than reporting an empty window when the anchor or the bot login is missing, since both filters fail open.

Step 5: Deduplicate

Before creating issues or PRs, check for existing ones:

gh issue list --state open --limit 200 --json number,title,body
gh issue list --state closed --json number,title,closedAt --limit 200
# --state all: a merged PR is the most common way a finding is already fixed
gh pr list --state all --limit 200 --json number,title,state
# Bundled-skill defects are filed upstream (Step 6), and the queries above only
# see this repo — dedup against tend before filing there.
gh pr list --repo max-sixty/tend --state all --limit 200 --json number,title,state
gh issue list --repo max-sixty/tend --state all --limit 200 --json number,title

Search the titles for related keywords, then read the bodies of the candidates (gh pr view <n> --json body).

A fix merged upstream still reproduces here. The action ref is pinned per release, so a skill fix that merged in max-sixty/tend stays dormant on this repo until the next release tags. Observing the bug is therefore not evidence the fix is missing — check tend's merged PRs before filing, or the report is churn on something already landed.

Step 6: Act on findings

Improvements target repo-local files by default:

• .claude/skills/ — update or create skill overlays with guidance that prevents the identified problem. Prefer updating existing skill files over creating new ones.
• .config/tend.yaml — adjust workflow configuration if the problem is structural (e.g., wrong cron schedule, missing setup step).
• Project instruction file (CLAUDE.md or AGENTS.md) — add project-specific guidance if the problem is about code conventions or patterns the bot keeps getting wrong.

Bundled-skill defects. If the root cause is a gap or bug in a bundled skill (plugins/tend-ci-runner/skills/... in max-sixty/tend) — the same pattern would fire in every consumer — file the fix against tend per Other Repos in /tend-ci-runner:running-in-ci. Signal: the fix reads as generic guidance that would apply to any consumer.

Prefer PRs over issues. A PR with a clear description is immediately actionable.

Editing .claude/skills/ requires the read-only-mount workaround (bind-mounted read-only, plus a harness write-guard on .claude/skills/ paths) — see references/skill-pr-workflow.md in /tend-ci-runner:running-in-ci. Adapted for review-runs (base on HEAD since this runs on a schedule, not a PR checkout; move each edited file into place):

git worktree add "$TMPDIR/review-runs-fix" -b daily/review-runs-$GITHUB_RUN_ID HEAD

# Author each edited skill file at $TMPDIR/<name>.md.
# Then move the files into place:
cd "$TMPDIR/review-runs-fix/.claude/skills/running-tend" && mv "$TMPDIR/running-tend.md" SKILL.md
# Repeat per skill file being updated.

cd "$TMPDIR/review-runs-fix"
git add .claude/skills/
# Set git identity first if not already done this session — a fresh worktree has
# none and the commit fails with `Author identity unknown`. See "Configure git
# identity before the first commit" in /tend-ci-runner:running-in-ci's references/pr-creation.md.
git commit -m "skills(running-tend): ..."
git push -u origin daily/review-runs-$GITHUB_RUN_ID
gh pr create --title "..." --body-file "$TMPDIR/pr-body.md" --head daily/review-runs-$GITHUB_RUN_ID
cd -
git worktree remove "$TMPDIR/review-runs-fix" --force

.config/tend.yaml and project instruction files are not under the read-only mount, but if you're already in the worktree for a .claude/skills/ edit, do those edits there too so the branch stays self-contained.

• PR (default): Branch daily/review-runs-$GITHUB_RUN_ID, fix, commit, push, create with label review-runs. Write the description for a maintainer deciding whether the current change fixes the general behavior gap, following Reader-facing prose in /tend-ci-runner:running-in-ci. Link the tracking issue where it holds prior observations of the same behavior, and carry the evidence that justified promoting this finding — the run IDs, the log excerpt, and the gate assessment — in the body or a <details> block.
• Issue (fallback): Only for problems too large or ambiguous to fix directly.

Limit to at most 2 PRs per run. Pick the highest-confidence findings; note the rest in the tracking issue.

Step 7: Summary

If no problems found (or none passed the gates), report "all clear" with: runs analyzed, sessions reviewed, brief quality assessment, and any below-threshold findings recorded in the tracking issue.

Save the summary to $GITHUB_STEP_SUMMARY (a later workflow step copies this into the GitHub Actions step summary):

cat > "$GITHUB_STEP_SUMMARY" << 'EOF'
## Review-runs summary
...
EOF