CI Score

ci-score

CI Score gauges best-practice adherence on CI speed and gate hygiene: a straightforward pass/fail rubric of best-practice checks computed from a repo's workflow configuration. Eleven configuration facts, each self-verifiable in the repo's own YAML in under a minute; score = checks passed / applicable; one refusal (no workflows to check). Registry: references/ci-score-spec.json (frozen v0.1.4); rubric write-up: references/ci-score-methodology.md.

The score measures adherence, not speed — measured, faster repos can hold lower scores. The report says this beside the card; never let a user read the score as a speed verdict. Nor is it a security audit: the rubric grades best practices, and exactly two of its eleven checks (action pinning, job-scoped OIDC tokens) happen to be security-related — it claims nothing further about workflow security.

Requirements

• A full local checkout of the target repo. The score reads three input classes — workflow YAML, local composite actions, and repo-root build-tool configs — so a partial view (an API fetch of workflows alone) silently inflates scores. The collector refuses outside a git checkout. A full clone you make yourself is a valid path (it carries the full input surface — the ban is on partial fetches, never on cloning): ask before cloning (network + disk), and a score of a repo the user does not own stays for their eyes — never published or shared.
• PyYAML (pip install pyyaml). The scripts exit loudly with that hint when it is missing.
• No network access is needed or used; everything reads the local tree.

Flow

All paths below are relative to this skill's directory.

Interaction contract (steps 0 and 4). Both user-facing questions are a single structured question — one question, fixed-order options, nothing open-ended, no machinery narration — via your platform's structured-question tool where one exists: AskUserQuestion on Claude Code; on Codex, its built-in user-input request tool (request_user_input / tool/requestUserInput, experimental — call it when exposed; check for it and attempt the call before ever falling back, exactly as ci-speedup does — a live ci-score Codex run skipped straight to the plain-message branch because this contract predicted the tool would be absent instead of trying it). Only when the tool is genuinely absent or the call fails, ask the same question as one plain message — same options, same order, the close's save option still last and verbatim (None, just save the report (.md)), the target confirm still one keystroke ("Reply y to score <owner/repo>, or name a different repo/path"). Only the delivery mechanism varies; the contract is agent-independent. This whole contract mirrors ci-speedup's interaction contract — same tools, same fallback shape, same verbatim save option — keep the two parallel when either changes.

0. Confirm the target before scoring. Default is the repo you are standing in: git rev-parse --show-toplevel is the checkout root, and the origin remote (when GitHub) names it <owner/repo>. Ask ONE structured question per the interaction contract — ≥ 2 options: one confirms the detected <owner/repo> (or path when there is no GitHub remote), one is "a different repo or path" (its pick or Other supplies the target). When the working directory is not a git checkout (git rev-parse fails) there is no local default to confirm — deliver the question as the plain-message ask or an Other-only prompt (the ≥ 2-option confirm/different shape needs a detected default) and have the user name the target (<owner/repo> or a path). If the user already named a target, re-confirm only if ambiguous (the named target doesn't resolve to a single local checkout or a single clonable <owner/repo>). The confirm is load-bearing even though scoring is local, fast, and free: any applied fix — and the report, once saved — land in the scored repo's root, so a wrong working directory acts on the wrong project. A pick that is another local checkout is scored directly; a pick that is an <owner/repo> not already on disk follows the clone-then-score rule in Requirements (confirm access with gh repo view — in sandboxed agent shells (Codex) a keyring credential can be unreachable and gh false-fails: retry with host access before reading a failure as no-access, mirroring ci-speedup's gh gate — then ask before cloning — network + disk). Do not run the collector until the target is settled.
1. Collect + score (one command, writes the findings document): python3 scripts/collect_config.py --repo <checkout-root> --out <workdir>/findings.json. <workdir> = a scratch directory outside the target checkout — the session scratchpad if one exists, else mktemp -d; use a per-repo subdirectory (<workdir>/<repo-name>/) so a second repo or a re-run never overwrites a prior run's files, and never default to cwd (an untracked findings.json inside the repo makes the next run's provenance -dirty). Exit 0 = scored or an honest refusal in the stamp; exit 2 = collection refusal (not a checkout / no parseable workflows); exit 3 = scoring failed (data_sources.ci_score_error records why). Every outcome is stamped in the document — read it, don't guess.
2. Render the report to scratch: python3 scripts/render_report.py --findings <workdir>/findings.json --out <workdir>/report.md — score card, the adherence-not-speed disclosure, then one recommendation per failed check ranked by impact × risk, each with a fix recipe, its best-practices page, and an agent handoff prompt. The rendered file stays in <workdir> until the user asks for it (the ci-speedup convention, issue #18 there): writing it into the target repo is what the close's None, just save the report (.md) pick does — copy <workdir>/report.md to ./ci-score-report.md (the target repo root) and say where it landed in one clause (a generated artifact they can gitignore or delete — never auto-commit it or edit their .gitignore). No other pick writes the report into the working tree. This also keeps the run self-clean: an unsaved report leaves no untracked file behind to flag the NEXT run's provenance -dirty. Raw findings.json stays in <workdir> always.
3. Verify before presenting: python3 scripts/verify_report.py --findings <workdir>/findings.json --report <workdir>/report.md must print report: OK. If it fails, re-run step 2 and re-verify once (a stale report.md or mismatched findings/report pair is the common cause). If it still fails, the rendered report is unsafe to present: tell the user verification failed (a skill bug, not their repo), give them their result by reading the score line directly from the ci_score stamp, and withhold the recommendations; show the violation lines only if asked.
4. Close per the close/kickoff protocol — read references/close-contract.md first (the "Close contract — invariants" section below is the point-of-use summary, not the whole protocol) — two parts, both in the SAME turn: (a) the close text (banner → disclosure → recommendations → report note), sent as an ordinary message and never packed into the question call, then (b) the kickoff question per the interaction contract above — a SEPARATE structured-question tool call where one exists (AskUserQuestion on Claude Code); where none does (most Codex runs), the question IS the FINAL block of that message: the numbered option list itself, same fixed options in the same order, save option last and verbatim, and the turn ends on it. Writing close text never completes the close — on EITHER platform, only the question does (the tool call, or that final printed option list). Ending the turn after (a) leaves the user with no choices at all (live miss #2, quasar 2026-07-29: the close referenced "the last option below" and then ended — no question ever appeared; the user had to type their pick freehand).

Debug tracing: set STARSLING_LOG_LEVEL=DEBUG (logs counts, file names, and check states — never file contents).

Close contract — invariants

The full close/kickoff protocol lives in references/close-contract.md. These non-negotiables stay here, at point of use, because burying them regressed live runs before — in this skill, text position is behavior. Each carries its one-line incident anchor; hold them while you follow the protocol:

• Banner: copy it VERBATIM from the collector's stdout — never redraw, re-pad, or adjust it freehand (a hand-drawn bar mis-counted 29 of 30 blocks in a live run). A refusal or recorded scoring error prints NO banner and keeps its plain-sentence close — never a banner with an empty or invented bar. Read the score straight from the ci_score stamp; never fabricate a score to fill a gap.
• The close = the close text, THEN the question, in the SAME turn. Writing the close text never completes the close — on either platform, only the question does. The options exist ONLY in the question: the structured-question tool where the platform has one (AskUserQuestion on Claude Code); the final printed option list where it doesn't (most Codex runs). Never narrate the menu in prose ahead of the question (three live misses — quasar 2026-07-29 twice, plus a Codex run — had perfect content and asked no question at all).
• Menu shapes. First close: up to TWO fix slots (a one-edit-closes-both bundle counts as ONE slot; each slot carries that fix's consent scope + risk note), then a different recommendation, then last and verbatim None, just save the report (.md). The post-apply re-offer leads with "Commit this branch and open a PR" — that pick IS the commit + push + PR ask (owner 2026-07-30: a developer lands work before starting more). A ship pick ENDS the loop; NEVER merge. A reported merge starts a fresh banner-led round — re-score the merged base and present it as a full first close, never a bare "re-scored, N/100" aside (live miss 2026-07-30).
• Only the save pick writes the report into the working tree (copies <workdir>/report.md → ./ci-score-report.md); no other pick does.
• One recommendation per approval; stop at the diff. The apply "yes" authorizes the EDIT only — never commit, push, or open a PR on its strength (a live session treated apply as a commit license). The re-scored check is the completion oracle; never fabricate a score and never mutate YAML just to force a pass.

READ references/close-contract.md BEFORE composing the close, every session — it is the executable protocol (consent scope rules, deliberate-absence judgment, ship and post-merge rounds, refusal closes, PR-request rules); the invariants above are what you hold in mind while following it.

Gotchas

• Never write findings.json OR the unsaved report into the target repo — an untracked file makes the NEXT run's provenance -dirty (and published profiles forbid dirty). Both render to <workdir>; only the user's explicit save pick copies the report to ./ci-score-report.md. A saved report left untracked will honestly read -dirty on a later run; tell the user to gitignore or delete it first when a clean-provenance run matters.
• The score is not a speed verdict — measured, the correlation runs the other way (the lowest-scoring repo held the fastest gate in calibration). The disclosure line beside the card exists for this; keep it visible.
• Path filters are the one risky recommendation — a wrong filter can skip CI that should run, and a skipped required check blocks merges. The report's risk note says so; repeat it when applying that fix.
• A subdirectory path still scores the whole repo — the collector anchors to the git top level (a partial view would inflate the score), so --repo anywhere inside the checkout is equivalent.
• -dirty provenance is conservative — untracked files count, and an unverifiable tree (git status failing) is marked dirty, never clean.

Boundaries

ci-score grades configuration hygiene. It never measures speed, never estimates savings, and never renders money. This ban is operational, not just descriptive: if the user asks about speed, cost, or wall-clock at any point — including right after their score ("so why is our CI slow?") — do not improvise an answer from the config facts (a missing cache is not a measurement); say so in one line and route to ci-speedup. For measured wall-clock and runner-minute findings, that is ci-speedup — a separate skill; a ci-speedup run never surfaces a score, and this skill never claims a measurement. Prompt-routing contract (which skill answers what): evals/prompt-routing.json.