# Harness Catalog Retrospective > Harness Catalog Retrospective - Skill: `intense-visions/harness-catalog-retrospective` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add intense-visions/harness-catalog-retrospective` - Raw SKILL.md: https://api.skillmd.com/api/skills/intense-visions/harness-catalog-retrospective/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: Intense-Visions (https://skillmd.com/u/intense-visions) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/intense-visions/harness-catalog-retrospective --- # Harness Catalog Retrospective > Monthly retrospective over skill-adoption telemetry. Reads `.harness/metrics/adoption.jsonl`, ranks the most-invoked, most-failing, and abandoned-mid-workflow skills, flags ever-invoked skills that have gone quiet, and reports how much of the catalog emits any telemetry at all. Produces a dated Markdown report and surfaces the highlights that warrant follow-up. Compounding-via-learning at the catalog grain. ## When to Use - On a regular cadence (monthly is the intended grain) to review how the skill catalog is actually being used. - After a milestone, to see which skills carried the work and which never fired. - When deciding what to prune, fix, or promote in the catalog — the report feeds catalog-rationalization work. - NOT as a real-time dashboard — for point-in-time lookups use `harness adoption skills` / `harness adoption recent`. - NOT when `.harness/metrics/adoption.jsonl` is absent or empty (the report will be honest but empty; there is nothing to retrospect on yet). ## Process ### Iron Law **Separate real signal from telemetry gaps before you recommend anything.** The single most common misread of this data is treating "no telemetry" as "abandoned." Adoption records are emitted only from instrumented entry points, so a skill with zero records is usually uninstrumented, not unused. The report's coverage line and the stale-skills section exist to keep that distinction visible — never collapse it. --- ### Phase 1: SCAN — Derive the Report 1. Run the retrospective command from the project root: ```bash harness adoption retrospective # writes docs/retrospectives/.md harness adoption retrospective --no-write # print to stdout without writing a file harness adoption retrospective --json # structured output for further processing ``` Optional flags: `--inactive-days ` (stale threshold, default 90), `--top ` (rows per section, default 10), `--out ` (override the output file). 2. The command reads `.harness/metrics/adoption.jsonl` via `readAdoptionRecords`, derives the report via `getCatalogRetrospectiveReport`, and renders it via `renderRetrospectiveMarkdown` (all in `@harness-engineering/core`). Coverage context is computed against the skills discovered under `agents/skills/claude-code/`; when that directory is absent (a consumer project), the coverage line is omitted rather than reporting a false zero. 3. The report is followed by a **Bayesian skill-effectiveness** section (`## Bayesian skill effectiveness`), computed by the skill scorer in `@harness-engineering/intelligence`. This applies the same Laplace-smoothed success rate (α = 1) the persona scorer uses, so a skill invoked once cannot claim 0% or 100% certainty. It surfaces the least-effective skills, failing skills, and abandoned-mid-workflow skills ranked sample-aware — the view you should act on when deciding what to fix or prune. In `--json` mode the same data appears under the `skillEffectiveness` key (`scores`, `failing`, `abandoned`). 4. If there are no records, say so plainly and stop: "No adoption telemetry found at `.harness/metrics/adoption.jsonl`. Nothing to retrospect on yet." ### Phase 2: INTERPRET — Signal vs. Noise Read the report with these discriminations: - **Coverage line first.** "N/total catalog skills have emitted telemetry" frames everything below it. A low ratio means the rankings describe the _instrumented_ slice, not the whole catalog. State the ratio explicitly in your summary. - **Failing skills are not automatically broken.** Some commands fail _by design_ (a gate like `ci.check` returning non-zero on a real violation is the gate working). Before flagging a high failure rate as a problem, note whether the skill is a checker/gate whose failures are expected. Call out only failures that look like defects (crashes, unexpected non-zero on clean input). - **Abandoned mid-workflow** uses the broadened definition: an explicit `abandoned` outcome, or a non-completed run that had already reached ≥1 phase. Small counts here are meaningful — a workflow skill people start and bail out of mid-way is a UX signal even at n=1. - **Stale skills** are drawn only from _ever-invoked_ skills quiet ≥ the threshold. When the record window is shorter than the threshold, the section says so and reports nothing — do not present that emptiness as "everything is healthy." - **Bayesian ranking over raw counts.** The top-failing and abandoned-mid-workflow tables rank by raw count; the Bayesian skill-effectiveness section ranks the same signals by Laplace-smoothed success rate, so a skill that failed 1/1 does not outrank one that failed 30/50. When the two disagree, trust the Bayesian view for prioritization — it discounts low-volume noise. A skill needs at least a couple of failures (or abandonments) before it appears in the Bayesian failing/abandoned tables at all. ### Phase 3: REPORT — Surface and Recommend 1. Confirm where the report was written (`docs/retrospectives/.md`) or present the rendered Markdown. 2. Summarize the 3–5 findings that actually warrant action, each tied to a section and a number. Prefer specifics ("`cli/review-ci` failed 23/23 — likely a real defect, worth a look") over restating the tables. 3. When the coverage ratio is the dominant finding (e.g. most of the catalog is uninstrumented), name that as the top follow-up: instrumentation, not pruning, is the lever. 4. Do not delete, prune, or edit skills from this skill — it reports. Hand any pruning decision to the human or to catalog-rationalization work. --- ## Harness Integration - **`harness adoption retrospective`** — the CLI entry point (a subcommand of the existing `adoption` command group). Writes `docs/retrospectives/.md` by default. - **`getCatalogRetrospectiveReport` / `renderRetrospectiveMarkdown` / `isAbandonedMidWorkflow`** — the pure aggregation + render functions in `@harness-engineering/core` (`packages/core/src/adoption/retrospective.ts`). Callers can supply a fixed `now` and `catalogSkills` for deterministic output. - **`readAdoptionRecords`** — reads and parses `.harness/metrics/adoption.jsonl` (read-only; the file is appended by the adoption-tracker hook, never by this skill). - **`computeSkillEffectiveness` / `detectFailingSkills` / `detectAbandonedSkills`** — the Bayesian skill scorer in `@harness-engineering/intelligence` (`packages/intelligence/src/effectiveness/skill-scorer.ts`). The skill-grain counterpart to the persona scorer: same Laplace smoothing and threshold-based detectors, applied to adoption records instead of graph outcome nodes. The retrospective command consumes these to render the Bayesian skill-effectiveness section and the `skillEffectiveness` JSON key. - **`harness adoption skills` / `recent` / `skill `** — the point-in-time siblings; this skill is the periodic, persisted counterpart. ## Success Criteria 1. The retrospective is derived from `.harness/metrics/adoption.jsonl` without mutating it. 2. The report ranks top-invoked, top-failing, and abandoned-mid-workflow skills, and flags ever-invoked stale skills. 3. The report includes a Bayesian skill-effectiveness section that ranks failing and abandoned-mid-workflow skills by Laplace-smoothed success rate, so low-volume skills do not dominate the recommendation. 4. The report states telemetry coverage (how much of the catalog emits any signal) when the catalog is discoverable. 5. The stale-skills section notes when the record window is shorter than the inactivity threshold instead of silently reporting zero. 6. The summary separates real signal (defect-shaped failures, mid-workflow abandonment) from telemetry gaps (uninstrumented skills) and from expected gate failures. 7. A dated report is written to `docs/retrospectives/.md` (unless `--no-write`/`--json` is requested). 8. The skill recommends but never prunes, edits, or deletes catalog skills. ## Rationalizations to Reject | Rationalization | Reality | | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | "756 skills have no telemetry, so 756 skills are abandoned — recommend deleting them." | Zero telemetry almost always means _uninstrumented_, not unused. The coverage line exists to prevent exactly this misread; the fix is instrumentation, not pruning. | | "`ci.check` fails 97% of the time — it's broken." | A gate returning non-zero on a real violation is the gate working. Check whether the skill is a checker before flagging its failures as defects. | | "The stale section is empty, so the catalog is healthy." | If the record window is shorter than the threshold, the signal simply cannot fire yet. Empty ≠ healthy; report the window caveat. | | "n=1 abandonment is noise, ignore it." | A workflow skill users start and bail out of mid-way is a UX signal even at low n. Surface it; let the human weigh it. | | "I'll prune the low-usage skills while I'm here." | This skill reports; it does not prune. Pruning is a human/catalog-rationalization decision. | ## Examples ### Example: Monthly retrospective on the dogfood repo **Context:** ~67 days of telemetry (3.2k records) across 50 invoked skills, run at month end. **Phase 1 — SCAN:** ```bash harness adoption retrospective # → Catalog retrospective written to docs/retrospectives/2026-07-06.md # 3246 records · 50 skills · window 67d ``` **Phase 2 — INTERPRET:** The report's overview leads with `Telemetry coverage: 5/761 catalog skills have emitted telemetry (756 never invoked)`. Top-invoked is dominated by `cli/*` commands; `cli/ci.check` shows a 97% failure rate; the abandoned-mid-workflow section lists four workflow skills at n=1; the stale section is empty with the note that the 67-day window is shorter than the 90-day threshold. **Phase 3 — REPORT:** ``` Retrospective written to docs/retrospectives/2026-07-06.md. Findings that warrant action: 1. Coverage is the headline: only 5/761 catalog skills emit any telemetry. The rankings describe the instrumented slice, not the catalog. Top follow-up is instrumentation, not pruning. 2. `cli/ci.check` failed 97% (721/741) — but it is a gate; non-zero on a real violation is expected. Not flagged as a defect. 3. `cli/review-ci` failed 23/23 (100%) — defect-shaped, worth a look. Routed to debugging. 4. Four workflow skills were abandoned mid-run (n=1 each) — low volume, a UX signal to watch. 5. Stale skills: none — the 67-day window can't satisfy the 90-day threshold yet. ``` No skills were pruned or edited — the pruning decision is left to the human. ## Gates - **No mutation of `adoption.jsonl`.** The telemetry file is read-only input. - **No catalog edits.** This skill never adds, edits, or deletes skills — it only reports on them. - **Coverage must be stated when discoverable.** If the catalog can be scanned, the summary must include the telemetry-coverage ratio so rankings are read in context. ## Escalation - **When there are no records:** State that telemetry is absent/empty and stop. Suggest confirming the adoption-tracker hook is installed if telemetry is unexpectedly missing. - **When coverage is very low:** Name instrumentation as the top follow-up and note that the rankings describe only the instrumented slice. - **When a failing skill looks like a real defect:** Flag it specifically (skill, failure count/rate, why it looks defect-shaped) and route it to debugging or an issue, rather than burying it in the table.