Agent Metrics Collection Utility
Purpose
This utility collects and reports metrics on agent usage from git history. It collects 4 of the metrics defined in docs/agent-metrics.md (Invocation Rate, Coverage, Infrastructure Review, Usage Distribution) for measuring agent system health, effectiveness, and adoption.
Triggers
| Trigger Phrase |
Operation |
collect agent metrics |
Run collect_metrics.py with default 30-day window |
generate metrics dashboard |
Run with markdown output for reporting |
check agent adoption rate |
Run and highlight Metric 2 (agent coverage) |
weekly metrics report |
Run with 7-day window, markdown output |
export metrics as JSON |
Run with JSON output for automation |
When to Use
Use this skill when:
- Measuring agent system health or adoption trends
- Producing periodic dashboards or reports
- Evaluating whether agent usage is balanced across types
- Checking infrastructure review coverage
Use manual git log inspection instead when:
- Investigating a single commit's agent attribution
- Debugging a specific CI run's metrics workflow
Process
- Run the metrics collection script for the desired time range
- Review generated reports for agent usage patterns
- Identify trends and anomalies in adoption metrics
Anti-Patterns
| Avoid |
Why |
Instead |
| Running without specifying time window |
Default 30 days may not match your intent |
Use --since with explicit day count |
| Comparing metrics across different time windows |
Misleading trends |
Normalize to same window size |
| Ignoring zero agent coverage |
Indicates broken detection patterns |
Verify commit message conventions match patterns |
| Manual commit counting |
Error-prone, misses patterns |
Use the script for consistent detection |
| Storing JSON output without markdown |
Loses human-readable context |
Generate both formats for archival |
Verification
After execution:
Available Scripts
| Script |
Platform |
Usage |
collect_metrics.py |
Python 3.8+ |
Cross-platform |
backlog_provenance.py |
Python 3.10+, gh |
Read-only issue provenance report |
defect_lineage.py |
Python 3.10+, git |
Check, report, and recheck defect lineage records |
Quick Start
# Basic usage (30 days, summary output)
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics/collect_metrics.py"
# Last 90 days as markdown
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics/collect_metrics.py" --since 90 --output markdown
# JSON output for automation
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics/collect_metrics.py" --output json
# Backlog provenance of issues created in the last 7 days (Markdown)
SCRIPTS_DIR="${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics"
python "$SCRIPTS_DIR/backlog_provenance.py" --days 7
# Same report as JSON, for an explicit UTC interval ending 2026-09-29 (exclusive)
python "$SCRIPTS_DIR/backlog_provenance.py" --days 29 --until 2026-09-29T00:00:00Z --output-format json
# Defect lineage records (this repo keeps them in .project-toolkit/metrics/lineage/):
# validate, report, and recheck against origin/<default_branch> (run git fetch origin first)
python "$SCRIPTS_DIR/defect_lineage.py" check <records-dir>
python "$SCRIPTS_DIR/defect_lineage.py" report <records-dir>
python "$SCRIPTS_DIR/defect_lineage.py" recheck <records-dir>/<id>.json --write
# --root belongs to the top-level parser, so it goes before the subcommand
python "$SCRIPTS_DIR/defect_lineage.py" --root <repo> recheck <records-dir>/<id>.json --write
Metrics Collected
The utility collects the following metrics:
| Metric |
Description |
Target |
| Metric 1: Invocation Rate |
Agent usage distribution |
Proportional to task types |
| Metric 2: Agent Coverage |
% of commits with agent involvement |
50% |
| Metric 4: Infrastructure Review |
% of infra changes with security review |
100% |
| Metric 5: Usage Distribution |
Agent utilization patterns |
Balanced distribution |
Backlog Provenance
backlog_provenance.py is a read-only diagnostic for the backlog provenance epic. It reads every
issue created in an explicit UTC interval and reports the split by recorded
source:human and source:agent label, plus two labeled heuristics. It does not
score productivity, and it never files, labels, or closes an issue.
- Buckets are mutually exclusive:
human-only, agent-only, conflict (both labels), unknown (neither).
- Labels record provenance. They do not verify who selected the work.
- Machinery share matches a whole-word title term or the
area-validation label. Bursts are 3 or more issues by one login within 10 minutes of the first. Both are heuristics, not causal attribution.
- The read pages until an empty page. Any failed or malformed page exits 3 and no partial report prints. Pull requests are excluded and issues are deduplicated by number.
- Markdown is the default on every terminal and redirect. Empty ratios print
N/A, or null in JSON.
Baseline, reproducible with --days 29 --until 2026-09-29T00:00:00Z (UTC, all issue states, retrieved 2026-09-29): 212 issues created 2026-08-31 through 2026-09-28. All 212 are unknown because no source:* label existed yet on that history. 87 (41.0%) match the machinery heuristic and 16 bursts cover 79 issues. Reconciled against the GitHub search API created:2026-08-31..2026-09-28 count of 212. The earlier epic figure of 405 issues created in 30 days (2026-09-10 snapshot, about 9 in 10 machinery by title) is a historical claim. It used a different title rule and window, so it is not comparable to this run. The weekly cost-governance review cites this script's output.
Defect Lineage
A later fix makes an old defect vanish from the current tree. A report of only
the current state then loses the evidence of the review race, and a report of
only the old defect calls a fixed problem live. defect_lineage.py keeps the
chain. One JSON record per reviewed chain lives in a records
directory that each command takes as an argument. The schema and its rules are
in the docstring of defect_lineage_record.py.
- A record holds the introducing PR and merge time, the detection source and time, the follow-up PR and merge time, the live status, a recheck, and its limitations. Timestamps are UTC to the second, so a review 5 s after a merge orders correctly.
supports says what the chain shows: defect, review_race, process_gap, or observation. Record planned work with supports: ["observation"] and follow_up.kind: planned_work. A follow-up counts as a defect only when the record says so.
- Status is
CURRENT, FIXED, REVERTED, SUPERSEDED, or UNVERIFIED. It says whether the record's claim holds at the recorded live SHA. REVERTED means the claim holds again after a recorded reverted_by. SUPERSEDED means a recorded superseded_by replaced the chain.
recheck reads recheck.path at a revision with git show and matches recheck.pattern. It appends one status_history entry and never deletes the chain. A missing file gives UNVERIFIED, never FIXED. So does a pattern that stops matching when the record has no merged defect_repair follow-up and no superseded_by (for example, the code moved). --rev must be a branch, tag, or SHA: HEAD~1 and main^ are rejected as unsafe with exit 2. An unresolved revision exits 3 and records nothing.
report groups records into live, unverified, historical, and observation buckets. Live holds CURRENT and REVERTED. Unverified holds UNVERIFIED. Historical holds FIXED and SUPERSEDED. Observation holds every observation-only record, whatever its status. Only the live bucket holds findings present at the recorded SHA.
check rejects contradictory records, including: a review race that did not follow the merge, a repair merged before detection, FIXED without a merged repair, a stored delay that disagrees with the timestamps, and out-of-order history, a follow-up merged before the introduction, and a FIXED history entry dated before the follow-up merged.
- The records cite PR, issue, and review URLs. They do not copy bodies, and they do not blame a model or reviewer for a late review.
Exit codes: 0 ok, 1 invalid record, 2 bad path, JSON, or argument, 3 git cannot run or the revision does not resolve.
Detection Patterns
Agent Detection
The utility detects agents in commit messages using these patterns:
- Direct agent names:
orchestrator, analyst, architect, etc.
- Review attribution:
Reviewed by: security
- Agent tags:
agent: implementer or [security-agent]
Infrastructure Files
Infrastructure commits are identified by these patterns:
.github/workflows/*.{yml,yaml}
.github/actions/**
- Root
lefthook and .lefthook configs, with optional -local suffix
.config/lefthook configs, with optional -local suffix
- Lefthook config extensions:
.yml, .yaml, .json, .jsonc, .toml
build/**, scripts/**
Dockerfile*
docker-compose*
*.tf, *.tfvars
.env*
.agents/**
Commit Types
Conventional commit prefixes are classified:
feat: - Feature
fix: - Bug fix
docs: - Documentation
ci: - CI/CD
refactor: - Refactoring
Output Formats
Summary (Default)
Human-readable console output with key metrics highlighted.
Markdown
Formatted markdown suitable for dashboards and reports. Can be saved directly to .project-toolkit/metrics/ for archival.
JSON
Structured data for programmatic consumption and CI integration.
CI Integration
See .github/workflows/agent-metrics.yml for automated weekly metrics collection.
The workflow:
- Runs weekly on Sundays
- Collects metrics for the previous 7 days
- Generates a markdown report
- Creates a PR with the report (if significant changes)
Manual Report Generation
To generate a monthly dashboard report:
# Generate report
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics/collect_metrics.py" \
--since 30 \
--output markdown \
> .project-toolkit/metrics/report-$(date +%Y-%m).md
# Review and commit
git add .project-toolkit/metrics/
git commit -m "docs(metrics): add monthly metrics report"
Extending the Utility
Adding New Metrics
- Define the metric in
docs/agent-metrics.md
- Add collection logic to
collect_metrics.py
- Update the output formatters
- Add tests if applicable
Adding New Agent Patterns
Update the AGENT_PATTERNS list to detect new agent references.
Adding Infrastructure Patterns
Update the INFRASTRUCTURE_PATTERNS list for new infrastructure file types.
Troubleshooting
No Agents Detected
- Ensure commit messages reference agents explicitly
- Check that conventional commit format is used
- Verify the patterns match your team's conventions
Git Errors
- Confirm you're in a git repository
- Check that the repository has commits in the date range
- Verify git is available in PATH
Related Documents
Backticked paths below are in the rjmurillo/ai-agents repository. They do not ship with this skill; a consumer install cannot resolve them.
docs/agent-metrics.md. Agent metrics definitions.
.project-toolkit/metrics/dashboard-template.md. Dashboard template.
.project-toolkit/metrics/baseline-report.md. Baseline report.
.github/workflows/agent-metrics.yml. CI workflow.
1---2name: metrics-33description: Collect agent usage metrics from git history and generate health reports. Use when measuring agent adoption, reviewing system health, or producing periodic dashboards. Collects Invocation Rate, Coverage, Infrastructure Review, and Usage Distribution. Use when you say "collect agent metrics", "generate metrics dashboard", or "weekly metrics report".4metadata:5 source: https://skillmd.com/skills/rjmurillo/metrics-36---78# Agent Metrics Collection Utility910## Purpose1112This utility collects and reports metrics on agent usage from git history. It collects 4 of the metrics defined in `docs/agent-metrics.md` (Invocation Rate, Coverage, Infrastructure Review, Usage Distribution) for measuring agent system health, effectiveness, and adoption.1314## Triggers1516| Trigger Phrase | Operation |17|----------------|-----------|18| `collect agent metrics` | Run collect_metrics.py with default 30-day window |19| `generate metrics dashboard` | Run with markdown output for reporting |20| `check agent adoption rate` | Run and highlight Metric 2 (agent coverage) |21| `weekly metrics report` | Run with 7-day window, markdown output |22| `export metrics as JSON` | Run with JSON output for automation |2324---2526## When to Use2728Use this skill when:2930- Measuring agent system health or adoption trends31- Producing periodic dashboards or reports32- Evaluating whether agent usage is balanced across types33- Checking infrastructure review coverage3435Use manual git log inspection instead when:3637- Investigating a single commit's agent attribution38- Debugging a specific CI run's metrics workflow3940---4142## Process43441. Run the metrics collection script for the desired time range452. Review generated reports for agent usage patterns463. Identify trends and anomalies in adoption metrics4748---4950## Anti-Patterns5152| Avoid | Why | Instead |53|-------|-----|---------|54| Running without specifying time window | Default 30 days may not match your intent | Use --since with explicit day count |55| Comparing metrics across different time windows | Misleading trends | Normalize to same window size |56| Ignoring zero agent coverage | Indicates broken detection patterns | Verify commit message conventions match patterns |57| Manual commit counting | Error-prone, misses patterns | Use the script for consistent detection |58| Storing JSON output without markdown | Loses human-readable context | Generate both formats for archival |5960---6162## Verification6364After execution:6566- [ ] Script exits with code 067- [ ] Output contains all 4 collected metrics (Invocation Rate, Coverage, Infrastructure Review, Distribution)68- [ ] Agent coverage percentage is plausible (not 0% unless truly no agent commits)69- [ ] Time window matches intended period70- [ ] For markdown output: report file created at expected path7172---7374## Available Scripts7576| Script | Platform | Usage |77|--------|----------|-------|78| `collect_metrics.py` | Python 3.8+ | Cross-platform |79| `backlog_provenance.py` | Python 3.10+, `gh` | Read-only issue provenance report |80| `defect_lineage.py` | Python 3.10+, `git` | Check, report, and recheck defect lineage records |8182## Quick Start8384```bash85# Basic usage (30 days, summary output)86python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics/collect_metrics.py"8788# Last 90 days as markdown89python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics/collect_metrics.py" --since 90 --output markdown9091# JSON output for automation92python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics/collect_metrics.py" --output json9394# Backlog provenance of issues created in the last 7 days (Markdown)95SCRIPTS_DIR="${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics"96python "$SCRIPTS_DIR/backlog_provenance.py" --days 79798# Same report as JSON, for an explicit UTC interval ending 2026-09-29 (exclusive)99python "$SCRIPTS_DIR/backlog_provenance.py" --days 29 --until 2026-09-29T00:00:00Z --output-format json100101# Defect lineage records (this repo keeps them in .project-toolkit/metrics/lineage/):102# validate, report, and recheck against origin/<default_branch> (run git fetch origin first)103python "$SCRIPTS_DIR/defect_lineage.py" check <records-dir>104python "$SCRIPTS_DIR/defect_lineage.py" report <records-dir>105python "$SCRIPTS_DIR/defect_lineage.py" recheck <records-dir>/<id>.json --write106107# --root belongs to the top-level parser, so it goes before the subcommand108python "$SCRIPTS_DIR/defect_lineage.py" --root <repo> recheck <records-dir>/<id>.json --write109```110111## Metrics Collected112113The utility collects the following metrics:114115| Metric | Description | Target |116|--------|-------------|--------|117| Metric 1: Invocation Rate | Agent usage distribution | Proportional to task types |118| Metric 2: Agent Coverage | % of commits with agent involvement | 50% |119| Metric 4: Infrastructure Review | % of infra changes with security review | 100% |120| Metric 5: Usage Distribution | Agent utilization patterns | Balanced distribution |121122## Backlog Provenance123124`backlog_provenance.py` is a read-only diagnostic for the backlog provenance epic. It reads every125issue created in an explicit UTC interval and reports the split by recorded126`source:human` and `source:agent` label, plus two labeled heuristics. It does not127score productivity, and it never files, labels, or closes an issue.128129- Buckets are mutually exclusive: `human-only`, `agent-only`, `conflict` (both labels), `unknown` (neither).130- Labels record provenance. They do not verify who selected the work.131- Machinery share matches a whole-word title term or the `area-validation` label. Bursts are 3 or more issues by one login within 10 minutes of the first. Both are heuristics, not causal attribution.132- The read pages until an empty page. Any failed or malformed page exits 3 and no partial report prints. Pull requests are excluded and issues are deduplicated by number.133- Markdown is the default on every terminal and redirect. Empty ratios print `N/A`, or `null` in JSON.134135Baseline, reproducible with `--days 29 --until 2026-09-29T00:00:00Z` (UTC, all issue states, retrieved 2026-09-29): 212 issues created 2026-08-31 through 2026-09-28. All 212 are `unknown` because no `source:*` label existed yet on that history. 87 (41.0%) match the machinery heuristic and 16 bursts cover 79 issues. Reconciled against the GitHub search API `created:2026-08-31..2026-09-28` count of 212. The earlier epic figure of 405 issues created in 30 days (2026-09-10 snapshot, about 9 in 10 machinery by title) is a historical claim. It used a different title rule and window, so it is not comparable to this run. The weekly cost-governance review cites this script's output.136137## Defect Lineage138139A later fix makes an old defect vanish from the current tree. A report of only140the current state then loses the evidence of the review race, and a report of141only the old defect calls a fixed problem live. `defect_lineage.py` keeps the142chain. One JSON record per reviewed chain lives in a records143directory that each command takes as an argument. The schema and its rules are144in the docstring of `defect_lineage_record.py`.145146- A record holds the introducing PR and merge time, the detection source and time, the follow-up PR and merge time, the live status, a recheck, and its limitations. Timestamps are UTC to the second, so a review 5 s after a merge orders correctly.147- `supports` says what the chain shows: `defect`, `review_race`, `process_gap`, or `observation`. Record planned work with `supports: ["observation"]` and `follow_up.kind: planned_work`. A follow-up counts as a defect only when the record says so.148- Status is `CURRENT`, `FIXED`, `REVERTED`, `SUPERSEDED`, or `UNVERIFIED`. It says whether the record's `claim` holds at the recorded live SHA. `REVERTED` means the claim holds again after a recorded `reverted_by`. `SUPERSEDED` means a recorded `superseded_by` replaced the chain.149- `recheck` reads `recheck.path` at a revision with `git show` and matches `recheck.pattern`. It appends one `status_history` entry and never deletes the chain. A missing file gives `UNVERIFIED`, never `FIXED`. So does a pattern that stops matching when the record has no merged `defect_repair` follow-up and no `superseded_by` (for example, the code moved). `--rev` must be a branch, tag, or SHA: `HEAD~1` and `main^` are rejected as unsafe with exit 2. An unresolved revision exits 3 and records nothing.150- `report` groups records into live, unverified, historical, and observation buckets. Live holds `CURRENT` and `REVERTED`. Unverified holds `UNVERIFIED`. Historical holds `FIXED` and `SUPERSEDED`. Observation holds every observation-only record, whatever its status. Only the live bucket holds findings present at the recorded SHA.151- `check` rejects contradictory records, including: a review race that did not follow the merge, a repair merged before detection, `FIXED` without a merged repair, a stored delay that disagrees with the timestamps, and out-of-order history, a follow-up merged before the introduction, and a `FIXED` history entry dated before the follow-up merged.152- The records cite PR, issue, and review URLs. They do not copy bodies, and they do not blame a model or reviewer for a late review.153154Exit codes: 0 ok, 1 invalid record, 2 bad path, JSON, or argument, 3 git cannot run or the revision does not resolve.155156## Detection Patterns157158### Agent Detection159160The utility detects agents in commit messages using these patterns:161162- Direct agent names: `orchestrator`, `analyst`, `architect`, etc.163- Review attribution: `Reviewed by: security`164- Agent tags: `agent: implementer` or `[security-agent]`165166### Infrastructure Files167168Infrastructure commits are identified by these patterns:169170- `.github/workflows/*.{yml,yaml}`171- `.github/actions/**`172- Root `lefthook` and `.lefthook` configs, with optional `-local` suffix173- `.config/lefthook` configs, with optional `-local` suffix174- Lefthook config extensions: `.yml`, `.yaml`, `.json`, `.jsonc`, `.toml`175- `build/**`, `scripts/**`176- `Dockerfile*`177- `docker-compose*`178- `*.tf`, `*.tfvars`179- `.env*`180- `.agents/**`181182### Commit Types183184Conventional commit prefixes are classified:185186- `feat:` - Feature187- `fix:` - Bug fix188- `docs:` - Documentation189- `ci:` - CI/CD190- `refactor:` - Refactoring191192## Output Formats193194### Summary (Default)195196Human-readable console output with key metrics highlighted.197198### Markdown199200Formatted markdown suitable for dashboards and reports. Can be saved directly to `.project-toolkit/metrics/` for archival.201202### JSON203204Structured data for programmatic consumption and CI integration.205206## CI Integration207208See `.github/workflows/agent-metrics.yml` for automated weekly metrics collection.209210The workflow:2112121. Runs weekly on Sundays2132. Collects metrics for the previous 7 days2143. Generates a markdown report2154. Creates a PR with the report (if significant changes)216217## Manual Report Generation218219To generate a monthly dashboard report:220221```bash222# Generate report223python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/metrics/collect_metrics.py" \224 --since 30 \225 --output markdown \226 > .project-toolkit/metrics/report-$(date +%Y-%m).md227228# Review and commit229git add .project-toolkit/metrics/230git commit -m "docs(metrics): add monthly metrics report"231```232233## Extending the Utility234235### Adding New Metrics2362371. Define the metric in `docs/agent-metrics.md`2382. Add collection logic to `collect_metrics.py`2393. Update the output formatters2404. Add tests if applicable241242### Adding New Agent Patterns243244Update the `AGENT_PATTERNS` list to detect new agent references.245246### Adding Infrastructure Patterns247248Update the `INFRASTRUCTURE_PATTERNS` list for new infrastructure file types.249250## Troubleshooting251252### No Agents Detected253254- Ensure commit messages reference agents explicitly255- Check that conventional commit format is used256- Verify the patterns match your team's conventions257258### Git Errors259260- Confirm you're in a git repository261- Check that the repository has commits in the date range262- Verify git is available in PATH263264## Related Documents265266Backticked paths below are in the `rjmurillo/ai-agents` repository. They do not ship with this skill; a consumer install cannot resolve them.267268- `docs/agent-metrics.md`. Agent metrics definitions.269- `.project-toolkit/metrics/dashboard-template.md`. Dashboard template.270- `.project-toolkit/metrics/baseline-report.md`. Baseline report.271- `.github/workflows/agent-metrics.yml`. CI workflow.272273<!-- vendor-portability: declared. This skill reads the consumer's .agents/* artifacts as metric inputs and can archive formatted output to .project-toolkit/metrics/. Inputs are whatever the consumer repo contains; the archive path is an optional write target created on demand. It also cites docs/agent-metrics.md, .project-toolkit/metrics/dashboard-template.md, .project-toolkit/metrics/baseline-report.md, and .github/workflows/agent-metrics.yml as background reading. -->