Dev AI Coding Metrics

AI Coding Metrics

Measures coding assistants and coding agents without collapsing results into vanity metrics or one blended score.

The critical distinction is mode: assistants help inline or in chat; agents execute multi-step work and need task-level measurement. Do not measure them as if they were the same thing.

When to Use This Skill

Defaults

Workflow

1. Define the decision.
2. Pick the program mode: assistant, agent, or mixed.
3. Build the minimum viable scorecard.
4. Choose the study design.
5. Produce one deliverable.

ASCII Flow

AI coding metrics request
  -> decision to support: buy, renew, improve, prove, or diagnose
  -> split mode: assistant, agent, or mixed
  -> select scorecard families: adoption, delivery, quality, economics, experience
  -> choose study design and baseline window
  -> collect team-level and task-level evidence
  -> report confidence, sample size, and confounds
  -> deliver ROI model, dashboard, experiment plan, or executive report

Quick Reference

Decision to Deliverable Map

Program Modes

Metric Families

Use the smallest scorecard that can answer the decision:

Study Design Defaults

Minimum baseline: 8 weeks of pre-intervention data. Two-week baselines produce noisy causal inference — week-to-week variance in PR throughput, review lag, and defect escape routinely exceeds the signal size of AI tooling effects.

Measurement Checklist

Use before publishing any AI coding report:

• Baseline established (≥8 weeks before intervention)
• Assistant and agent funnels tracked separately
• Every speed metric paired with at least one quality metric
• Sample size, confidence level, and study design stated
• Confounds documented (team changes, release pressure, policy changes)
• Vendor evidence labeled as vendor evidence
• Usage measured after stabilization (not week-1 novelty period)
• Review burden and rework cost included in ROI model
• Edit-capable agents measured across evolving-spec checkpoints, including late-checkpoint cost and quality slopes
• Aggregated at team level (no manager-visible individual dashboards)

Current Evidence Posture (as of 2026-08-21)

Anti-Gaming Checklist

Reject a scorecard or report if any of the following apply:

• Single blended AI productivity score mixing usage, speed, sentiment, and quality
• Seat activation or prompt volume cited as delivery impact
• Cross-team comparison without controlling for stack, task mix, staffing, or release pressure
• Measurement period is <8 weeks or includes week-1 novelty window
• Vendor benchmark cited as production ROI evidence
• Review burden excluded from ROI model
• Individual-level AI usage visible to managers
• Directional before/after movement stated as causal without controlled design
• SlopCodeBench averages or trajectory signals used as organizational targets or causal ROI evidence

Navigation

References

• references/adoption-metrics.md — assistant and agent adoption funnels, metric definitions, stall patterns, privacy rules
• references/productivity-metrics.md — DORA and SPACE applied to AI workflows, delivery stack decomposition, confound management
• references/quality-metrics.md — defect, complexity, test, security, and technical debt metrics with targets and alert thresholds
• references/roi-framework.md — full cost model (including review burden), benefit model, scenario planning, executive report structure
• references/developer-experience-metrics.md — satisfaction surveys, cognitive load, friction indicators, trust calibration, DX anti-patterns
• references/agent-execution-metrics.md — agent funnel, core metrics, reviewer burden, scorecards for pilot / scaling / executive decisions
• references/benchmarking-methodology.md — A/B, before/after, crossover, shadow designs; statistical rigor; confound management
• references/theory-of-constraints-applied.md — bottleneck identification before instrumenting, throughput accounting for ROI, DBR for review-queue protection, CRT for stalled rollouts, evaporating cloud for adoption-vs-quality tensions
• references/evidence-update.md — load when citing current research: METR RCT (2025 baseline), METR 2026 update (selection-bias caveat), DORA 2025 AI report, DX Core 4, Faros 2026 telemetry

Assets and data

• assets/metric-dashboard-template.md
• assets/adoption-survey-template.md
• assets/roi-calculator-template.md
• assets/executive-report-template.md
• assets/experiment-design-template.md
• data/sources.json
• data/sample-ai-metrics.json

Scripts

• scripts/roi_calculator.py
• scripts/README.md

Cross-References

• ../dev-context-engineering/SKILL.md
• ../ai-agents/SKILL.md
• ../qa-observability/SKILL.md
• ../product-management/SKILL.md

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Verify current research claims, benchmark status, and vendor telemetry specifics before final advice.
• Prefer peer-reviewed, official, and first-party telemetry docs over social or vendor marketing claims.
• If live verification is unavailable, mark current-evidence claims as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.