Traust Metrics

Traust Metrics

Paths. analysis-results/… and progress-tracker/… in this skill are the default workspace layout. They resolve through locations.yaml in $TRAUST_CONFIG_HOME (docs/setup.md, Storage locations); substitute your configured roots.

Maintain one Markdown scoreboard — progress-tracker/metrics/traust-metrics.md — holding every number the team quotes in leadership decks and status reports, so those numbers are regenerated from the pipeline's own artifacts instead of being copied around and going stale.

The skill never authors a metric. collect_harness_metrics.py parses each value from a generated artifact (or counts it from the harness tree) and cites the source next to it; anything it cannot find renders as "—" and is flagged in the data-freshness table. Narrative claims no artifact can compute live in a manual block that survives regeneration.

Input

$ARGUMENTS is optional and may be any combination of:

Procedure

1. Run the collector:

   python3 traust/harnessing/traust-metrics/scripts/collect_harness_metrics.py
   

   It writes the scoreboard and exits non-zero if any source artifact is missing (the file is still written, with the gaps flagged).

2. Check the Data freshness table in the output. Each source row shows the artifact's own Generated: date. If a source is missing, or older than the latest audit activity and the user asked for --refresh-sources (or asks for current numbers), rebuild it with the owning skill first, then rerun the collector:

   The collector reads eleven sources (this table mirrors its sources dict — keep them in sync; docs-verification 2026-07-31 found it documenting 4 of 11):

   Stale source	Rebuild with
   progress-tracker/metrics/dashboards/Executive-summary-findings.metrics.json (machine sidecar; the .md is the human companion)	/executive-summary-findings
   progress-tracker/metrics/dashboards/census/census.json	/census
   progress-tracker/metrics/dashboards/insecure-patterns/insecure-patterns.json	/insecure-patterns
   progress-tracker/metrics/dashboards/benchmark/benchmark.json	/recall-benchmark (a full benchmark run — never rebuild casually; report staleness)
   progress-tracker/metrics/dashboards/trends/trends.json	/findings-trends
   progress-tracker/metrics/dashboards/Live-validation-fuzz-dashboard.md	/validation-fuzz-dashboard
   fuzz campaign sidecar (fuzz-campaign.json, beside the validation dashboard)	/validation-fuzz-dashboard
   progress-tracker/metrics/dashboards/threat-register/threat-register.md	/threat-register
   progress-tracker/tracking/opened-tickets.md	human-curated by the filing flow (/file-security-defect reporting) — report staleness, never rebuild
   cloud-config lane (findings.db)	/census (findings-db projection)
   harness repo tree/git metadata	nothing to rebuild — read live

   When several sources are stale at once, prefer the full chain over per-artifact rebuilds: /refresh-dashboards (python3 -m traust.cli dashboard refresh) rebuilds every deterministic dashboard in dependency order and runs this collector last.

3. Report back. Read the generated file and give the user the headline numbers grouped as the file groups them (scale, impact, validation, delivery, maturity), calling out: any MISSING sources, any source whose Generated: date is more than ~a week old, and any metric that moved sharply since the previous version of the file (use git -C progress-tracker diff traust-metrics.md when the file is tracked).

4. Manual block (optional). A hand-added <!-- MANUAL:BEGIN/END --> block is preserved verbatim across runs. If the user supplies narrative claims (cost comparisons, remediation stories, the ask), edit them inside that block only — never hand-edit generated rows; fix the upstream artifact instead.

Metrics history — the append-only improvement ledger

Every run (unless --no-history) appends one immutable snapshot row to progress-tracker/metrics/metrics-history.jsonl and re-renders metrics/metrics-history.md — a metrics × snapshots table with per-snapshot notes. This is the campaign's as-reported record: prior rows are NEVER revised, so the original raw numbers (e.g. the 2026-07-14 pre-triage baseline) survive every counting-policy change, and methodology shifts appear as explained steps (via --note "...") instead of silent rewrites. This matters because every other view rewrites history by design: regenerated dashboards overwrite in place, and findings-trends replays by occurred_at, so late event ingestion (e.g. a ledger backfill) retroactively changes past buckets. Pass --note with one line of context whenever a run follows a methodology or pipeline change. Never hand-edit or prune the JSONL.

Output

progress-tracker/metrics/traust-metrics.md — sections: Scale & coverage · Findings & impact · Empirical validation · Delivery & adoption · Platform maturity & reliability · optional manual block (preserved if present) · Data freshness.

Notes

• Deterministic at metric time: like the dashboards, this skill only replays already-derived artifacts — no model call participates in producing a number, so reruns are reproducible and diffs are meaningful.
• The maturity metrics (version, commits, releases, LoC, test count) are counted live from the harness checkout, so they are correct even when no campaign artifact has been rebuilt recently.
• Severity counts quote the executive summary's totals (occurrences) and note unique counts alongside, matching how that report itself leads.

Spend metrics (separate dashboard, not harvested here)

The collector does not read spend data and the scoreboard carries no spend section — spend lives in its own dashboard (progress-tracker/metrics/dashboards/spend/, rebuilt by python3 -m traust.cli metrics spend, stage 14 of /refresh-dashboards). An earlier revision of this skill claimed the scoreboard harvested model-spend:* ledger rows; no such code ever existed (docs-verification 2026-07-31, wiring c5). When reporting scoreboard numbers, point spend questions at the spend dashboard; treat a

7-day-old spend dashboard as stale and rebuild via /refresh-dashboards. Real-time view and the full tracking chain live in your deployment's private configuration repository, if it keeps one.