Performance Regression
Establish whether the reported workload regressed, which operation multiplied, and whether the benchmark can distinguish the known-good and known-bad builds. Preserve the user's requested scope: an investigation does not authorize production fixes, publishing reports, or cloud operations.
Evidence and workload selection
Search docs/fixes for the affected command, subsystem, and symptoms. Use the relevant fix logs as the source for incident history, reproduction conditions, and known-good/bad revisions; verify those details against the implementation and current behavior.
Record the command, flags, scale, configuration shape, and external operations that trigger the slowdown. Reproduce the reported command itself: a faster neighboring command does not establish that the affected path improved.
Use the suite coverage and field-test results in magefiles/README.md to select or extend a workload. Preserve the conditions that reach the expensive path, verify expected outputs outside timing, and count repeated operations where useful. State which parts of the original workload a local fixture cannot reproduce.
Measure through Mage
Keep automation in magefiles, benchmark workloads in Go, and scratch results
under .context/ or build/benchmarks/. The maintained command contract is in
the Mage README; do not add parallel shell orchestration to the skill.
Collect a new candidate with go tool mage bench:run <candidateDir>.
Builds and setup stay outside measurements. Do not run compilation, tests, or
other heavy work concurrently with timed samples. CLI samples use fresh processes
and isolated caches; harness allocations are not CLI memory measurements.
Retrieve historical benchmarks
Stored measurements are GitHub Actions artifacts from the
Benchmarks workflow.
For an existing PR comparison, follow the sticky comment's measurements link
and download that run's benchmarks-<runID>-<attempt> artifact into
build/benchmarks/candidate/, preserving bench.txt and manifest.json together.
Use its recorded BaseSHA as <main-base-sha> below.
With GITHUB_TOKEN available, retrieve history and compare through Mage:
go tool mage bench:fetch cloudposse/atmos <main-base-sha> build/benchmarks/history
go tool mage bench:compare build/benchmarks/history build/benchmarks/candidate build/benchmarks/report
Fetch expects the sibling candidate/ directory; set
ATMOS_BENCH_CANDIDATE_DIR when using another location. The history directory
must be empty. Fetch selects up to ten compatible, successful main snapshots
reachable from that immutable SHA and preceding the candidate's cutoff. PR
artifacts never qualify as main baselines. A local candidate needs a compatible
runner, CPU, toolchain, build settings, and workload; a Mac run cannot compare
to Linux CI snapshots.
Inspect report.md, report.json, and each benchstat-*.txt in the report
directory. Downloaded snapshots retain their measurements and provenance;
history/unavailable.json explains skipped or unavailable history. Preserve
compatibility checks and sample counts. Every snapshot is compared independently
to the same candidate measurements.
Artifacts expire after 90 days, and history only starts accumulating after the workflow is deployed. Missing or expired results mean comparison unavailable; malformed or incomplete measurements are failures. When stored history does not exist, explicitly rebuild selected revisions as a field test:
go tool mage bench:fieldTest <good-ref>,<bad-ref>,<fixed-ref> .context/performance-field-test
The field-test target uses the same current CLI harness for each build. Select
the relevant workload with ATMOS_BENCH_FIELD_FILTER; see the Mage README for
supported settings. Use fresh output directories. Label rebuilt results as
field-test evidence; do not present them as historical CI snapshots.
Interpret and close the loop
Report medians, percentage changes, sample counts, significance, backend-read
counts when relevant, and fixture/environment limitations. The advisory rule is
at least a 10% median increase with p < 0.05; read the actual configured threshold.
The newest compatible snapshot drives the headline; warnings against older
snapshots still matter and are correlated, not independent confirmations.
If a known slowdown is absent, inspect workload coverage before concluding it was fixed. Demonstrate that the benchmark distinguishes the known-good and known-bad builds. Preserve residual warnings after a large improvement, and do not relax a budget merely to hide a regression.
When fixes are in scope, retain output/caching correctness checks, then remeasure
the same workload. Add the verified reproducer to the curated suite and its
required inventory; record incident-specific findings in docs/fixes and update
the suite coverage documentation. Keep that history out of this skill.
Performance warnings are advisory; do not manually set Quality Gate labels or
rewrite the reporter's sticky comment to make an investigation appear complete.
