Perf Autofiler Triage

Performance Auto-Filer Triage

Systematically parse and triage performance regression issues created by the dotnet/performance auto-filing system in the dotnet/perf-autofiling-issues repository.

Auto-Filed Issue Structure

Issues in dotnet/perf-autofiling-issues follow a standard format:

Title: [regression] System.Numerics.Tests.Perf_Vector128<Int32>.Benchmark
Body:
  Benchmark: System.Numerics.Tests.Perf_Vector128<Int32>.Benchmark
  Profile: ...
  Regression: X.XX (ratio)
  Baseline commit: <sha>
  Compare commit: <sha>
  Baseline run: <date>
  Compare run: <date>
  Configuration: CompilationMode=wasm, RunKind=micro, ...

Key Fields to Extract

Comments May Contain Grouped Regressions

The auto-filer often groups related regressions into comments on the same issue. Always read ALL comments — the issue body may show 1 regression but comments may list 50+.

Step 1: Parse the Issue

1. Fetch the issue with `gh issue view <number> --repo dotnet/perf-autofiling-issues`
2. Fetch ALL comments with `gh issue view <number> --repo dotnet/perf-autofiling-issues --comments`
3. Extract every benchmark name and regression ratio
4. Extract the baseline and compare commit SHAs
5. Extract the date range (baseline run → compare run)
6. Note the configuration (especially CompilationMode, RunKind, Architecture)

Step 2: Categorize by Severity Tier

SQL Tracking Template

CREATE TABLE IF NOT EXISTS regression_categories (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  issue_number INT,
  benchmark TEXT,
  ratio REAL,
  tier INT,
  category TEXT,      -- 'simd', 'general', 'noise', 'infrastructure'
  status TEXT DEFAULT 'untriaged',  -- 'untriaged', 'investigating', 'infrastructure', 'real', 'noise'
  notes TEXT
);

-- Example categorization query
SELECT tier, category, COUNT(*) as count, 
       ROUND(MAX(ratio), 2) as max_ratio, 
       ROUND(AVG(ratio), 2) as avg_ratio
FROM regression_categories
WHERE issue_number = 69444
GROUP BY tier, category
ORDER BY tier, count DESC;

Step 3: Detect Data Gaps

Data gaps are the #1 cause of false positives. A data gap occurs when perf runs are broken for an extended period, then resume with different tooling.

How to Detect

1. Check the date range: baseline run date vs compare run date
2. If gap > 2 weeks, investigate why runs were missing
3. Common causes:
   • New .NET version broke BDN (net11.0 bump, net12.0 bump)
   • Perf machine pool change
   • BDN version upgrade with breaking changes
   • WASM workload not available for new SDK preview

How to Verify

# Check dotnet/performance PRs in the gap window
gh pr list --repo dotnet/performance --state merged \
  --search "merged:2025-12-01..2026-03-01" \
  --limit 50 --json number,title,mergedAt

# Look for framework version bumps
gh search prs --repo dotnet/performance "net11 OR net12 OR TFM OR framework" \
  --merged --sort updated --limit 20

# Look for BDN changes
gh search prs --repo dotnet/performance "BenchmarkDotNet OR BDN OR benchmark-main" \
  --merged --sort updated --limit 20

Step 4: Check Infrastructure Changes

Before investigating runtime code, always check dotnet/performance repo changes first. Infrastructure changes cause the majority of auto-filed false positives.

Changes That Cause False Regressions

Key Files to Check in dotnet/performance

Definitive Infrastructure Test

If you find an infrastructure change in the regression window:

1. Check if the baseline run used the OLD methodology
2. Check if the compare run used the NEW methodology
3. If yes → the regression is an infrastructure artifact, not a runtime issue
4. Verify by checking that the runtime commit in both runs has no relevant code changes

Step 5: Classify and Report

Classification Decision Tree

Is there a data gap > 2 weeks?
├── YES → Were dotnet/performance changes merged during the gap?
│   ├── YES → Methodology/infrastructure change? 
│   │   ├── YES → INFRASTRUCTURE ARTIFACT — re-baseline needed
│   │   └── NO  → Investigate runtime changes in the gap
│   └── NO  → Investigate why perf runs were missing
└── NO  → Were dotnet/performance changes merged in the window?
    ├── YES → Do they affect measurement methodology?
    │   ├── YES → INFRASTRUCTURE ARTIFACT
    │   └── NO  → Investigate runtime changes
    └── NO  → LIKELY REAL REGRESSION — proceed to runtime investigation

Report Template

## Triage Summary: dotnet/perf-autofiling-issues#XXXX

**Classification:** [Infrastructure Artifact | Real Regression | Needs Investigation]
**Confidence:** [High | Medium | Low]

### Regression Summary
- Total benchmarks affected: N
- Tier 1 (>10x): N benchmarks
- Tier 2 (1.5-10x): N benchmarks  
- Tier 3 (1.1-1.5x): N benchmarks

### Date Range
- Baseline run: YYYY-MM-DD (commit SHA)
- Compare run: YYYY-MM-DD (commit SHA)
- Gap: N days

### Infrastructure Changes in Window
- [ ] BDN version change
- [ ] Entry point change
- [ ] SDK type change
- [ ] Machine pool change
- [ ] Trimming config change

### Evidence
[Binary comparison data, bisection results, or methodology diff]

### Recommendation
[Close as infrastructure artifact | Assign to team | Needs bisection]

Common Patterns

Pattern: "Everything regressed by 100x+"

Almost always a measurement methodology change. The auto-filer compares absolute numbers, not relative to methodology. Check for entry point or SDK type changes.

Pattern: "Only SIMD benchmarks regressed"

Check PackedSimd.IsSupported and SIMD instruction count in dotnet.native.wasm. If SIMD instructions dropped to 0, it's a build config regression. Use the wasm-binary-analysis skill.

Pattern: "Small regressions across many benchmarks"

Could be:

• Machine pool change (different hardware)
• GC configuration change
• Emscripten version bump
• Legitimate runtime change affecting hot paths

Needs bisection to confirm.

Pattern: "Regressions only in one configuration"

Compare the configuration details carefully. If only CompilationMode=wasm regressed but CompilationMode=jit is stable, the issue is WASM-specific. Check Mono interpreter and emscripten changes.