Dotnet Test Quality

dotnet-test-quality

Test quality analysis for .NET projects. Covers code coverage collection with coverlet, human-readable coverage reports with ReportGenerator, CRAP (Change Risk Anti-Patterns) score analysis to identify undertested complex code, mutation testing with Stryker.NET to evaluate test suite effectiveness, and strategies for detecting and managing flaky tests.

Version assumptions: Coverlet 6.x+, ReportGenerator 5.x+, Stryker.NET 4.x+ (.NET 8.0+ baseline). Coverlet supports both the MSBuild integration (coverlet.msbuild) and the coverlet.collector data collector; examples use coverlet.collector as the recommended approach.

Scope

• Coverlet code coverage collection and configuration
• ReportGenerator for human-readable coverage reports
• CRAP score analysis for undertested complex code
• Stryker.NET mutation testing for test suite evaluation
• Flaky test detection and management strategies

Out of scope

• Test project scaffolding (creating projects, package references, coverlet setup) -- see [skill:dotnet-add-testing]
• Testing strategy and test type decisions -- see [skill:dotnet-testing-strategy]
• CI test reporting and pipeline integration -- see [skill:dotnet-gha-build-test] and [skill:dotnet-ado-build-test]

Prerequisites: Test project already scaffolded via [skill:dotnet-add-testing] with coverlet packages referenced. .NET 8.0+ baseline required.

Cross-references: [skill:dotnet-testing-strategy] for deciding what to test and coverage target guidance, [skill:dotnet-xunit] for xUnit test framework features and configuration.

Code Coverage with Coverlet

Coverlet is the standard open-source code coverage library for .NET. It instruments assemblies at build time or via a data collector and produces coverage reports in multiple formats.

Packages

<!-- Data collector approach (recommended) -->
<PackageReference Include="coverlet.collector" Version="8.0.0">
  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
  <PrivateAssets>all</PrivateAssets>
</PackageReference>

```text

### Collecting Coverage

```bash

# Collect coverage with Cobertura output (default for ReportGenerator)
dotnet test --collect:"XPlat Code Coverage"

# Specify output format explicitly
dotnet test --collect:"XPlat Code Coverage" \
  -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura

# Multiple formats
dotnet test --collect:"XPlat Code Coverage" \
  -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura,opencover

```text

Coverage results are written to `TestResults/<guid>/coverage.cobertura.xml` under each test project's output directory.

### Filtering Coverage

Exclude generated code, test projects, or specific namespaces:

```bash

dotnet test --collect:"XPlat Code Coverage" \
  -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Exclude="[*.Tests]*,[*.IntegrationTests]*" \
  DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.ExcludeByAttribute="GeneratedCodeAttribute,ObsoleteAttribute,ExcludeFromCodeCoverageAttribute"

```bash

Or configure via a `runsettings` file for repeatability:

```xml

<!-- coverlet.runsettings -->
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat Code Coverage">
        <Configuration>
          <Format>cobertura</Format>
          <Exclude>[*.Tests]*,[*.IntegrationTests]*</Exclude>
          <ExcludeByAttribute>
            GeneratedCodeAttribute,ObsoleteAttribute,ExcludeFromCodeCoverageAttribute
          </ExcludeByAttribute>
          <ExcludeByFile>**/Migrations/**</ExcludeByFile>
          <IncludeTestAssembly>false</IncludeTestAssembly>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

```text

```bash

dotnet test --settings coverlet.runsettings

```bash

### Merge Coverage from Multiple Test Projects

When a solution has multiple test projects, merge their coverage into a single report:

```bash

# Run all tests, collecting coverage per project
dotnet test --collect:"XPlat Code Coverage"

# Find all coverage files and merge via ReportGenerator (see next section)

```text

---

## Coverage Reports with ReportGenerator

ReportGenerator converts raw coverage data (Cobertura, OpenCover) into human-readable HTML reports with line-level highlighting.

### Installation

```bash

# Install as a global tool
dotnet tool install -g dotnet-reportgenerator-globaltool

# Or as a local tool
dotnet tool install dotnet-reportgenerator-globaltool

```text

### Generating Reports

```bash

# Single coverage file
reportgenerator \
  -reports:"tests/MyApp.Tests/TestResults/*/coverage.cobertura.xml" \
  -targetdir:"coverage-report" \
  -reporttypes:"Html;TextSummary"

# Multiple test projects (glob pattern merges automatically)
reportgenerator \
  -reports:"**/TestResults/*/coverage.cobertura.xml" \
  -targetdir:"coverage-report" \
  -reporttypes:"Html;Cobertura;TextSummary"

```xml

### Report Types

| Type | Description | Use Case |
|------|-------------|----------|
| `Html` | Interactive HTML with line highlighting | Local developer review |
| `HtmlInline_AzurePipelines` | HTML optimized for Azure DevOps | CI artifact |
| `Cobertura` | Merged Cobertura XML | Input for other tools |
| `TextSummary` | Plain text summary | CLI/CI output |
| `Badges` | SVG coverage badges | README badges |
| `MarkdownSummaryGithub` | GitHub-flavored markdown | PR comments |

### Example: Full Coverage Pipeline

```bash

#!/bin/bash
# clean previous results
rm -rf coverage-report TestResults

# run tests with coverage
dotnet test --collect:"XPlat Code Coverage" --results-directory TestResults

# generate merged HTML report
reportgenerator \
  -reports:"**/TestResults/*/coverage.cobertura.xml" \
  -targetdir:"coverage-report" \
  -reporttypes:"Html;TextSummary;Badges"

# display summary
cat coverage-report/Summary.txt

```text

### Setting Coverage Thresholds

Enforce minimum coverage in CI by parsing the text summary or using a threshold parameter:

```bash

# ReportGenerator does not enforce thresholds directly.
# Parse the summary or use dotnet-coverage (Microsoft) for threshold enforcement.

# Alternative: use coverlet's built-in threshold via MSBuild
dotnet test /p:CollectCoverage=true \
  /p:Threshold=80 \
  /p:ThresholdType=line \
  /p:ThresholdStat=total

```text

**Note:** The `/p:Threshold` parameter requires the `coverlet.msbuild` package (not `coverlet.collector`). For `coverlet.collector` workflows, enforce thresholds by parsing the ReportGenerator text summary in your CI script.

---

## CRAP Analysis

CRAP (Change Risk Anti-Patterns) scores identify methods that are both complex and poorly tested. A high CRAP score means the method has high cyclomatic complexity and low code coverage -- a risky combination.

### Formula

```text

CRAP(m) = complexity(m)^2 * (1 - coverage(m)/100)^3 + complexity(m)

```text

Where:
- `complexity(m)` = cyclomatic complexity of method m
- `coverage(m)` = code coverage percentage of method m (0-100)

### Interpreting CRAP Scores

| CRAP Score | Risk Level | Action |
|------------|------------|--------|
| < 5 | Low | Method is simple or well-tested |
| 5-15 | Moderate | Review -- may need additional tests |
| 15-30 | High | Prioritize: add tests or reduce complexity |
| > 30 | Critical | Refactor and add tests immediately |

### Generating CRAP Reports

ReportGenerator includes CRAP analysis when using OpenCover format as input:

```bash

# Step 1: Collect coverage in OpenCover format
dotnet test --collect:"XPlat Code Coverage" \
  -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover

# Step 2: Generate report with risk hotspot analysis
reportgenerator \
  -reports:"**/TestResults/*/coverage.opencover.xml" \
  -targetdir:"coverage-report" \
  -reporttypes:"Html;RiskHotspots"

```xml

The Risk Hotspots report highlights methods sorted by CRAP score, showing:
- Method name and containing class
- Cyclomatic complexity
- Code coverage percentage
- Computed CRAP score

### Using CRAP Scores Effectively

```csharp

// Example: a method with high complexity and low coverage
// Cyclomatic complexity: 12, Coverage: 20%
// CRAP = 12^2 * (1 - 0.20)^3 + 12 = 144 * 0.512 + 12 = 85.7 (Critical)
public decimal CalculateShipping(Order order)
{
    if (order.Items.Count == 0) return 0;

    decimal baseRate = order.DestinationCountry switch
    {
        "US" => 5.99m,
        "CA" => 9.99m,
        "UK" => 12.99m,
        _ => 19.99m
    };

    if (order.Total > 100) baseRate *= 0.5m;
    if (order.IsPriority) baseRate *= 2.0m;
    if (order.Items.Any(i => i.IsFragile)) baseRate += 4.99m;
    if (order.Items.Any(i => i.IsOversized)) baseRate += 14.99m;
    if (order.HasInsurance) baseRate += order.Total * 0.02m;
    if (order.IsExpedited && order.DestinationCountry != "US") baseRate *= 1.5m;

    return Math.Round(baseRate, 2);
}

```text

Address high CRAP scores by:
1. **Adding targeted tests** for uncovered branches to reduce the score via higher coverage
2. **Reducing complexity** by extracting methods (e.g., separate `CalculateBaseRate` and `ApplySurcharges` methods)
3. **Both** -- the most effective approach combines better coverage with simpler methods

---

## Mutation Testing with Stryker.NET

Mutation testing evaluates test suite quality by introducing small changes (mutations) to production code and checking whether tests detect them. If a mutation survives (tests still pass), the test suite has a gap.

### Installation

```bash

# Install as a global tool
dotnet tool install -g dotnet-stryker

# Or as a local tool (recommended for team consistency)
dotnet tool install dotnet-stryker

```text

### Running Stryker.NET

```bash

# From the test project directory
cd tests/MyApp.Tests
dotnet stryker

# Specify the source project explicitly
dotnet stryker --project MyApp.csproj

# Target specific files
dotnet stryker --mutate "src/Services/**/*.cs"

```csharp

### Configuration File

Create `stryker-config.json` in the test project directory:

```json

{
  "$schema": "https://raw.githubusercontent.com/stryker-mutator/stryker-net/master/src/Stryker.Core/Stryker.Core/stryker-config.schema.json",
  "stryker-config": {
    "project": "MyApp.csproj",
    "reporters": ["html", "progress", "cleartext"],
    "mutation-level": "Standard",
    "thresholds": {
      "high": 80,
      "low": 60,
      "break": 50
    },
    "mutate": [
      "src/Services/**/*.cs",
      "!src/Services/Migrations/**/*.cs"
    ],
    "ignore-mutations": [
      "string",
      "linq"
    ]
  }
}

```text

### Understanding Mutation Results

Stryker reports mutations in four categories:

| Status | Meaning | Action |
|--------|---------|--------|
| **Killed** | A test detected the mutation (failed) | Good -- test suite caught the defect |
| **Survived** | No test detected the mutation (all passed) | Gap -- add or strengthen tests |
| **No Coverage** | No test covers the mutated code | Gap -- add tests for this code |
| **Timeout** | Mutation caused an infinite loop or timeout | Usually killed (counts as detected) |

### Mutation Score

```text

Mutation Score = Killed / (Killed + Survived + NoCoverage) * 100

```text

A mutation score of 80%+ indicates a strong test suite. Below 60% suggests significant gaps.

### Example: Identifying Test Gaps

Given this production code:

```csharp

public class PricingService
{
    public decimal CalculateDiscount(decimal price, CustomerTier tier) =>
        tier switch
        {
            CustomerTier.Bronze => price * 0.05m,
            CustomerTier.Silver => price * 0.10m,
            CustomerTier.Gold => price * 0.15m,
            CustomerTier.Platinum => price * 0.20m,
            _ => 0m
        };
}

```text

If tests only verify `Gold` tier, Stryker generates mutations like:
- Replace `0.05m` with `0.06m` (survived -- no Bronze test)
- Replace `0.10m` with `0.11m` (survived -- no Silver test)
- Replace `0.15m` with `0.16m` (killed -- Gold test catches this)
- Replace `0.20m` with `0.21m` (survived -- no Platinum test)
- Replace `0m` with `1m` (survived -- no default test)

The HTML report highlights each surviving mutation with the exact code change, guiding where to add tests.

### Stryker Thresholds

```json

{
  "thresholds": {
    "high": 80,   // Green: mutation score >= 80%
    "low": 60,    // Yellow: 60% <= mutation score < 80%
    "break": 50   // Red: mutation score < 50% -> exit code 1
  }
}

```text

The `break` threshold causes Stryker to return a non-zero exit code, useful for CI gates.

---

## Flaky Test Detection

Flaky tests pass and fail intermittently without code changes. They erode trust in the test suite and slow development.

### Common Causes

| Cause | Symptom | Fix |
|-------|---------|-----|
| **Shared mutable state** | Tests fail when run in specific order | Use proper test isolation (see [skill:dotnet-xunit] for fixtures) |
| **Time-dependent logic** | Tests fail near midnight or at specific times | Inject `TimeProvider` (or `ISystemClock`) instead of using `DateTime.Now` |
| **Race conditions** | Tests fail intermittently under parallel execution | Use `ICollectionFixture` for shared resources; avoid shared static state |
| **External dependencies** | Tests fail when network/services unavailable | Mock external calls; use Testcontainers for infrastructure |
| **Port conflicts** | Tests fail when another process uses the same port | Use dynamic port allocation (WebApplicationFactory handles this) |
| **File system contention** | Tests fail under parallel execution | Use unique temp directories per test (see [skill:dotnet-xunit] `IAsyncLifetime` patterns) |

### Detecting Flaky Tests

#### Repeated Runs

```bash

# Run tests multiple times to surface flakiness
for i in $(seq 1 10); do
  dotnet test --logger "trx;LogFileName=run-$i.trx" || echo "Run $i failed"
done

```text

#### xUnit Conditional Skip

**xUnit v3** has built-in conditional skip via `Skip` on `[Fact]`:

```csharp

// xUnit v3 — built-in conditional skip
[Fact(Skip = "Requires external service")]
public async Task ExternalApi_ReturnsData()
{
    var result = await _client.GetDataAsync();
    Assert.NotEmpty(result);
}

// xUnit v3 — runtime skip via Assert.Skip
[Fact]
public async Task ExternalApi_ReturnsData()
{
    if (!await IsServiceAvailable())
        Assert.Skip("External service unavailable");

    var result = await _client.GetDataAsync();
    Assert.NotEmpty(result);
}

```text

### Time-Dependent Tests

Replace `DateTime.Now`/`DateTime.UtcNow` with .NET 8's `TimeProvider`:

```csharp

// Production code
public class SubscriptionService(TimeProvider timeProvider)
{
    public bool IsExpired(Subscription sub)
    {
        var now = timeProvider.GetUtcNow();
        return sub.ExpiresAt < now;
    }
}

// Test code
[Fact]
public void IsExpired_PastExpiry_ReturnsTrue()
{
    var fakeTime = new FakeTimeProvider(
        new DateTimeOffset(2025, 6, 15, 0, 0, 0, TimeSpan.Zero));

    var service = new SubscriptionService(fakeTime);
    var sub = new Subscription
    {
        ExpiresAt = new DateTimeOffset(2025, 6, 14, 0, 0, 0, TimeSpan.Zero)
    };

    Assert.True(service.IsExpired(sub));
}

[Fact]
public void IsExpired_FutureExpiry_ReturnsFalse()
{
    var fakeTime = new FakeTimeProvider(
        new DateTimeOffset(2025, 6, 15, 0, 0, 0, TimeSpan.Zero));

    var service = new SubscriptionService(fakeTime);
    var sub = new Subscription
    {
        ExpiresAt = new DateTimeOffset(2025, 6, 16, 0, 0, 0, TimeSpan.Zero)
    };

    Assert.False(service.IsExpired(sub));
}

```text

**Note:** `FakeTimeProvider` is available in `Microsoft.Extensions.TimeProvider.Testing` (NuGet).

### Quarantine Strategy

When a flaky test cannot be fixed immediately:

```csharp

// Mark as skipped with a tracking issue
[Fact(Skip = "Flaky: tracking in #1234 -- race condition in event handler")]
public async Task EventHandler_ConcurrentEvents_ProcessesAll()
{
    // ...
}

```text

Do not delete flaky tests. Skip them with an issue reference and fix them systematically.

---

## Key Principles

- **Coverage is a lagging indicator, not a target.** High coverage does not guarantee good tests. A test suite with 90% coverage can still miss critical bugs if the assertions are weak.
- **Use CRAP scores to prioritize.** Focus testing effort on methods with high complexity and low coverage rather than chasing overall coverage percentage.
- **Run mutation testing on critical paths.** Mutation testing is computationally expensive. Focus on business-critical code (pricing, authentication, data validation) rather than running it on the entire codebase.
- **Fix flaky tests immediately or quarantine them.** A flaky test that remains in the suite trains developers to ignore failures, undermining the entire test suite's value.
- **Measure trends, not snapshots.** Track coverage and mutation scores over time. A declining trend indicates test quality erosion even if absolute numbers look acceptable.
- **Exclude generated code from coverage.** Migrations, generated clients, and scaffolded code inflate or deflate coverage numbers without reflecting actual test quality.

---

## Agent Gotchas

1. **Do not confuse `coverlet.collector` with `coverlet.msbuild`.** The `coverlet.collector` package uses the `--collect:"XPlat Code Coverage"` CLI flag. The `coverlet.msbuild` package uses `/p:CollectCoverage=true` MSBuild properties. Do not mix flags across packages -- they are independent integration points.
2. **Do not hardcode coverage result paths.** The GUID in `TestResults/<guid>/coverage.cobertura.xml` changes every run. Always use glob patterns (`**/TestResults/*/coverage.cobertura.xml`) when referencing coverage output files.
3. **Do not set coverage thresholds too high initially.** Starting with 90%+ thresholds on an existing project blocks all PRs. Begin with the current baseline and increase incrementally (e.g., 5% per quarter).
4. **Do not run Stryker.NET on the entire solution for CI.** Mutation testing is CPU-intensive. In CI, limit mutations to changed files (`--since:main`) or critical paths. Reserve full runs for nightly builds.
5. **Do not ignore survived mutations in trivial code.** While some survived mutations are in code that does not warrant testing (logging, `ToString()`), review each one. Configure `ignore-mutations` in `stryker-config.json` for categories you have consciously decided not to test.
6. **Do not use `[ExcludeFromCodeCoverage]` as a blanket fix for low coverage.** This attribute hides the problem rather than solving it. Use it only for genuinely untestable code (platform interop, generated code) and ensure the reason is documented.

---

## References

- [Coverlet GitHub repository](https://github.com/coverlet-coverage/coverlet)
- [ReportGenerator GitHub repository](https://github.com/danielpalme/ReportGenerator)
- [ReportGenerator usage guide](https://github.com/danielpalme/ReportGenerator/wiki)
- [Stryker.NET documentation](https://stryker-mutator.io/docs/stryker-net/introduction/)
- [Stryker.NET configuration](https://stryker-mutator.io/docs/stryker-net/configuration/)
- [TimeProvider in .NET 8](https://learn.microsoft.com/en-us/dotnet/api/system.timeprovider)
- [Microsoft.Extensions.TimeProvider.Testing](https://www.nuget.org/packages/Microsoft.Extensions.TimeProvider.Testing)
- [CRAP metric explanation](https://testing.googleblog.com/2011/02/this-code-is-crap.html)

Code Navigation (Serena MCP)

Primary approach: Use Serena symbol operations for efficient code navigation:

1. Find definitions: serena_find_symbol instead of text search
2. Understand structure: serena_get_symbols_overview for file organization
3. Track references: serena_find_referencing_symbols for impact analysis
4. Precise edits: serena_replace_symbol_body for clean modifications

When to use Serena vs traditional tools:

• Use Serena: Navigation, refactoring, dependency analysis, precise edits
• Use Read/Grep: Reading full files, pattern matching, simple text operations
• Fallback: If Serena unavailable, traditional tools work fine

Example workflow:

# Instead of:
Read: src/Services/OrderService.cs
Grep: "public void ProcessOrder"

# Use:
serena_find_symbol: "OrderService/ProcessOrder"
serena_get_symbols_overview: "src/Services/OrderService.cs"