Testing Conformance Harnesses

Conformance Test Harnesses

The One Rule: "Specifications aren't suggestions, they're contracts." A conformance harness mechanically verifies every MUST/SHOULD clause. If it's not tested, it's not conformant.

The Loop (Mandatory)

1. IDENTIFY    → What is the specification? (RFC, API spec, reference impl, formal grammar)
2. EXTRACT     → Enumerate every testable requirement (MUST > SHOULD > MAY)
3. FIXTURE     → Generate reference outputs (run reference impl → golden files)
4. HARNESS     → Build infrastructure: fixture loader, comparator, verdict engine
5. COVER       → Write tests: one per requirement, table-driven, tagged by level
6. DIVERGE     → Document every INTENTIONAL deviation in DISCREPANCIES.md
7. MATRIX      → Generate compliance report: features × status × platform
8. MAINTAIN    → Regenerate fixtures when reference impl updates; diff review

Coverage Accounting Matrix (Mandatory)

Before claiming conformance, prove it:

Rule: Score < 0.95 for MUST clauses = NOT conformant. Ship with known gaps documented, never with unknown gaps.

Decision Tree: Which Conformance Pattern?

What is the specification source?
│
├─ Reference implementation exists (Go, Python, C)
│  └─ DIFFERENTIAL TESTING (Pattern 1)
│     Run both implementations, compare outputs byte-for-byte
│     Examples: charmed_rust (Go→Rust), mcp_agent_mail_rust (Python→Rust)
│
├─ Formal spec exists (RFC, ISO, W3C)
│  └─ SPEC-DERIVED TESTS (Pattern 4)
│     One test per MUST/SHOULD clause, tagged by requirement level
│     Examples: JSON RFC 7159, HTTP/2 RFC 7540, SQL spec
│
├─ Serialization format
│  └─ ROUND-TRIP + GOLDEN FILES (Patterns 2 + 3)
│     Fixtures from reference impl + serialize→deserialize identity
│     Examples: protobuf, MessagePack, CBOR, database page format
│
├─ Network protocol
│  └─ PROCESS-BASED CONFORMANCE (Pattern 6)
│     External test runner drives your server/client
│     Examples: Connect conformance suite, WPT, test262
│
└─ API contract (OpenAPI, GraphQL)
   └─ CONTRACT TESTING (Pattern 5)
      Consumer-driven contracts, provider verification
      Examples: Pact, Dredd, Hurl

The Six Patterns

Pattern 1: Differential Testing (Reference Implementation)

Use when: A canonical implementation exists. This is the gold standard.

Architecture (from charmed_rust, mcp_agent_mail_rust):

tests/conformance/
├── src/
│   ├── harness/
│   │   ├── mod.rs          # Entry point
│   │   ├── traits.rs       # ConformanceTest trait (see below)
│   │   ├── runner.rs       # Collects + executes all tests
│   │   ├── fixtures.rs     # Loads golden files from reference impl
│   │   ├── comparison.rs   # Byte-level, structural, fuzzy comparison
│   │   ├── context.rs      # Test context: paths, config, temp dirs
│   │   └── logging.rs      # Structured JSON-line results
│   └── bin/
│       ├── run_conformance.rs    # `cargo run --bin run_conformance`
│       └── generate_report.rs    # Markdown compliance matrix
├── fixtures/
│   └── go_outputs/          # Generated by: go run ./cmd/gen-fixtures
│       └── lipgloss/
│           ├── border_rounded.golden
│           └── style_padding.golden
├── DISCREPANCIES.md          # Every intentional divergence
└── COVERAGE.md               # What's tested vs what's not

The ConformanceTest Trait:

pub trait ConformanceTest: Send + Sync {
    fn name(&self) -> &str;
    fn category(&self) -> TestCategory;
    fn requirement_level(&self) -> RequirementLevel;  // MUST, SHOULD, MAY
    fn run(&self, ctx: &TestContext) -> TestResult;
}

#[derive(Debug, Serialize)]
pub enum TestCategory { Unit, Integration, EdgeCase, Performance }

#[derive(Debug, Serialize)]
pub enum RequirementLevel { Must, Should, May }

#[derive(Debug, Serialize)]
#[serde(tag = "status")]
pub enum TestResult {
    Pass,
    Fail { reason: String },
    Skipped { reason: String },
    ExpectedFailure { reason: String },  // Known divergence (XFAIL)
}

Fixture-driven differential test:

#[test]
fn conformance_lipgloss_border_rounded() {
    let fixture = load_fixture("go_outputs/lipgloss/border_rounded.golden");
    let actual = Style::new()
        .border(Border::Rounded)
        .padding(1, 2)
        .render("Hello, World!");

    assert_eq!(actual, fixture.expected,
        "Rust rendering diverges from Go reference\n\
         Go output:   {:?}\n\
         Rust output: {:?}\n\
         Fixture:     {}",
        fixture.expected, actual, fixture.path.display());
}

Pattern 2: Golden File Testing

Use when: Output is complex, correct once verified, then frozen.

fn assert_golden(test_name: &str, actual: &str) {
    let golden_path = Path::new("tests/golden")
        .join(format!("{test_name}.golden"));

    if std::env::var("UPDATE_GOLDENS").is_ok() {
        fs::create_dir_all(golden_path.parent().unwrap()).unwrap();
        fs::write(&golden_path, actual).unwrap();
        eprintln!("UPDATED golden: {}", golden_path.display());
        return;
    }

    let expected = fs::read_to_string(&golden_path)
        .unwrap_or_else(|_| panic!(
            "Golden file not found: {}\n\
             Run with UPDATE_GOLDENS=1 to create it",
            golden_path.display()
        ));

    if actual != expected {
        let actual_path = golden_path.with_extension("actual");
        fs::write(&actual_path, actual).unwrap();
        panic!(
            "GOLDEN MISMATCH: {}\n\
             diff {} {}",
            test_name,
            golden_path.display(),
            actual_path.display(),
        );
    }
}

Workflow:

# First run: create golden files
UPDATE_GOLDENS=1 cargo test

# Subsequent runs: compare
cargo test

# After intentional changes:
diff tests/golden/report.golden tests/golden/report.actual
UPDATE_GOLDENS=1 cargo test
git diff tests/golden/  # Review every change before committing

Pattern 3: Round-Trip Conformance

Use when: Data must survive a serialize→deserialize cycle perfectly, AND must interoperate with a reference implementation.

/// Cross-implementation round-trip:
/// 1. Reference impl produces fixture
/// 2. Our impl parses it (must succeed)
/// 3. Our impl re-serializes it
/// 4. Reference impl parses our output (must succeed and match)
fn conformance_cross_impl_roundtrip(fixtures_dir: &Path) {
    for fixture_path in glob(fixtures_dir, "*.bin") {
        let reference_bytes = fs::read(&fixture_path).unwrap();

        // Step 1: We must be able to parse reference output
        let parsed = our_parser::parse(&reference_bytes)
            .unwrap_or_else(|e| panic!(
                "Cannot parse reference fixture {}: {e}",
                fixture_path.display()));

        // Step 2: We re-serialize
        let our_bytes = our_serializer::serialize(&parsed);

        // Step 3: Reference must be able to parse our output
        let reparsed = reference_parser::parse(&our_bytes)
            .unwrap_or_else(|e| panic!(
                "Reference cannot parse our output for {}: {e}",
                fixture_path.display()));

        // Step 4: Parsed values must match
        assert_eq!(parsed, reparsed,
            "Cross-impl round-trip diverged for {}",
            fixture_path.display());
    }
}

Pattern 4: Spec-Derived Test Matrix

Use when: Implementing an RFC or formal specification.

struct ConformanceCase {
    id: &'static str,           // "RFC7159-2.1"
    section: &'static str,      // "2"
    level: RequirementLevel,    // Must, Should, May
    description: &'static str,
    input: &'static str,
    expected: Result<Value, ()>,
}

const RFC7159_CASES: &[ConformanceCase] = &[
    // Section 2: JSON Grammar
    ConformanceCase {
        id: "RFC7159-2.1",
        section: "2",
        level: RequirementLevel::Must,
        description: "A JSON text is a serialized value",
        input: "42",
        expected: Ok(Value::Number(42)),
    },
    ConformanceCase {
        id: "RFC7159-7.1",
        section: "7",
        level: RequirementLevel::Must,
        description: "Unicode escape sequences \\uXXXX",
        input: r#""\u0041""#,
        expected: Ok(Value::String("A".into())),
    },
    // ... one per MUST/SHOULD clause
];

#[test]
fn rfc7159_full_conformance() {
    let mut pass = 0;
    let mut fail = 0;
    let mut xfail = 0; // Expected failures (known divergences)

    for case in RFC7159_CASES {
        let result = our_parser::parse(case.input);
        let verdict = match (&result, &case.expected) {
            (Ok(a), Ok(b)) if a == b => { pass += 1; "PASS" }
            (Err(_), Err(())) => { pass += 1; "PASS" }
            _ => {
                if is_known_divergence(case.id) {
                    xfail += 1;
                    "XFAIL"
                } else {
                    fail += 1;
                    eprintln!("FAIL {}: {}\n  expected: {:?}\n  actual: {:?}",
                        case.id, case.description, case.expected, result);
                    "FAIL"
                }
            }
        };
        // Structured JSON-line output for CI parsing
        eprintln!("{{\"id\":\"{}\",\"verdict\":\"{verdict}\",\"level\":\"{:?}\"}}",
            case.id, case.level);
    }

    let total = pass + fail + xfail;
    eprintln!("\nRFC 7159: {pass}/{total} pass, {fail} fail, {xfail} expected-fail");
    assert_eq!(fail, 0, "{fail} conformance tests failed");
}

Pattern 5: Contract Testing (API Conformance)

Use when: Testing API compatibility between services.

// Consumer-driven contract: client defines expectations
// Provider must satisfy ALL consumer contracts

// Consumer side (tests what we NEED from the API)
describe("User API Contract", () => {
  it("GET /users/:id returns user with email", async () => {
    const user = await api.getUser("user-123");
    expect(user).toHaveProperty("id");
    expect(user).toHaveProperty("email");
    expect(typeof user.email).toBe("string");
    // Contract: we don't care about other fields — only what we consume
  });
});

// Provider side (verifies all consumer contracts are satisfied)
// Run against actual API implementation

# Hurl: HTTP-level conformance testing
# tests/conformance/api/users.hurl
GET http://localhost:3000/api/v1/users/me
Authorization: Bearer {{token}}
HTTP 200
[Asserts]
jsonpath "$.id" exists
jsonpath "$.email" isString
jsonpath "$.subscriptionStatus" matches /^(none|active|past_due|cancelled)$/

Pattern 6: Process-Based Conformance (External Runner)

Use when: A standard conformance runner exists for your protocol.

# Connect conformance suite: tests gRPC/Connect protocol compliance
connectconformance --mode server \
  --config conformance-config.yaml \
  -- ./our-server

# Web Platform Tests: browser conformance
wpt run --channel dev --product our-browser

# test262: JavaScript engine conformance
test262-harness --hostType our-engine --hostPath ./our-engine

DISCREPANCIES.md (Mandatory)

Every conformance harness accumulates intentional divergences. Document them ALL.

# Known Conformance Divergences

## DISC-001: Unicode width tables
- **Reference:** Uses Unicode 13.0 width tables (go-runewidth v0.14)
- **Our impl:** Uses Unicode 15.1 width tables (unicode-width v0.2)
- **Impact:** Some CJK chars have different widths → alignment differs
- **Resolution:** ACCEPTED — newer Unicode tables are more correct
- **Tests affected:** lipgloss/cjk_alignment_*
- **Review date:** 2026-03-15

## DISC-002: Error message format
- **Reference:** Returns "invalid input at byte 42"
- **Our impl:** Returns "parse error: unexpected byte 0x2A at offset 42"
- **Impact:** Error strings differ (semantics identical)
- **Resolution:** ACCEPTED — we test error categories, not messages
- **Tests affected:** parser/error_*

Rules for DISCREPANCIES.md:

1. Every divergence gets a sequential ID (DISC-NNN)
2. Must state whether ACCEPTED, INVESTIGATING, or WILL-FIX
3. Must list affected test cases
4. Must include review date (divergences can become stale)
5. Tests for accepted divergences use XFAIL, not SKIP

Fixture Provenance (Non-Negotiable)

Every fixture must record how it was generated:

tests/conformance/fixtures/
├── PROVENANCE.md              # How fixtures were generated
├── go_outputs/
│   ├── generated_with: go1.22.1
│   ├── command: go run ./cmd/gen-fixtures > fixtures/go_outputs/
│   └── git_ref: abc123 (tag: v0.15.2)
└── python_reference.json
    ├── generated_with: python3.12 + mcp-agent-mail 0.9.1
    └── command: python -m mcp_agent_mail.conformance.generate > python_reference.json

Why: When fixtures are regenerated 6 months later and results change, you need to know what version generated the originals to diagnose whether the change is a bug or a new feature.

Compliance Report Generator

fn generate_compliance_report(results: &[TestResult]) -> String {
    let mut by_section: BTreeMap<&str, SectionStats> = BTreeMap::new();

    for result in results {
        let section = by_section.entry(result.section).or_default();
        match result.level {
            Must => section.must_total += 1,
            Should => section.should_total += 1,
            May => section.may_total += 1,
        }
        if result.verdict == Pass { section.passing += 1; }
        if result.verdict == XFail { section.xfail += 1; }
    }

    // Output: Markdown table
    // | Section | MUST (pass/total) | SHOULD (pass/total) | Score |
    // |---------|-------------------|---------------------|-------|
    // | §2      | 15/15             | 8/10                | 95.8% |
}

Anti-Patterns (Hard Constraints)

Checklist (Before Claiming Conformance)

• Specification source identified and version pinned
• Coverage matrix built: every MUST/SHOULD clause enumerated
• MUST clause coverage ≥ 95% (100% target)
• Fixtures generated from reference impl with recorded provenance
• DISCREPANCIES.md documents every intentional divergence
• XFAIL (not SKIP) for accepted divergences
• Compliance report generated automatically
• Round-trip tests for all serializable types
• Error cases tested (not just happy paths)
• CI regenerates report on every PR
• Fixture update workflow documented and tested

References

Relationship to Other Testing Skills