# Testing Conformance Harnesses > Build conformance test harnesses that verify implementations against specifications. Use when: porting libraries across languages, implementing RFCs/specs, building database engines, validating protocol compliance, cross-platform compatibility, API contract testing, golden file testing, round-trip validation, compliance matrices, differential testing against reference implementations. - Skill: `lev-os/testing-conformance-harnesses` (Agent Skill, multi-file: 6 files) - Install (CLI): `npx skillmds@latest add lev-os/testing-conformance-harnesses` - Raw SKILL.md: https://api.skillmd.com/api/skills/lev-os/testing-conformance-harnesses/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Integrations & APIs - Author: lev-os (https://skillmd.com/u/lev-os) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/lev-os/testing-conformance-harnesses --- # Conformance Test Harnesses > **The One Rule:** "Specifications aren't suggestions, they're contracts." > A conformance harness mechanically verifies every MUST/SHOULD clause. > If it's not tested, it's not conformant. ## The Loop (Mandatory) ``` 1. IDENTIFY → What is the specification? (RFC, API spec, reference impl, formal grammar) 2. EXTRACT → Enumerate every testable requirement (MUST > SHOULD > MAY) 3. FIXTURE → Generate reference outputs (run reference impl → golden files) 4. HARNESS → Build infrastructure: fixture loader, comparator, verdict engine 5. COVER → Write tests: one per requirement, table-driven, tagged by level 6. DIVERGE → Document every INTENTIONAL deviation in DISCREPANCIES.md 7. MATRIX → Generate compliance report: features × status × platform 8. MAINTAIN → Regenerate fixtures when reference impl updates; diff review ``` ## Coverage Accounting Matrix (Mandatory) Before claiming conformance, prove it: | Spec Section | MUST Clauses | SHOULD Clauses | Tested | Passing | Divergent | Score | |-------------|:-----------:|:--------------:|:------:|:-------:|:---------:|-------| | *Section N* | count | count | count | count | count | Pass/(MUST+SHOULD) | **Rule:** Score < 0.95 for MUST clauses = NOT conformant. Ship with known gaps documented, never with unknown gaps. --- ## Decision Tree: Which Conformance Pattern? ``` What is the specification source? │ ├─ Reference implementation exists (Go, Python, C) │ └─ DIFFERENTIAL TESTING (Pattern 1) │ Run both implementations, compare outputs byte-for-byte │ Examples: charmed_rust (Go→Rust), mcp_agent_mail_rust (Python→Rust) │ ├─ Formal spec exists (RFC, ISO, W3C) │ └─ SPEC-DERIVED TESTS (Pattern 4) │ One test per MUST/SHOULD clause, tagged by requirement level │ Examples: JSON RFC 7159, HTTP/2 RFC 7540, SQL spec │ ├─ Serialization format │ └─ ROUND-TRIP + GOLDEN FILES (Patterns 2 + 3) │ Fixtures from reference impl + serialize→deserialize identity │ Examples: protobuf, MessagePack, CBOR, database page format │ ├─ Network protocol │ └─ PROCESS-BASED CONFORMANCE (Pattern 6) │ External test runner drives your server/client │ Examples: Connect conformance suite, WPT, test262 │ └─ API contract (OpenAPI, GraphQL) └─ CONTRACT TESTING (Pattern 5) Consumer-driven contracts, provider verification Examples: Pact, Dredd, Hurl ``` --- ## The Six Patterns ### Pattern 1: Differential Testing (Reference Implementation) **Use when:** A canonical implementation exists. This is the gold standard. **Architecture (from charmed_rust, mcp_agent_mail_rust):** ``` tests/conformance/ ├── src/ │ ├── harness/ │ │ ├── mod.rs # Entry point │ │ ├── traits.rs # ConformanceTest trait (see below) │ │ ├── runner.rs # Collects + executes all tests │ │ ├── fixtures.rs # Loads golden files from reference impl │ │ ├── comparison.rs # Byte-level, structural, fuzzy comparison │ │ ├── context.rs # Test context: paths, config, temp dirs │ │ └── logging.rs # Structured JSON-line results │ └── bin/ │ ├── run_conformance.rs # `cargo run --bin run_conformance` │ └── generate_report.rs # Markdown compliance matrix ├── fixtures/ │ └── go_outputs/ # Generated by: go run ./cmd/gen-fixtures │ └── lipgloss/ │ ├── border_rounded.golden │ └── style_padding.golden ├── DISCREPANCIES.md # Every intentional divergence └── COVERAGE.md # What's tested vs what's not ``` **The ConformanceTest Trait:** ```rust pub trait ConformanceTest: Send + Sync { fn name(&self) -> &str; fn category(&self) -> TestCategory; fn requirement_level(&self) -> RequirementLevel; // MUST, SHOULD, MAY fn run(&self, ctx: &TestContext) -> TestResult; } #[derive(Debug, Serialize)] pub enum TestCategory { Unit, Integration, EdgeCase, Performance } #[derive(Debug, Serialize)] pub enum RequirementLevel { Must, Should, May } #[derive(Debug, Serialize)] #[serde(tag = "status")] pub enum TestResult { Pass, Fail { reason: String }, Skipped { reason: String }, ExpectedFailure { reason: String }, // Known divergence (XFAIL) } ``` **Fixture-driven differential test:** ```rust #[test] fn conformance_lipgloss_border_rounded() { let fixture = load_fixture("go_outputs/lipgloss/border_rounded.golden"); let actual = Style::new() .border(Border::Rounded) .padding(1, 2) .render("Hello, World!"); assert_eq!(actual, fixture.expected, "Rust rendering diverges from Go reference\n\ Go output: {:?}\n\ Rust output: {:?}\n\ Fixture: {}", fixture.expected, actual, fixture.path.display()); } ``` ### Pattern 2: Golden File Testing **Use when:** Output is complex, correct once verified, then frozen. ```rust fn assert_golden(test_name: &str, actual: &str) { let golden_path = Path::new("tests/golden") .join(format!("{test_name}.golden")); if std::env::var("UPDATE_GOLDENS").is_ok() { fs::create_dir_all(golden_path.parent().unwrap()).unwrap(); fs::write(&golden_path, actual).unwrap(); eprintln!("UPDATED golden: {}", golden_path.display()); return; } let expected = fs::read_to_string(&golden_path) .unwrap_or_else(|_| panic!( "Golden file not found: {}\n\ Run with UPDATE_GOLDENS=1 to create it", golden_path.display() )); if actual != expected { let actual_path = golden_path.with_extension("actual"); fs::write(&actual_path, actual).unwrap(); panic!( "GOLDEN MISMATCH: {}\n\ diff {} {}", test_name, golden_path.display(), actual_path.display(), ); } } ``` **Workflow:** ```bash # First run: create golden files UPDATE_GOLDENS=1 cargo test # Subsequent runs: compare cargo test # After intentional changes: diff tests/golden/report.golden tests/golden/report.actual UPDATE_GOLDENS=1 cargo test git diff tests/golden/ # Review every change before committing ``` ### Pattern 3: Round-Trip Conformance **Use when:** Data must survive a serialize→deserialize cycle perfectly, AND must interoperate with a reference implementation. ```rust /// Cross-implementation round-trip: /// 1. Reference impl produces fixture /// 2. Our impl parses it (must succeed) /// 3. Our impl re-serializes it /// 4. Reference impl parses our output (must succeed and match) fn conformance_cross_impl_roundtrip(fixtures_dir: &Path) { for fixture_path in glob(fixtures_dir, "*.bin") { let reference_bytes = fs::read(&fixture_path).unwrap(); // Step 1: We must be able to parse reference output let parsed = our_parser::parse(&reference_bytes) .unwrap_or_else(|e| panic!( "Cannot parse reference fixture {}: {e}", fixture_path.display())); // Step 2: We re-serialize let our_bytes = our_serializer::serialize(&parsed); // Step 3: Reference must be able to parse our output let reparsed = reference_parser::parse(&our_bytes) .unwrap_or_else(|e| panic!( "Reference cannot parse our output for {}: {e}", fixture_path.display())); // Step 4: Parsed values must match assert_eq!(parsed, reparsed, "Cross-impl round-trip diverged for {}", fixture_path.display()); } } ``` ### Pattern 4: Spec-Derived Test Matrix **Use when:** Implementing an RFC or formal specification. ```rust struct ConformanceCase { id: &'static str, // "RFC7159-2.1" section: &'static str, // "2" level: RequirementLevel, // Must, Should, May description: &'static str, input: &'static str, expected: Result, } const RFC7159_CASES: &[ConformanceCase] = &[ // Section 2: JSON Grammar ConformanceCase { id: "RFC7159-2.1", section: "2", level: RequirementLevel::Must, description: "A JSON text is a serialized value", input: "42", expected: Ok(Value::Number(42)), }, ConformanceCase { id: "RFC7159-7.1", section: "7", level: RequirementLevel::Must, description: "Unicode escape sequences \\uXXXX", input: r#""\u0041""#, expected: Ok(Value::String("A".into())), }, // ... one per MUST/SHOULD clause ]; #[test] fn rfc7159_full_conformance() { let mut pass = 0; let mut fail = 0; let mut xfail = 0; // Expected failures (known divergences) for case in RFC7159_CASES { let result = our_parser::parse(case.input); let verdict = match (&result, &case.expected) { (Ok(a), Ok(b)) if a == b => { pass += 1; "PASS" } (Err(_), Err(())) => { pass += 1; "PASS" } _ => { if is_known_divergence(case.id) { xfail += 1; "XFAIL" } else { fail += 1; eprintln!("FAIL {}: {}\n expected: {:?}\n actual: {:?}", case.id, case.description, case.expected, result); "FAIL" } } }; // Structured JSON-line output for CI parsing eprintln!("{{\"id\":\"{}\",\"verdict\":\"{verdict}\",\"level\":\"{:?}\"}}", case.id, case.level); } let total = pass + fail + xfail; eprintln!("\nRFC 7159: {pass}/{total} pass, {fail} fail, {xfail} expected-fail"); assert_eq!(fail, 0, "{fail} conformance tests failed"); } ``` ### Pattern 5: Contract Testing (API Conformance) **Use when:** Testing API compatibility between services. ```typescript // Consumer-driven contract: client defines expectations // Provider must satisfy ALL consumer contracts // Consumer side (tests what we NEED from the API) describe("User API Contract", () => { it("GET /users/:id returns user with email", async () => { const user = await api.getUser("user-123"); expect(user).toHaveProperty("id"); expect(user).toHaveProperty("email"); expect(typeof user.email).toBe("string"); // Contract: we don't care about other fields — only what we consume }); }); // Provider side (verifies all consumer contracts are satisfied) // Run against actual API implementation ``` ```bash # Hurl: HTTP-level conformance testing # tests/conformance/api/users.hurl GET http://localhost:3000/api/v1/users/me Authorization: Bearer {{token}} HTTP 200 [Asserts] jsonpath "$.id" exists jsonpath "$.email" isString jsonpath "$.subscriptionStatus" matches /^(none|active|past_due|cancelled)$/ ``` ### Pattern 6: Process-Based Conformance (External Runner) **Use when:** A standard conformance runner exists for your protocol. ```bash # Connect conformance suite: tests gRPC/Connect protocol compliance connectconformance --mode server \ --config conformance-config.yaml \ -- ./our-server # Web Platform Tests: browser conformance wpt run --channel dev --product our-browser # test262: JavaScript engine conformance test262-harness --hostType our-engine --hostPath ./our-engine ``` --- ## DISCREPANCIES.md (Mandatory) Every conformance harness accumulates intentional divergences. Document them ALL. ```markdown # Known Conformance Divergences ## DISC-001: Unicode width tables - **Reference:** Uses Unicode 13.0 width tables (go-runewidth v0.14) - **Our impl:** Uses Unicode 15.1 width tables (unicode-width v0.2) - **Impact:** Some CJK chars have different widths → alignment differs - **Resolution:** ACCEPTED — newer Unicode tables are more correct - **Tests affected:** lipgloss/cjk_alignment_* - **Review date:** 2026-03-15 ## DISC-002: Error message format - **Reference:** Returns "invalid input at byte 42" - **Our impl:** Returns "parse error: unexpected byte 0x2A at offset 42" - **Impact:** Error strings differ (semantics identical) - **Resolution:** ACCEPTED — we test error categories, not messages - **Tests affected:** parser/error_* ``` **Rules for DISCREPANCIES.md:** 1. Every divergence gets a sequential ID (DISC-NNN) 2. Must state whether ACCEPTED, INVESTIGATING, or WILL-FIX 3. Must list affected test cases 4. Must include review date (divergences can become stale) 5. Tests for accepted divergences use XFAIL, not SKIP --- ## Fixture Provenance (Non-Negotiable) Every fixture must record how it was generated: ``` tests/conformance/fixtures/ ├── PROVENANCE.md # How fixtures were generated ├── go_outputs/ │ ├── generated_with: go1.22.1 │ ├── command: go run ./cmd/gen-fixtures > fixtures/go_outputs/ │ └── git_ref: abc123 (tag: v0.15.2) └── python_reference.json ├── generated_with: python3.12 + mcp-agent-mail 0.9.1 └── command: python -m mcp_agent_mail.conformance.generate > python_reference.json ``` **Why:** When fixtures are regenerated 6 months later and results change, you need to know what version generated the originals to diagnose whether the change is a bug or a new feature. --- ## Compliance Report Generator ```rust fn generate_compliance_report(results: &[TestResult]) -> String { let mut by_section: BTreeMap<&str, SectionStats> = BTreeMap::new(); for result in results { let section = by_section.entry(result.section).or_default(); match result.level { Must => section.must_total += 1, Should => section.should_total += 1, May => section.may_total += 1, } if result.verdict == Pass { section.passing += 1; } if result.verdict == XFail { section.xfail += 1; } } // Output: Markdown table // | Section | MUST (pass/total) | SHOULD (pass/total) | Score | // |---------|-------------------|---------------------|-------| // | §2 | 15/15 | 8/10 | 95.8% | } ``` --- ## Anti-Patterns (Hard Constraints) | ✗ Never | Why | Fix | |---------|-----|-----| | Test implementation details, not spec behavior | Brittle, breaks on refactors | Test observable behavior only | | No DISCREPANCIES.md | Intentional divergences look like bugs to next developer | Document EVERY known deviation | | Golden files without `UPDATE_GOLDENS` workflow | Tedious to update → people skip updates | Add update mechanism + diff review | | Incomplete COVERAGE.md | False confidence in compliance | Track what ISN'T tested | | Fixtures without provenance | Can't reproduce or upgrade | Record generator version + command | | SKIP instead of XFAIL for known divergences | Skipped tests are invisible in reports | XFAIL documents AND tracks | | Test only happy paths | Error handling divergences are the most dangerous | Test invalid inputs too | | Regenerate fixtures without diff review | New fixture bugs look like passing tests | Always `git diff fixtures/` before commit | --- ## Checklist (Before Claiming Conformance) - [ ] Specification source identified and version pinned - [ ] Coverage matrix built: every MUST/SHOULD clause enumerated - [ ] MUST clause coverage ≥ 95% (100% target) - [ ] Fixtures generated from reference impl with recorded provenance - [ ] DISCREPANCIES.md documents every intentional divergence - [ ] XFAIL (not SKIP) for accepted divergences - [ ] Compliance report generated automatically - [ ] Round-trip tests for all serializable types - [ ] Error cases tested (not just happy paths) - [ ] CI regenerates report on every PR - [ ] Fixture update workflow documented and tested --- ## References | Need | Reference | |------|-----------| | Harness architecture deep-dive | [HARNESS-ARCHITECTURE.md](references/HARNESS-ARCHITECTURE.md) | | Fixture management patterns | [FIXTURE-PATTERNS.md](references/FIXTURE-PATTERNS.md) | | Cross-language porting guide | [CROSS-LANGUAGE.md](references/CROSS-LANGUAGE.md) | | Real-world examples from our projects | [OUR-PROJECTS.md](references/OUR-PROJECTS.md) | ## Relationship to Other Testing Skills | Technique | Use INSTEAD when | Use TOGETHER when | |-----------|-----------------|-------------------| | /testing-metamorphic | No spec exists (oracle problem) | MRs fill gaps where spec is ambiguous | | /testing-fuzzing | Finding crashes, not compliance | Fuzz-generated inputs feed conformance checks | | /extreme-software-optimization | Performance, not correctness | Conformance suite is the regression gate for optimizations | | /porting-to-rust | Need the full porting methodology | Conformance harness is PART of the porting workflow |