testing-conformance-harnesses

name	testing-conformance-harnesses
description	Build conformance test harnesses that verify implementations against specifications. Use when: porting libraries across languages, implementing RFCs/specs, building database engines, validating protocol compliance, cross-platform compatibility, API contract testing, golden file testing, round-trip validation, compliance matrices, differential testing against reference implementations.
metadata	{"filePattern":["**/conformance*","**/golden*","**/fixtures/**","**/reference*"],"bashPattern":["\\b(conformance|compliance|golden|differential|reference.impl|wire.compat)\\b"],"priority":60}

Conformance Test Harnesses

The One Rule: "Specifications aren't suggestions, they're contracts." A conformance harness mechanically verifies every MUST/SHOULD clause. If it's not tested, it's not conformant.

The Loop (Mandatory)

1. IDENTIFY    → What is the specification? (RFC, API spec, reference impl, formal grammar)
2. EXTRACT     → Enumerate every testable requirement (MUST > SHOULD > MAY)
3. FIXTURE     → Generate reference outputs (run reference impl → golden files)
4. HARNESS     → Build infrastructure: fixture loader, comparator, verdict engine
5. COVER       → Write tests: one per requirement, table-driven, tagged by level
6. DIVERGE     → Document every INTENTIONAL deviation in DISCREPANCIES.md
7. MATRIX      → Generate compliance report: features × status × platform
8. MAINTAIN    → Regenerate fixtures when reference impl updates; diff review

Coverage Accounting Matrix (Mandatory)

Before claiming conformance, prove it:

Spec Section	MUST Clauses	SHOULD Clauses	Tested	Passing	Divergent	Score
Section N	count	count	count	count	count	Pass/(MUST+SHOULD)

Rule: Score < 0.95 for MUST clauses = NOT conformant. Ship with known gaps documented, never with unknown gaps.

---

Decision Tree: Which Conformance Pattern?

What is the specification source?
│
├─ Reference implementation exists (Go, Python, C)
│  └─ DIFFERENTIAL TESTING (Pattern 1)
│     Run both implementations, compare outputs byte-for-byte
│     Examples: charmed_rust (Go→Rust), mcp_agent_mail_rust (Python→Rust)
│
├─ Formal spec exists (RFC, ISO, W3C)
│  └─ SPEC-DERIVED TESTS (Pattern 4)
│     One test per MUST/SHOULD clause, tagged by requirement level
│     Examples: JSON RFC 7159, HTTP/2 RFC 7540, SQL spec
│
├─ Serialization format
│  └─ ROUND-TRIP + GOLDEN FILES (Patterns 2 + 3)
│     Fixtures from reference impl + serialize→deserialize identity
│     Examples: protobuf, MessagePack, CBOR, database page format
│
├─ Network protocol
│  └─ PROCESS-BASED CONFORMANCE (Pattern 6)
│     External test runner drives your server/client
│     Examples: Connect conformance suite, WPT, test262
│
└─ API contract (OpenAPI, GraphQL)
   └─ CONTRACT TESTING (Pattern 5)
      Consumer-driven contracts, provider verification
      Examples: Pact, Dredd, Hurl

---

The Six Patterns

Pattern 1: Differential Testing (Reference Implementation)

Use when: A canonical implementation exists. This is the gold standard.

Architecture (from charmed_rust, mcp_agent_mail_rust):

tests/conformance/
├── src/
│   ├── harness/
│   │   ├── mod.rs          # Entry point
│   │   ├── traits.rs       # ConformanceTest trait (see below)
│   │   ├── runner.rs       # Collects + executes all tests
│   │   ├── fixtures.rs     # Loads golden files from reference impl
│   │   ├── comparison.rs   # Byte-level, structural, fuzzy comparison
│   │   ├── context.rs      # Test context: paths, config, temp dirs
│   │   └── logging.rs      # Structured JSON-line results
│   └── bin/
│       ├── run_conformance.rs    # `cargo run --bin run_conformance`
│       └── generate_report.rs    # Markdown compliance matrix
├── fixtures/
│   └── go_outputs/          # Generated by: go run ./cmd/gen-fixtures
│       └── lipgloss/
│           ├── border_rounded.golden
│           └── style_padding.golden
├── DISCREPANCIES.md          # Every intentional divergence
└── COVERAGE.md               # What's tested vs what's not

The ConformanceTest Trait:

pub trait ConformanceTest: Send + Sync {
    fn name(&self) -> &str;
    fn category(&self) -> TestCategory;
    fn requirement_level(&self) -> RequirementLevel;  // MUST, SHOULD, MAY
    fn run(&self, ctx: &TestContext) -> TestResult;
}

#[derive(Debug, Serialize)]
pub enum TestCategory { Unit, Integration, EdgeCase, Performance }

#[derive(Debug, Serialize)]
pub enum RequirementLevel { Must, Should, May }

#[derive(Debug, Serialize)]
#[serde(tag = "status")]
pub enum TestResult {
    Pass,
    Fail { reason: String },
    Skipped { reason: String },
    ExpectedFailure { reason: String },  // Known divergence (XFAIL)
}

Fixture-driven differential test:

#[test]
fn conformance_lipgloss_border_rounded() {
    let fixture = load_fixture("go_outputs/lipgloss/border_rounded.golden");
    let actual = Style::new()
        .border(Border::Rounded)
        .padding(1, 2)
        .render("Hello, World!");

    assert_eq!(actual, fixture.expected,
        "Rust rendering diverges from Go reference\n\
         Go output:   {:?}\n\
         Rust output: {:?}\n\
         Fixture:     {}",
        fixture.expected, actual, fixture.path.display());
}

Pattern 2: Golden File Testing

Use when: Output is complex, correct once verified, then frozen.

fn assert_golden(test_name: &str, actual: &str) {
    let golden_path = Path::new("tests/golden")
        .join(format!("{test_name}.golden"));

    if std::env::var("UPDATE_GOLDENS").is_ok() {
        fs::create_dir_all(golden_path.parent().unwrap()).unwrap();
        fs::write(&golden_path, actual).unwrap();
        eprintln!("UPDATED golden: {}", golden_path.display());
        return;
    }

    let expected = fs::read_to_string(&golden_path)
        .unwrap_or_else(|_| panic!(
            "Golden file not found: {}\n\
             Run with UPDATE_GOLDENS=1 to create it",
            golden_path.display()
        ));

    if actual != expected {
        let actual_path = golden_path.with_extension("actual");
        fs::write(&actual_path, actual).unwrap();
        panic!(
            "GOLDEN MISMATCH: {}\n\
             diff {} {}",
            test_name,
            golden_path.(),
            actual_path.(),
        );
    }
}

Workflow:

# First run: create golden files
UPDATE_GOLDENS=1 cargo test

# Subsequent runs: compare
cargo test

# After intentional changes:
diff tests/golden/report.golden tests/golden/report.actual
UPDATE_GOLDENS=1 cargo test
git diff tests/golden/  # Review every change before committing

Pattern 3: Round-Trip Conformance

Use when: Data must survive a serialize→deserialize cycle perfectly, AND must interoperate with a reference implementation.

/// Cross-implementation round-trip:
/// 1. Reference impl produces fixture
/// 2. Our impl parses it (must succeed)
/// 3. Our impl re-serializes it
/// 4. Reference impl parses our output (must succeed and match)
fn conformance_cross_impl_roundtrip(fixtures_dir: &Path) {
    for fixture_path in glob(fixtures_dir, "*.bin") {
        let reference_bytes = fs::read(&fixture_path).unwrap();

        // Step 1: We must be able to parse reference output
        let parsed = our_parser::parse(&reference_bytes)
            .unwrap_or_else(|e| panic!(
                "Cannot parse reference fixture {}: {e}",
                fixture_path.display()));

        // Step 2: We re-serialize
        let our_bytes = our_serializer::serialize(&parsed);

        // Step 3: Reference must be able to parse our output
        let reparsed = reference_parser::parse(&our_bytes)
            .unwrap_or_else(|e| panic!(
                "Reference cannot parse our output for {}: {e}",
                fixture_path.display()));

        // Step 4: Parsed values must match
        assert_eq!(parsed, reparsed,
            "Cross-impl round-trip diverged for {}",
            fixture_path.());
    }
}

Pattern 4: Spec-Derived Test Matrix

Use when: Implementing an RFC or formal specification.

struct ConformanceCase {
    id: &'static str,           // "RFC7159-2.1"
    section: &'static str,      // "2"
    level: RequirementLevel,    // Must, Should, May
    description: &'static str,
    input: &'static str,
    expected: Result<Value, ()>,
}

const RFC7159_CASES: &[ConformanceCase] = &[
    // Section 2: JSON Grammar
    ConformanceCase {
        id: "RFC7159-2.1",
        section: "2",
        level: RequirementLevel::Must,
        description: "A JSON text is a serialized value",
        input: "42",
        expected: Ok(Value::Number(42)),
    },
    ConformanceCase {
        id: "RFC7159-7.1",
        section: "7",
        level: RequirementLevel::Must,
        description: "Unicode escape sequences \\uXXXX",
        input: r#""\u0041""#,
        expected: Ok(Value::String("A".into())),
    },
    // ... one per MUST/SHOULD clause
];

#[test]
fn rfc7159_full_conformance() {
    let mut pass = 0;
    let mut fail = 0;
    let mut  = ; 

       RFC7159_CASES {
          = our_parser::(case.input);
          =  (&result, &case.expected) {
            ((a), (b))  a == b => { pass += ;  }
            ((_), (())) => { pass += ;  }
            _ => {
                 (case.id) {
                    xfail += ;
                    
                }  {
                    fail += ;
                    (,
                        case.id, case.description, case.expected, result);
                    
                }
            }
        };
        
        (,
            case.id, case.level);
    }

      = pass + fail + xfail;
    ();
    (fail, , );
}

Pattern 5: Contract Testing (API Conformance)

Use when: Testing API compatibility between services.

// Consumer-driven contract: client defines expectations
// Provider must satisfy ALL consumer contracts

// Consumer side (tests what we NEED from the API)
describe("User API Contract", () => {
  it("GET /users/:id returns user with email", async () => {
    const user = await api.getUser("user-123");
    expect(user).toHaveProperty("id");
    expect(user).toHaveProperty("email");
    expect(typeof user.email).toBe("string");
    // Contract: we don't care about other fields — only what we consume
  });
});

// Provider side (verifies all consumer contracts are satisfied)
// Run against actual API implementation

# Hurl: HTTP-level conformance testing
# tests/conformance/api/users.hurl
GET http://localhost:3000/api/v1/users/me
Authorization: Bearer {{token}}
HTTP 200
[Asserts]
jsonpath "$.id" exists
jsonpath "$.email" isString
jsonpath "$.subscriptionStatus" matches /^(none|active|past_due|cancelled)$/

Pattern 6: Process-Based Conformance (External Runner)

Use when: A standard conformance runner exists for your protocol.

# Connect conformance suite: tests gRPC/Connect protocol compliance
connectconformance --mode server \
  --config conformance-config.yaml \
  -- ./our-server

# Web Platform Tests: browser conformance
wpt run --channel dev --product our-browser

# test262: JavaScript engine conformance
test262-harness --hostType our-engine --hostPath ./our-engine

---

DISCREPANCIES.md (Mandatory)

Every conformance harness accumulates intentional divergences. Document them ALL.

# Known Conformance Divergences

## DISC-001: Unicode width tables
- **Reference:** Uses Unicode 13.0 width tables (go-runewidth v0.14)
- **Our impl:** Uses Unicode 15.1 width tables (unicode-width v0.2)
- **Impact:** Some CJK chars have different widths → alignment differs
- **Resolution:** ACCEPTED — newer Unicode tables are more correct
- **Tests affected:** lipgloss/cjk_alignment_*
- **Review date:** 2026-03-15

## DISC-002: Error message format
- **Reference:** Returns "invalid input at byte 42"
- **Our impl:** Returns "parse error: unexpected byte 0x2A at offset 42"
- **Impact:** Error strings differ (semantics identical)
- **Resolution:** ACCEPTED — we test error categories, not messages
- **Tests affected:** parser/error_*

Rules for DISCREPANCIES.md:

1. Every divergence gets a sequential ID (DISC-NNN)
2. Must state whether ACCEPTED, INVESTIGATING, or WILL-FIX
3. Must list affected test cases
4. Must include review date (divergences can become stale)
5. Tests for accepted divergences use XFAIL, not SKIP

---

Fixture Provenance (Non-Negotiable)

Every fixture must record how it was generated:

tests/conformance/fixtures/
├── PROVENANCE.md              # How fixtures were generated
├── go_outputs/
│   ├── generated_with: go1.22.1
│   ├── command: go run ./cmd/gen-fixtures > fixtures/go_outputs/
│   └── git_ref: abc123 (tag: v0.15.2)
└── python_reference.json
    ├── generated_with: python3.12 + mcp-agent-mail 0.9.1
    └── command: python -m mcp_agent_mail.conformance.generate > python_reference.json

Why: When fixtures are regenerated 6 months later and results change, you need to know what version generated the originals to diagnose whether the change is a bug or a new feature.

---

Compliance Report Generator

fn generate_compliance_report(results: &[TestResult]) -> String {
    let mut by_section: BTreeMap<&str, SectionStats> = BTreeMap::new();

    for result in results {
        let section = by_section.entry(result.section).or_default();
        match result.level {
            Must => section.must_total += 1,
            Should => section.should_total += 1,
            May => section.may_total += 1,
        }
        if result.verdict == Pass { section.passing += 1; }
        if result.verdict == XFail { section.xfail += 1; }
    }

    // Output: Markdown table
    // | Section | MUST (pass/total) | SHOULD (pass/total) | Score |
    // |---------|-------------------|---------------------|-------|
    // | §2      | 15/15             | 8/10                | 95.8% |
}

---

Anti-Patterns (Hard Constraints)

✗ Never	Why	Fix
Test implementation details, not spec behavior	Brittle, breaks on refactors	Test observable behavior only
No DISCREPANCIES.md	Intentional divergences look like bugs to next developer	Document EVERY known deviation
Golden files without UPDATE_GOLDENS workflow	Tedious to update → people skip updates	Add update mechanism + diff review
Incomplete COVERAGE.md	False confidence in compliance	Track what ISN'T tested
Fixtures without provenance	Can't reproduce or upgrade	Record generator version + command
SKIP instead of XFAIL for known divergences	Skipped tests are invisible in reports	XFAIL documents AND tracks
Test only happy paths	Error handling divergences are the most dangerous	Test invalid inputs too
Regenerate fixtures without diff review	New fixture bugs look like passing tests	Always git diff fixtures/ before commit

---

Checklist (Before Claiming Conformance)

• Specification source identified and version pinned
• Coverage matrix built: every MUST/SHOULD clause enumerated
• MUST clause coverage ≥ 95% (100% target)
• Fixtures generated from reference impl with recorded provenance
• DISCREPANCIES.md documents every intentional divergence
• XFAIL (not SKIP) for accepted divergences
• Compliance report generated automatically
• Round-trip tests for all serializable types
• Error cases tested (not just happy paths)
• CI regenerates report on every PR
• Fixture update workflow documented and tested

---

References

Need	Reference
Harness architecture deep-dive	HARNESS-ARCHITECTURE.md
Fixture management patterns	FIXTURE-PATTERNS.md
Cross-language porting guide	CROSS-LANGUAGE.md
Real-world examples from our projects	OUR-PROJECTS.md

Relationship to Other Testing Skills

Technique	Use INSTEAD when	Use TOGETHER when
/testing-metamorphic	No spec exists (oracle problem)	MRs fill gaps where spec is ambiguous
/testing-fuzzing	Finding crashes, not compliance	Fuzz-generated inputs feed conformance checks
/extreme-software-optimization	Performance, not correctness	Conformance suite is the regression gate for optimizations
/porting-to-rust	Need the full porting methodology	Conformance harness is PART of the porting workflow