Conformance corpus

tests/conformance/ is the machine-owned corpus beside the handwritten tests/data/testcases.md suite. Each corpus directory has a MANIFEST.json that pins its upstream, revision, license, declared source evidence, and SHA-256 for every case file. A source entry has a stable source ID, locator, UTF-8 SHA-256, and either an external_excerpt or a hand_authored declaration. Hand-authored evidence also names the external rule it exercises. A case must bind its upstream_version, license, upstream_id, and source_id to that manifest evidence before the runner calls MyForm.

Every case records a source and target format, provenance, a minimization witness, and one executable property: idempotence, spec_conformance, differential, ast_equivalence, or metamorphic. Oracle kinds are an allowlist: specification, reference_implementation, and hand_verified_invariant. Their reference is a source ID, so it always resolves to checked-in evidence. A spec_conformance expected value must equal that evidence’s source text; target-normalization claims belong in other property types until an external target-format oracle is available. differential also requires a supported live oracle tool and exact version. The seed uses CommonMark 0.31.2 material under CC-BY-SA-4.0 and a hand-authored rST fixture checked against a Docutils 0.22.4 public-domain rule and live oracle.

Admission checks linkage and byte identity; it cannot establish that a human label or remote locator is honest. In particular, hand_verified_invariant records a declared relation, rather than proving it externally. A review must assess those declarations against the cited primary source. The loader limits the entire loaded cohort to 16 manifests, 64 cases, 64 KiB of source text, and 256 KiB of fixtures; it also limits each manifest to 64 KiB and each fixture to 16 KiB before reading it. Every case needs a non-empty witness tied to the checked-in source ID.

Run the focused harness and emit its review artifact with:

uv run pytest -q tests/test_conformance.py
uv run python -m tests.conformance.runner --scoreboard

The scoreboard is sorted by corpus, format pair, property, manifest SHA-256, and source SHA-256. It reports real passed and failed counts, and supplies the artifact identities needed to compare two executions.