# Conformance corpus

`tests/conformance/` is the machine-owned corpus beside the handwritten
`tests/data/testcases.md` suite. Each corpus directory has a `MANIFEST.json` that pins its
upstream, revision, license, declared source evidence, and SHA-256 for every case file. A source
entry has a stable source ID, locator, UTF-8 SHA-256, and either an `external_excerpt` or a
`hand_authored` declaration. Hand-authored evidence also names the external rule it exercises. A
case must bind its
`upstream_version`, `license`, `upstream_id`, and `source_id` to that manifest evidence before
the runner calls MyForm.

Every case records a source and target format, provenance, a minimization witness, and one
executable property: `idempotence`, `spec_conformance`, `differential`,
`ast_equivalence`, or `metamorphic`. Oracle kinds are an allowlist:
`specification`, `reference_implementation`, and `hand_verified_invariant`. Their
`reference` is a source ID, so it always resolves to checked-in evidence. A
`spec_conformance` expected value must equal that evidence's source text; target-normalization claims belong in
other property types until an external target-format oracle is available. `differential` also
requires a supported live oracle tool and exact version. The seed uses CommonMark 0.31.2 material
under CC-BY-SA-4.0 and a hand-authored rST fixture checked against a Docutils 0.22.4
public-domain rule and live oracle.

Admission checks linkage and byte identity; it cannot establish that a human label or remote
locator is honest. In particular, `hand_verified_invariant` records a declared relation, rather
than proving it externally. A review must assess those declarations against the cited primary
source. The loader limits the entire loaded cohort to 16 manifests, 64 cases, 64 KiB of source
text, and 256 KiB of fixtures; it also limits each manifest to 64 KiB and each fixture to 16 KiB
before reading it. Every case needs a non-empty witness tied to the checked-in source ID.

Run the focused harness and emit its review artifact with:

```console
uv run pytest -q tests/test_conformance.py
uv run python -m tests.conformance.runner --scoreboard
```

The scoreboard is sorted by corpus, format pair, property, manifest SHA-256, and source SHA-256.
It reports real passed and failed counts, and supplies the artifact identities needed to compare
two executions.