myform — the universal docs-format translator

Spec of record for the myform universal-translator pivot. Confirmed with the operator 2026-07-20; supersedes the scope implied by the package’s current name. Provenance: ~/my/self/dredge/memy/26-07-13_typst-first-fleet-migration-prompt.md.

1. Vision

myst-mdformat becomes myform: one absolute, typed document AST in the middle, format frontends and backends around it.

The one-sentence contract

myform.convert(text, src, dst) parses src-format text into a lossless document tree and renders that tree as idiomatic dst-format text, degrading unrepresentable constructs visibly and intelligently — never silently.

Formatting is the identity special case: convert(text, f, f) is the formatter, and for MyST/GFM it must remain byte-identical to what the mdformat plugin pipeline produces today. That fixed-point property is the safety rail for the whole pivot: the existing corpus and CommonMark-compliancy suites keep their full force.

1.1. Launch matrix (confirmed 2026-07-20 — superseded by shipped state)

Note

This table, and the “Read 2, write 4” decision in §2 and the reading non-goals in §1, record the plan as confirmed on 2026-07-20. All four readers shipped in v1.0.0, including the Typst reader this table defers. README.md carries the current matrix; universal-graph-and-sites.md carries the horizon beyond it. The plan is preserved here as written rather than rewritten, because a spec of record is evidence of what was decided.

Format	Read	Write	Engine
MyST Markdown	✅ launch	✅ launch	markdown-it pipeline (existing Parser)
GFM	✅ launch	✅ launch	markdown-it pipeline (mdformat-gfm)
Sphinx rST	🔜 fast-follow	✅ launch	docutils (read); native writer (write)
Typst	⏳ deferred	✅ launch	native writer; see §8 and the Typst-ecosystem memo

1.2. Non-goals at launch

• Typst reading (no viable Python parse path today — see the Typst-ecosystem memo).

• Pandoc-style long-tail formats (LaTeX, docx, …). The architecture must not preclude them; the launch must not wait for them.

• Semantic rewriting beyond degradation (the LM seam is for lossy-boundary judgment calls only).

2. Design decisions (operator-confirmed, 2026-07-20)

1. Pivot in place. This repo becomes myform (PyPI name verified free). Git history, test corpus, and CI carry over. The mdformat plugin survives as one facade inside the new package.

2. Read 2, write 4. Readers: MyST, GFM. Writers: MyST, GFM, rST, Typst.

3. LM assists are opt-in and degrade-only. Off by default; zero network calls when disabled; deterministic fallback always defined; routed through the local gateway (localhost:4000); disk-cached by node hash.

4. Lossy policy: annotate + report. Nearest-equivalent replacement that carries attendant info (captions, labels, headers); a provenance comment where the target supports comments; a per-file conversion report. Zero silent losses.

5. basis, lightest cut. my-basis becomes a dependency for RegexStore (wikiparse’s idiom) and light utilities — but import cost is measured and budgeted (§10); heavy leaves stay lazy.

3. Architecture

The pipeline generalizes from MD -> AST -> MD to:

        readers                     writers
MyST ──┐                                ┌── MyST   (via token bridge + mdformat)
GFM  ──┤→ markdown-it tokens → ┐        ├── GFM    (via token bridge + mdformat)
rST  ──┘ (docutils, follow-up) ├→ Doc ──┤── rST    (direct AST walk)
                               ┘   │    └── Typst  (direct AST walk)
                                   │
                     degradation engine + assist seam
                     (runs on Doc, per-writer capability table)

3.1. Package layout

myform/
├── __init__.py        # public API: convert(), format(), Doc; PEP 562-lazy like basis
├── model/
│   ├── nodes.py       # typed node model (myst-spec/mdast-aligned)
│   ├── bridge.py      # tokens ⇄ Doc (bijective for the markdown pipeline)
│   └── walk.py        # visitors, transforms, node addressing
├── readers/
│   ├── base.py        # Reader protocol + registry
│   ├── myst.py        # markdown-it + Parser rules → Doc
│   └── gfm.py         # markdown-it + gfm rules → Doc
├── writers/
│   ├── base.py        # Writer protocol, Capability table, registry
│   ├── myst.py        # Doc → tokens → mdformat render (fixed point)
│   ├── gfm.py         # same engine, GFM capability table
│   ├── rst.py         # direct AST walk
│   └── typst.py       # direct AST walk
├── degrade/
│   ├── engine.py      # capability-driven lowering pass over Doc
│   ├── lowerings.py   # declarative lowering rules (node kind × target)
│   └── report.py      # ConversionReport (+ stderr summary, JSON sidecar)
├── assist/
│   ├── base.py        # Assist protocol; NullAssist (default)
│   ├── gateway.py     # localhost:4000 client (lazy import, stdlib HTTP)
│   └── cache.py       # disk cache keyed by (node hash, target, model)
├── config/
│   ├── options.py     # MyformConfig (generalizes MystOptions; per-format tags)
│   └── extensions.py  # .py extension loading (register readers/writers/lowerings)
├── markdown/          # the existing engine, relocated intact:
│   ├── Parser.py, Renderer.py, Spacer.py, Postprocessor.py,
│   ├── Sembr.py, Ignore.py, constants.py
├── plugin.py          # mdformat entry point facade (unchanged behavior)
├── cli.py             # `myform` CLI (convert + format verbs)
├── options.py         # MystOptions (mdformat-facing surface, kept)
└── utils.py           # NO_ESC, parse_match, ... (RegexStore migration lands here)

myst_mdformat (the old import path) is not preserved — the package has no external consumers yet; a clean break now is the whole point of pivoting before launch (dredge note). The mdformat entry-point name myst is preserved, so mdformat --extensions myst and mdformat.text(..., extensions={'myst'}) behave identically.

3.2. The token bridge (the load-bearing novelty)

MyST/GFM writing does not reimplement CommonMark rendering. Instead model/bridge.py maintains a faithful mapping between markdown-it token streams (including our myst_* token types) and the typed Doc tree:

Reader path

markdown-it tokens → Doc — every token becomes a typed node; unknown/plugin tokens are captured as UnknownNode with their full token payload (absoluteness guarantee).

Writer path (markdown targets)

Doc → tokens → mdformat.MDRenderer with this package’s RENDERERS/POSTPROCESSORS — so spacing (Spacer), sembr, ignore-comments, and every existing MyST opinion apply unchanged.

The bridge must satisfy to_tokens(from_tokens(T)) == T for every token stream the launch readers produce; this is property-tested over the whole corpus and the CommonMark spec suite. This is what makes AC3 (fixed point == today’s output) hold by construction rather than by re-implementation.

rST and Typst writers walk Doc directly; they never see tokens.

4. The core AST (myform.model)

Node vocabulary aligns with myst-spec / mdast so we inherit a documented, ecosystem-tested taxonomy instead of inventing one:

• Flow: Root, Paragraph, Heading, Blockquote, List/ListItem, Code, ThematicBreak, Table/TableRow/TableCell, Definition*/Footnote*, Math, Directive, Target, Comment, BlockBreak, FrontMatter, Ignored (verbatim span), Unknown.

• Phrasing: Text, Emphasis, Strong, Delete, InlineCode, InlineMath, Link, Image, Break, FootnoteReference, Role, InlineUnknown.

Every node is a pydantic model with:

type

The myst-spec node name (discriminator).

children

Typed child list (flow or phrasing, enforced).

span

Source position when the reader had one (line/col start–end); None on synthesized nodes.

info

Format-agnostic core fields (e.g. Heading.depth, Code.lang, Directive.name/args/options, Role.name).

raw

The escape hatch — the originating token/payload preserved verbatim, so nothing the reader saw is ever unrecoverable (Unknown nodes are only raw).

Design constraints:

• No behavior in nodes. Rendering/lowering logic lives in writers and the degradation engine; nodes are data (N-02: policy/mechanism separation).

• Attendant-info accessors. Directive.caption, Directive.label, Table.header, Image.alt etc. are derived properties used by the degradation engine’s carriage rules — one definition, every writer benefits.

• Hashing. Every node has a stable content hash (used by the assist cache and the report).

5. Writers and capability tables

Each writer declares a capability table: a mapping node kind → Capability where

class Capability(Enum):
    NATIVE = auto()   # renders idiomatically, no loss
    APPROX = auto()   # rendered via a lowering rule; report as 'approximated'
    CARRY  = auto()   # cannot render; attendant info is carried into a replacement; report as 'replaced'
    DROP   = auto()   # cannot render; dropped with a report entry (never silently)

The degradation engine (§6) consults the table before the writer runs, lowering the Doc to a tree the writer renders 100% natively. Writers therefore stay simple — they never contain “if this can’t be rendered” branches (N-06: the constraint lives in one gate).

Illustrative capability decisions (full tables live beside each writer and are corpus-tested):

Node	MyST	GFM	rST	Typst
Admonition directive	NATIVE	APPROX → > [!NOTE] alert	NATIVE (.. note::)	APPROX → prelude callout / quote
Role	NATIVE	CARRY → content + report	NATIVE	APPROX (ref/cite map; else CARRY)
Target	NATIVE	CARRY → <a id> when HTML allowed, else DROP	NATIVE (.. _t:)	NATIVE (<label>)
Math	NATIVE	APPROX → $...$ (GitHub math)	NATIVE	NATIVE (delimiter translation)
Block break +++	NATIVE	DROP (comment marker)	APPROX → transition	APPROX → #pagebreak() opt
Footnote	NATIVE	NATIVE	NATIVE	NATIVE
Tables (pipe)	NATIVE	NATIVE	APPROX → list-table	NATIVE (#table)

6. Degradation engine (myform.degrade)

A single pre-render pass: lower(doc, writer.capabilities, config, assist) -> (doc', ConversionReport).

• Lowering rules are declarative entries (node kind, target format) -> transform, registered the same decorator way as parser/renderer rules today. Rules must carry attendant info: a dropped figure directive contributes its caption as an emphasized paragraph under the image it wrapped; toctree stays native for rST and becomes a readable entry list for GFM/Typst, with Typst retaining the caption and reporting every dropped navigation-only option.

• Provenance comments: where the target has comments (rST .. myform: ..., Typst // myform: ..., MyST % ...), replaced/dropped nodes leave a one-line marker (config-suppressible: degrade.comments: false).

• ConversionReport: pydantic model listing (span, node kind, action, rule, note); rendered as a stderr summary by default and as JSON via CLI flag/API. An action != NATIVE entry for every non-native node is an invariant the tests assert — this is AC4’s “zero silent losses”.

• Strict escape hatch: degrade.mode: annotate | silent | strict — default annotate; strict raises on the first CARRY/DROP.

7. Assist seam (myform.assist)

The deterministic system runs first and always produces a result; the assist can only improve a lowering the rules flagged as judgment-call (e.g. “summarize this dropped raw-HTML block into a one-line caption”, “choose prose placement for an unmappable option”).

• Assist.suggest(request: AssistRequest) -> AssistResult | None; None means “use the deterministic fallback”.

• NullAssist is the default and the only path when assist.enabled: false — tests assert zero network attempts in that mode (AC5).

• GatewayAssist posts tiny JSON-schema-constrained prompts to the OpenAI-compatible gateway at localhost:4000 (model from config, default a super-light tier), with assist/cache.py memoizing on (node hash, target, model) under ~/.cache/myform/. Unreachable gateway ⇒ log once, fall back, never crash.

• Imports of the HTTP client are function-local; the assist package costs nothing at import myform.

8. Typst writer notes (pre-memo)

Reconciled with the Typst-ecosystem memo (landed 2026-07-20): there is no Typst-side ecosystem to join — no Python-accessible source AST (typst-py wraps only the compiler; both tree-sitter grammars are stale, no wheels) and no plugin-extensible formatter (typstyle is the sole maintained formatter, CLI/Rust/WASM only; typstfmt is archived). Direct emission is confirmed convergent prior art: mystmd’s official myst-to-typst exporter uses exactly this architecture.

• Emit canonical Typst directly from Doc (headings =, emphasis _/strong *, code fences, $...$ math with delimiter/function translation, <label> targets, @ref references, #figure/#image with captions, #table, // comments).

• Admonitions and other MyST-only constructs lower to either plain constructs or, when typst.prelude: myform is set, to calls into a small shipped myform.typ prelude (callout functions) — the config default is plain (no implicit dependency on our prelude).

• Final normalization may optionally shell out to typstyle when present (typst.postformat: auto|off|require) — never a hard dependency.

• Joining a Typst-side plugin ecosystem (the dredge note’s ???) is decided by the memo; the writer seam is deliberately indifferent to that outcome.

9. Config layer (myform.config)

• File: myform.yaml (also [tool.myform] in pyproject.toml), discovered upward from the target file like .mdformat.toml is.

• Schema: MyformConfig (pydantic, JSON-Schema-published like MystOptions today) with per-format sections (myst:, gfm:, rst:, typst:) and shared sections (degrade:, assist:). Every option carries an applies_to tag set; requesting an option for a format it cannot affect warns loudly but never crashes (the dredge note’s “fine, as long as it’s tagged and doesn’t break integrations”).

• MystOptions remains the mdformat-facade surface and the authority for the MyST options: MyformConfig’s myst: section reuses that schema directly — one source of truth, two projections (mdformat TOML/CLI and myform YAML).

• .py extensions: extensions: [path.py, package.module] entries are imported and given a register(registry) hook to add readers, writers, lowerings, or assists — the same decorator registries used internally (N-05: the extension surface is the internal surface).

10. Performance budget

Import time is a launch gate, measured in CI (the cold-start gate in tests/test_cli_myform.py and the import-latency budget in tests/test_typst_native.py, warn-then-fail thresholds):

• Baseline (recorded 2026-07-20, host pepper, worktree venv, py3.14): import mdformat 11.2 ms; plugin discovery including this plugin and its deps 377.8 ms; mdformat --version cold ≈ 0.18 s wall. The plugin-discovery slice is the budget that matters.

• Budget: import myform ≤ baseline + 20%; myform <file> cold ≤ mdformat cold + 20%.

• Cold-start gate (A.4): The 0.216 s warn threshold (0.18 s mdformat baseline + 20%) is a non-blocking warning for v1.0.0. Crossing it emits a UserWarning but does not fail the suite. The hard regression ceiling is 0.432 s (2× the warn threshold); exceeding that is a test failure. This gives a 2× scheduler-noise band before the gate bites, keeping CI green on loaded hosts while keeping the budget visible.

• Levers: PEP 562 lazy leaves (like basis), function-local imports for docutils/assist/yaml, RegexStore lazy compilation, and — if import my’s eager tail proves too heavy — deferring the basis migration of regex tables behind a lazy boundary (decision recorded here with numbers, not vibes).

11. Testing strategy

1. Existing suites carry over unchanged in force: testcases.md corpus (API + CLI), CommonMark v0.29 compliancy (AST-equality + idempotency), options, sembr, ignore. They now exercise the readers.myst → bridge → writers.myst path (AC2/AC3).

2. Bridge property tests: to_tokens(from_tokens(T)) == T over every corpus case and spec example.

3. Cross-format corpus: testcases.md heading grammar grows sibling expectation blocks — ##### GFM, ##### RST, ##### Typst next to ##### Expected (loader keys off the title; absent block = case not asserted for that target). Same handwritten-corpus philosophy, one file, four truths.

4. Degradation invariants: for every corpus case × target, every non-NATIVE node appears in the report; strict mode raises exactly when the report is non-empty; carried captions/labels appear in output text.

5. Assist: NullAssist network-free (socket-guarded test); GatewayAssist against a fake local server fixture; cache hit path.

6. Import benchmark (§10).

7. Gates: uv run ruff check, uv run pyrefly check, uv run pytest — green before every commit.

12. Execution plan

Waves (each wave lands as reviewable commits on its own agent branch):

1. W2 pivot: mechanical rename to myform/ with markdown/ submodule move; suite green. (inline)

2. W3+W4 AST & bridge & readers: nodes, bridge, MyST/GFM readers; bridge property tests. (the hard core — adversarial review budgeted here)

3. W5 markdown writers: token-bridge writers; AC3 fixed-point tests.

4. W8+W10 degradation engine + config: capability tables, lowerings, report; config schema. (adversarial review: this is where silent-loss bugs would hide)

5. W6+W7 rST/Typst writers: parallel module builds against the writer seam (module-builder fan-out; disjoint files).

6. W9 assist seam: null + gateway + cache.

7. W11 corpus expansion + gates, W12 report/closure.

Safety posture (ai_safety.md): everything stays on the agent branch; the operator reviews and merges; PyPI publish (the myform name claim) is operator-only. Design canon most at play: N-02 (nodes are data; policy in engine), N-05 (extension surface = internal surface), N-06 (degradation is one gate, not scattered ifs), N-07 (ConversionReport is the feedback surface for every lossy act).

13. Problem space

• myform pivot — this spec; implementation waves 1–7 follow immediately ⭐

  • Typst-ecosystem memo — in flight; reconciles §8 and the dredge note’s ??? ⏳

  • rST reader (docutils) — fast-follow card once the writer seam is stable 💤

  • Typst reader — gated on ecosystem maturity; memo decides ⛔

  • mdsite/report-pipeline integration — the dredge note’s second half; separate card, after launch 💤

  • Repo/remote rename (libs/mdformat → libs/myform) + repolist/copier row — repository move complete; fleet row follows in release normalization ✅