# myform — the universal docs-format translator

> **Spec of record** for the myform universal-translator pivot.
> Confirmed with the operator 2026-07-20; supersedes the scope implied by the package's current name.
> Provenance: `~/my/self/dredge/memy/26-07-13_typst-first-fleet-migration-prompt.md`.

## `1.` Vision

`myst-mdformat` becomes **`myform`**: one absolute, typed document AST in the middle, format frontends and backends around it.

:::{admonition} The one-sentence contract
:class: important

`myform.convert(text, src, dst)` parses `src`-format text into a lossless document tree and renders that tree as idiomatic `dst`-format text, degrading unrepresentable constructs *visibly and intelligently* — never silently.
:::

Formatting is the identity special case: `convert(text, f, f)` **is** the formatter, and for MyST/GFM it must remain byte-identical to what the mdformat plugin pipeline produces today.
That fixed-point property is the safety rail for the whole pivot: the existing corpus and CommonMark-compliancy suites keep their full force.

### `1.1.` Launch matrix (confirmed 2026-07-20 — superseded by shipped state)

```{note}
This table, and the "Read 2, write 4" decision in §2 and the reading non-goals in §1, record the plan as confirmed on 2026-07-20.
All four readers shipped in v1.0.0, including the Typst reader this table defers.
`README.md` carries the current matrix; [`universal-graph-and-sites.md`](universal-graph-and-sites.md) carries the horizon beyond it.
The plan is preserved here as written rather than rewritten, because a spec of record is evidence of what was decided.
```

| Format        | Read           | Write     | Engine                                             |
| :------------ | :------------- | :-------- | :------------------------------------------------- |
| MyST Markdown | ✅ launch      | ✅ launch | markdown-it pipeline (existing `Parser`)             |
| GFM           | ✅ launch      | ✅ launch | markdown-it pipeline (mdformat-gfm)                |
| Sphinx rST    | 🔜 fast-follow | ✅ launch | docutils (read); native writer (write)             |
| Typst         | ⏳ deferred    | ✅ launch | native writer; see §8 and the Typst-ecosystem memo |

### `1.2.` Non-goals at launch

- Typst *reading* (no viable Python parse path today — see the [Typst-ecosystem memo](typst-ecosystem-memo.md)).
- Pandoc-style long-tail formats (LaTeX, docx, ...).
  The architecture must not preclude them; the launch must not wait for them.
- Semantic rewriting beyond degradation (the LM seam is for lossy-boundary judgment calls only).

## `2.` Design decisions (operator-confirmed, 2026-07-20)

1. **Pivot in place.** This repo becomes `myform` (PyPI name verified free).
   Git history, test corpus, and CI carry over.
   The mdformat plugin survives as one facade inside the new package.
2. **Read 2, write 4.** Readers: MyST, GFM.
   Writers: MyST, GFM, rST, Typst.
3. **LM assists are opt-in and degrade-only.** Off by default; zero network calls when disabled; deterministic fallback always defined; routed through the local gateway (`localhost:4000`); disk-cached by node hash.
4. **Lossy policy: annotate + report.** Nearest-equivalent replacement that carries attendant info (captions, labels, headers); a provenance comment where the target supports comments; a per-file conversion report.
   Zero silent losses.
5. **basis, lightest cut.** `my-basis` becomes a dependency for `RegexStore` (wikiparse's idiom) and light utilities — but import cost is measured and budgeted (§10); heavy leaves stay lazy.

## `3.` Architecture

The pipeline generalizes from `MD -> AST -> MD` to:

```
        readers                     writers
MyST ──┐                                ┌── MyST   (via token bridge + mdformat)
GFM  ──┤→ markdown-it tokens → ┐        ├── GFM    (via token bridge + mdformat)
rST  ──┘ (docutils, follow-up) ├→ Doc ──┤── rST    (direct AST walk)
                               ┘   │    └── Typst  (direct AST walk)
                                   │
                     degradation engine + assist seam
                     (runs on Doc, per-writer capability table)
```

### `3.1.` Package layout

```
myform/
├── __init__.py        # public API: convert(), format(), Doc; PEP 562-lazy like basis
├── model/
│   ├── nodes.py       # typed node model (myst-spec/mdast-aligned)
│   ├── bridge.py      # tokens ⇄ Doc (bijective for the markdown pipeline)
│   └── walk.py        # visitors, transforms, node addressing
├── readers/
│   ├── base.py        # Reader protocol + registry
│   ├── myst.py        # markdown-it + Parser rules → Doc
│   └── gfm.py         # markdown-it + gfm rules → Doc
├── writers/
│   ├── base.py        # Writer protocol, Capability table, registry
│   ├── myst.py        # Doc → tokens → mdformat render (fixed point)
│   ├── gfm.py         # same engine, GFM capability table
│   ├── rst.py         # direct AST walk
│   └── typst.py       # direct AST walk
├── degrade/
│   ├── engine.py      # capability-driven lowering pass over Doc
│   ├── lowerings.py   # declarative lowering rules (node kind × target)
│   └── report.py      # ConversionReport (+ stderr summary, JSON sidecar)
├── assist/
│   ├── base.py        # Assist protocol; NullAssist (default)
│   ├── gateway.py     # localhost:4000 client (lazy import, stdlib HTTP)
│   └── cache.py       # disk cache keyed by (node hash, target, model)
├── config/
│   ├── options.py     # MyformConfig (generalizes MystOptions; per-format tags)
│   └── extensions.py  # .py extension loading (register readers/writers/lowerings)
├── markdown/          # the existing engine, relocated intact:
│   ├── Parser.py, Renderer.py, Spacer.py, Postprocessor.py,
│   ├── Sembr.py, Ignore.py, constants.py
├── plugin.py          # mdformat entry point facade (unchanged behavior)
├── cli.py             # `myform` CLI (convert + format verbs)
├── options.py         # MystOptions (mdformat-facing surface, kept)
└── utils.py           # NO_ESC, parse_match, ... (RegexStore migration lands here)
```

`myst_mdformat` (the old import path) is **not** preserved — the package has no external consumers yet; a clean break now is the whole point of pivoting before launch (dredge note).
The mdformat entry-point name `myst` is preserved, so `mdformat --extensions myst` and `mdformat.text(..., extensions={'myst'})` behave identically.

### `3.2.` The token bridge (the load-bearing novelty)

MyST/GFM writing does **not** reimplement CommonMark rendering.
Instead `model/bridge.py` maintains a faithful mapping between markdown-it token streams (including our `myst_*` token types) and the typed `Doc` tree:

Reader path
: `markdown-it tokens → Doc` — every token becomes a typed node; unknown/plugin tokens are captured as `UnknownNode` with their full token payload (absoluteness guarantee).

Writer path (markdown targets)
: `Doc → tokens → mdformat.MDRenderer` with this package's `RENDERERS`/`POSTPROCESSORS` — so spacing (`Spacer`), sembr, ignore-comments, and every existing MyST opinion apply unchanged.

The bridge must satisfy `to_tokens(from_tokens(T)) == T` for every token stream the launch readers produce; this is property-tested over the whole corpus and the CommonMark spec suite.
This is what makes AC3 (fixed point == today's output) hold *by construction* rather than by re-implementation.

rST and Typst writers walk `Doc` directly; they never see tokens.

## `4.` The core AST (`myform.model`)

Node vocabulary aligns with [myst-spec](https://mystmd.org/spec) / mdast so we inherit a documented, ecosystem-tested taxonomy instead of inventing one:

- **Flow**: `Root`, `Paragraph`, `Heading`, `Blockquote`, `List`/`ListItem`, `Code`, `ThematicBreak`, `Table`/`TableRow`/`TableCell`, `Definition*`/`Footnote*`, `Math`, `Directive`, `Target`, `Comment`, `BlockBreak`, `FrontMatter`, `Ignored` (verbatim span), `Unknown`.
- **Phrasing**: `Text`, `Emphasis`, `Strong`, `Delete`, `InlineCode`, `InlineMath`, `Link`, `Image`, `Break`, `FootnoteReference`, `Role`, `InlineUnknown`.

Every node is a pydantic model with:

`type`
: The myst-spec node name (discriminator).

`children`
: Typed child list (flow or phrasing, enforced).

`span`
: Source position when the reader had one (line/col start–end); `None` on synthesized nodes.

`info`
: Format-agnostic core fields (e.g. `Heading.depth`, `Code.lang`, `Directive.name/args/options`, `Role.name`).

`raw`
: The escape hatch — the originating token/payload preserved verbatim, so nothing the reader saw is ever unrecoverable (`Unknown` nodes are *only* `raw`).

Design constraints:

- **No behavior in nodes.** Rendering/lowering logic lives in writers and the degradation engine; nodes are data (N-02: policy/mechanism separation).
- **Attendant-info accessors.** `Directive.caption`, `Directive.label`, `Table.header`, `Image.alt` etc. are derived properties used by the degradation engine's carriage rules — one definition, every writer benefits.
- **Hashing.** Every node has a stable content hash (used by the assist cache and the report).

## `5.` Writers and capability tables

Each writer declares a **capability table**: a mapping `node kind → Capability` where

```python
class Capability(Enum):
    NATIVE = auto()   # renders idiomatically, no loss
    APPROX = auto()   # rendered via a lowering rule; report as 'approximated'
    CARRY  = auto()   # cannot render; attendant info is carried into a replacement; report as 'replaced'
    DROP   = auto()   # cannot render; dropped with a report entry (never silently)
```

The degradation engine (§6) consults the table *before* the writer runs, lowering the `Doc` to a tree the writer renders 100% natively.
Writers therefore stay simple — they never contain "if this can't be rendered" branches (N-06: the constraint lives in one gate).

Illustrative capability decisions (full tables live beside each writer and are corpus-tested):

| Node                 | MyST   | GFM                                         | rST                 | Typst                             |
| :------------------- | :----- | :------------------------------------------ | :------------------ | :-------------------------------- |
| Admonition directive | NATIVE | APPROX → `> [!NOTE]` alert                    | NATIVE (`.. note::`)  | APPROX → prelude callout / quote  |
| Role                 | NATIVE | CARRY → content + report                    | NATIVE              | APPROX (`ref`/`cite` map; else CARRY) |
| Target               | NATIVE | CARRY → `<a id>` when HTML allowed, else DROP | NATIVE (`.. _t:`)     | NATIVE (`<label>`)                  |
| Math                 | NATIVE | APPROX → `$...$` (GitHub math)                | NATIVE              | NATIVE (delimiter translation)    |
| Block break `+++`      | NATIVE | DROP (comment marker)                       | APPROX → transition | APPROX → `#pagebreak()` opt         |
| Footnote             | NATIVE | NATIVE                                      | NATIVE              | NATIVE                            |
| Tables (pipe)        | NATIVE | NATIVE                                      | APPROX → list-table | NATIVE (`#table`)                   |

## `6.` Degradation engine (`myform.degrade`)

A single pre-render pass: `lower(doc, writer.capabilities, config, assist) -> (doc', ConversionReport)`.

- **Lowering rules** are declarative entries `(node kind, target format) -> transform`, registered the same decorator way as parser/renderer rules today.
  Rules must carry attendant info: a dropped `figure` directive contributes its caption as an emphasized paragraph under the image it wrapped; `toctree` stays native for rST and becomes a readable entry list for GFM/Typst, with Typst retaining the caption and reporting every dropped navigation-only option.
- **Provenance comments**: where the target has comments (rST `.. myform: ...`, Typst `// myform: ...`, MyST `% ...`), replaced/dropped nodes leave a one-line marker (config-suppressible: `degrade.comments: false`).
- **ConversionReport**: pydantic model listing `(span, node kind, action, rule, note)`; rendered as a stderr summary by default and as JSON via CLI flag/API.
  An `action != NATIVE` entry for every non-native node is an invariant the tests assert — this is AC4's "zero silent losses".
- **Strict escape hatch**: `degrade.mode: annotate | silent | strict` — default `annotate`; `strict` raises on the first CARRY/DROP.

## `7.` Assist seam (`myform.assist`)

The deterministic system runs first and always produces *a* result; the assist can only improve a lowering the rules flagged as judgment-call (e.g. "summarize this dropped raw-HTML block into a one-line caption", "choose prose placement for an unmappable option").

- `Assist.suggest(request: AssistRequest) -> AssistResult | None`; `None` means "use the deterministic fallback".
- `NullAssist` is the default and the only path when `assist.enabled: false` — tests assert **zero network attempts** in that mode (AC5).
- `GatewayAssist` posts tiny JSON-schema-constrained prompts to the OpenAI-compatible gateway at `localhost:4000` (model from config, default a super-light tier), with `assist/cache.py` memoizing on `(node hash, target, model)` under `~/.cache/myform/`.
  Unreachable gateway ⇒ log once, fall back, never crash.
- Imports of the HTTP client are function-local; the assist package costs nothing at `import myform`.

## `8.` Typst writer notes (pre-memo)

Reconciled with the [Typst-ecosystem memo](typst-ecosystem-memo.md) (landed 2026-07-20): there is **no Typst-side ecosystem to join** — no Python-accessible source AST (`typst-py` wraps only the compiler; both tree-sitter grammars are stale, no wheels) and no plugin-extensible formatter (`typstyle` is the sole maintained formatter, CLI/Rust/WASM only; `typstfmt` is archived).
Direct emission is confirmed convergent prior art: mystmd's official `myst-to-typst` exporter uses exactly this architecture.

- Emit canonical Typst directly from `Doc` (headings `=`, emphasis `_`/strong `*`, code fences, `$...$` math with delimiter/function translation, `<label>` targets, `@ref` references, `#figure`/`#image` with captions, `#table`, `//` comments).
- Admonitions and other MyST-only constructs lower to either plain constructs or, when `typst.prelude: myform` is set, to calls into a small shipped `myform.typ` prelude (callout functions) — the config default is plain (no implicit dependency on our prelude).
- Final normalization may optionally shell out to `typstyle` when present (`typst.postformat: auto|off|require`) — never a hard dependency.
- Joining a Typst-side *plugin ecosystem* (the dredge note's `???`) is decided by the memo; the writer seam is deliberately indifferent to that outcome.

## `9.` Config layer (`myform.config`)

- **File**: `myform.yaml` (also `[tool.myform]` in `pyproject.toml`), discovered upward from the target file like `.mdformat.toml` is.
- **Schema**: `MyformConfig` (pydantic, JSON-Schema-published like `MystOptions` today) with per-format sections (`myst:`, `gfm:`, `rst:`, `typst:`) and shared sections (`degrade:`, `assist:`).
  Every option carries an `applies_to` tag set; requesting an option for a format it cannot affect warns loudly but never crashes (the dredge note's "fine, as long as it's tagged and doesn't break integrations").
- **`MystOptions` remains** the mdformat-facade surface and the authority for the MyST options: `MyformConfig`'s `myst:` section reuses that schema directly — one source of truth, two projections (mdformat TOML/CLI and myform YAML).
- **`.py` extensions**: `extensions: [path.py, package.module]` entries are imported and given a `register(registry)` hook to add readers, writers, lowerings, or assists — the same decorator registries used internally (N-05: the extension surface is the internal surface).

## `10.` Performance budget

Import time is a launch gate, measured in CI (the cold-start gate in `tests/test_cli_myform.py` and the import-latency budget in `tests/test_typst_native.py`, warn-then-fail thresholds):

- Baseline (recorded 2026-07-20, host pepper, worktree venv, py3.14): `import mdformat` 11.2 ms; plugin discovery including this plugin and its deps 377.8 ms; `mdformat --version` cold ≈ 0.18 s wall. The plugin-discovery slice is the budget that matters.
- Budget: `import myform` ≤ baseline + 20%; `myform <file>` cold ≤ mdformat cold + 20%.
- **Cold-start gate (A.4):** The 0.216 s warn threshold (0.18 s mdformat baseline + 20%) is a **non-blocking warning** for v1.0.0. Crossing it emits a `UserWarning` but does not fail the suite. The hard regression ceiling is 0.432 s (2× the warn threshold); exceeding that is a test failure. This gives a 2× scheduler-noise band before the gate bites, keeping CI green on loaded hosts while keeping the budget visible.
- Levers: PEP 562 lazy leaves (like basis), function-local imports for docutils/assist/yaml, `RegexStore` lazy compilation, and — if `import my`'s eager tail proves too heavy — deferring the basis migration of regex tables behind a lazy boundary (decision recorded here with numbers, not vibes).

## `11.` Testing strategy

1. **Existing suites carry over unchanged in force**: `testcases.md` corpus (API + CLI), CommonMark v0.29 compliancy (AST-equality + idempotency), options, sembr, ignore.
   They now exercise the `readers.myst → bridge → writers.myst` path (AC2/AC3).
2. **Bridge property tests**: `to_tokens(from_tokens(T)) == T` over every corpus case and spec example.
3. **Cross-format corpus**: `testcases.md` heading grammar grows sibling expectation blocks — `##### GFM`, `##### RST`, `##### Typst` next to `##### Expected` (loader keys off the title; absent block = case not asserted for that target).
   Same handwritten-corpus philosophy, one file, four truths.
4. **Degradation invariants**: for every corpus case × target, every non-NATIVE node appears in the report; `strict` mode raises exactly when the report is non-empty; carried captions/labels appear in output text.
5. **Assist**: `NullAssist` network-free (socket-guarded test); `GatewayAssist` against a fake local server fixture; cache hit path.
6. **Import benchmark** (§10).
7. Gates: `uv run ruff check`, `uv run pyrefly check`, `uv run pytest` — green before every commit.

## `12.` Execution plan

Waves (each wave lands as reviewable commits on its own agent branch):

1. **W2 pivot**: mechanical rename to `myform/` with `markdown/` submodule move; suite green. *(inline)*
2. **W3+W4 AST & bridge & readers**: nodes, bridge, MyST/GFM readers; bridge property tests. *(the hard core — adversarial review budgeted here)*
3. **W5 markdown writers**: token-bridge writers; AC3 fixed-point tests.
4. **W8+W10 degradation engine + config**: capability tables, lowerings, report; config schema. *(adversarial review: this is where silent-loss bugs would hide)*
5. **W6+W7 rST/Typst writers**: parallel module builds against the writer seam (module-builder fan-out; disjoint files).
6. **W9 assist seam**: null + gateway + cache.
7. **W11 corpus expansion + gates**, **W12 report/closure**.

Safety posture (`ai_safety.md`): everything stays on the agent branch; the operator reviews and merges; PyPI publish (the `myform` name claim) is operator-only.
Design canon most at play: N-02 (nodes are data; policy in engine), N-05 (extension surface = internal surface), N-06 (degradation is one gate, not scattered ifs), N-07 (ConversionReport is the feedback surface for every lossy act).

## `13.` Problem space

- **myform pivot** — *this spec; implementation waves 1–7 follow immediately* ⭐
  - **Typst-ecosystem memo** — *in flight; reconciles §8 and the dredge note's `???`* ⏳
  - **rST reader (docutils)** — *fast-follow card once the writer seam is stable* 💤
  - **Typst reader** — *gated on ecosystem maturity; memo decides* ⛔
  - **mdsite/report-pipeline integration** — *the dredge note's second half; separate card, after launch* 💤
  - **Repo/remote rename (`libs/mdformat` → `libs/myform`) + repolist/copier row** — *repository move complete; fleet row follows in release normalization* ✅