# The downlevel-mode oracle `test/downlevel.mjs` is the judge for downlevel mode: it calls `transformSync(src, { mode: "downlevel", onError })` from `src/index.js` and checks the result against the oracle, `ts.transpileModule`. It is independent of the engine: it only exercises the wrapper's public API. ```sh node test/downlevel.mjs # run corpus/downlevel.jsonl node test/downlevel.mjs enum # filter by case name or category substring node test/downlevel.mjs --self-test # judge-itself tests, feeding fabricated engine output node test/downlevel.mjs --dir [--dir ...] [--grep ] [--baseline ] [--baseline-write] [--minutes N] ``` This doc describes the judge's method: what it checks, how it derives ground truth, and where its own rules live. For results, open findings, and known failures, see `corpus/downlevel-findings.jsonl` and `test/baselines/*.txt` instead of this file — this file only changes when the method changes. ## Cases `corpus/downlevel.jsonl` holds one JSON object per line: `{name, category, src}`. Every case's `src` is a self-contained TypeScript program ending in `export const out = ...` (or, for `import =`/`export =` cases, `export = ...`), exposing whatever the case wants checked. | Category | What it covers | |---|---| | `enum` | `enum`/`const enum` down-leveling | | `namespace` | instantiated-namespace down-leveling, merging, ambient members | | `paramprop` | constructor parameter properties | | `importeq` | `import x = require(...)` / `import x = A.B` | | `exporteq` | `export = ...` | | `importeq-exporteq` | both together | | `cast` | legacy `x` cast erasure (downlevel mode erases these; erase mode still reports them) | | `mixed` | combinations, generics, decorators, ASI-sensitive layouts | | `mixedmodule` | a file mixing CommonJS-only syntax with value-level ESM in one file (see "Mixed module styles in one file" below) | | `unhandled` | syntax downlevel mode still can't erase (e.g. `x as T + 1`'s precedence) — these pin the position-parity contract, not runtime behavior | For `judgeCase` (the per-case function `test/downlevel.mjs` shares between corpus mode and `--self-test`), every non-`unhandled` case is checked against four criteria: 1. **Runtime equivalence** against `ts.transpileModule`. 2. **Line preservation**: same line count as the input, and every line outside a down-leveled construct is byte-identical to erase mode's own output for that line. 3. **Erasable parity**: with no down-leveled constructs at all, downlevel output equals erase output exactly. 4. **Errors**: downlevel mode reports no error inside a down-leveled construct, and reports the same errors (by position) as erase mode everywhere else. ## Module kind A case needs `module: CommonJS` instead of `ESNext` exactly when it contains an `import x = require(...)` or `export = ...` statement anywhere in the source (`needsCommonJS` walks the full AST, not just top-level statements). `ts.transpileModule` silently mishandles both of these under `module: ESNext`, even with `verbatimModuleSyntax: true` (no diagnostic, wrong output), so the judge must pick the module kind itself rather than trust a fixed one. `import =` cases that target an external module use two fixed mocks, `MOCK_MODULES` in `test/downlevel.mjs` (`./math`, `./greeter`), registered with both the reference's and the candidate's `require`, so the comparison stays fair without touching the real filesystem. `judgeCase` also refuses a CommonJS case the type checker itself rejects for module syntax: `invalidCjsCase` re-checks the source under `module: CommonJS` + `verbatimModuleSyntax`, and a `TS1202`/`TS1203`/`TS1286`/`TS1287` diagnostic there marks the case `invalid case` rather than judged (unless the file is a mixed-module-style case, see below, which is a different, expected shape of the same checker rule). ### Module kind decided from ambient code `needsCommonJS` and `moduleKindReports` (below) both walk the AST through `forEachNonAmbient`, which skips everything nested inside a `declare`-flagged `ModuleDeclaration` (`declare module "x" {}`, `declare global {}`, `declare namespace N {}`). Ambient content is never emitted at runtime, so an `import =`/`export =` inside one doesn't make the *enclosing* file need CommonJS, and a plain `export {}` inside one doesn't make it ESM either. Both functions share this rule so a file's module kind is decided the same way everywhere it's decided. ## Down-leveled span detection Line preservation (criterion 2) and the error filter (criterion 4) both need to know which character ranges of the *input* are "a down-leveled construct." This is the judge's own rule (`downleveledSpans` in `test/downlevel.mjs`), syntactic rather than checker-driven: - `enum`/`const enum`, unless `declare` — the full span of the `EnumDeclaration`. - `namespace`/`module`, unless `declare` and unless **non-instantiated** (body contains only `interface`/`type` declarations or nested non-instantiated namespaces, recursively) — the full span of the `ModuleDeclaration`. `namespace A.B.C { ... }` parses as nested `ModuleDeclaration`s where each one's `body` is the next one, not a `ModuleBlock` directly, so `isInstantiatedModuleBody` unwraps that chain before looking for the real body. - `import x = require(...)` and `export = ...` — the statement's own span, unless it's a mixed-module-style construct downlevel mode reports instead of converting (see below), in which case it is *not* a down-leveled span. - A class whose constructor (the implementation, not a bodyless overload signature — `node.members.find((m) => ts.isConstructorDeclaration(m) && m.body)`) has any parameter with `public`/`private`/`protected`/`readonly`/`override` — the **whole class declaration's span**, not just the constructor or the touched parameters. - A legacy `x` cast (`ts.isTypeAssertionExpression`) — just its own `` prefix, *unless* erasing it there would be unsafe (`castEraseIsUnsafe`, see "Legacy cast erasure" below), in which case it's a **hole** instead of a span: a position the judge already knows downlevel mode must report an error at, same as erase mode does. The whole-class choice for parameter properties is deliberately conservative. `tsc`'s actual insertion point for the synthesized field assignments is itself part of what criterion 1 (runtime equivalence) exists to catch; predicting that exact line here would make line preservation sensitive to an implementation's chosen downlevel strategy, which this oracle must not be shaped by. `declare enum` and non-instantiated `namespace`/`declare namespace`/`declare module` are deliberately *not* marked down-leveled: they're erasable, so both erase mode and downlevel mode should blank them identically, checked by the ordinary line-preservation path. ## Legacy cast erasure Downlevel mode erases legacy `x` type-assertion casts (erase mode still reports them as unsupported, unchanged). The cast and its type blank to spaces, exactly like a type annotation; the operand is left in place and continues through the ordinary expression grammar, so precedence, postfix chaining, nesting, and the existing generic-arrow-vs-cast disambiguation all fall out of the existing expression parser. **The ASI carve-out.** A `return`/`throw`/non-starred-`yield`'s *own* expression is a restricted production: no line terminator is allowed between the keyword and the expression's first token, enforced by the JS engine that runs the erased output. If that first token is a legacy cast's `<`, and a line terminator appears anywhere between it and the cast's operand, erasing the cast would blank away the only thing sitting between the keyword and that line terminator — retroactively triggering ASI on the keyword and silently dropping the operand. The engine's own erasure guard detects this and falls back to reporting the cast exactly as erase mode does, instead of erasing it unsafely. `castEraseIsUnsafe` mirrors this from the AST, independently of the engine: `ctx` is threaded through the visitor (`"restricted"` for a `return`/`throw`/non-starred-`yield` operand, `"arrowBody"` for an arrow's concise body, `"stmtStart"` for an expression statement's own start, or `null`) rather than read off `node.parent`, and propagates both *down* into a cast chain's own operand (`{...}` only ever flags the innermost ``) and *through* whatever construct is built on top of a cast at one of these positions (`{} && x`, `{} ? a : b`, and similar — none of these change what the engine's own one-token lookahead sees). Two related positions carry the same hazard for a different reason: an arrow's concise body (`() => { id: "x" }` parses as a block with a labeled statement, not an object literal) and expression-statement start (`{`/`function`/`class` there parse as a block/function/class declaration). `castEraseIsUnsafe` checks the operand's own first token (via `firstTokenNode`, which descends through whatever postfix chain the cast's operand builds on top of) against `ObjectLiteralExpression` (both positions) or `FunctionExpression`/`ClassExpression` (statement start only). A `yield*`'s operand is never restricted this way: the grammar's "no line terminator" check is satisfied at the `*` itself, not re-checked between `*` and the expression, so a cast there is always safe to erase. See `cast-yield-star-no-hazard` in `corpus/downlevel.jsonl`. A cast's down-leveled span is only its own `` prefix, not the whole cast-plus-operand span: the operand's text is always copied through unchanged in either mode, so it never needs excluding from line comparison, and narrowing it keeps an outer safe cast's span from ever swallowing an inner unsafe one in a chain like `{...}`. ## Mixed module styles in one file A real file can mix plain `import`/`export` with `export =`/`import x = require()`. `ts.transpileModule` under `module: CommonJS` rewrites every import to a `require()` call, so its output is self-consistently CommonJS; downlevel mode's contract is narrower — it down-levels `import =`/`export =` but leaves plain `import`/`export` exactly as written. A file mixing both therefore can't run under either module system once the CommonJS-only construct is converted, so downlevel mode reports each CommonJS-only construct through `onError` and leaves it unconverted, the same way erase mode reports every construct it can't erase. The judge derives the same classification from the TS AST, independently of the engine: - `isEsmModuleDecl` — a value-level ESM import/export declaration: one that survives type erasure (not `import type`/`export type`, not an all-type-only clause), excluding anything reached through a namespace body (a namespace member export of the same shape is not a module-kind signal). - `isCjsOnlyConstruct` — `import x = require(...)` (not a plain namespace alias `import x = A.B`) or `export =`. - `moduleKindReports` — walks the file (via `forEachNonAmbient`) tracking whether it has committed to ESM (`mkEsm`), CommonJS (`mkCjs`), or both (`mixed`). Once mixed, it records which construct(s) downlevel mode is expected to report: every further CommonJS-only construct (`cjsReports`), plus — since streaming means the engine can't un-emit an earlier conversion — the later ESM declaration when CommonJS-only syntax came first (`esmReports`), substituting for the CommonJS construct that already streamed. `downleveledSpans` excludes every construct in `cjsReports` from its spans (it's reported, not down-leveled), and `judgeCase`'s expected-errors list adds one synthetic error per node in `esmReports` (sorted back into position order, since the engine always emits errors in source order). A mixed file whose candidate reports the conflicting construct(s) lands in `pass`/`pass-unhandled` (skip-runtime, since any reported error skips the runtime comparison); it still fails if the candidate reports nothing. `mixedModuleSelfTest` (part of `--self-test`) proves this in both streaming directions. ## Runtime equivalence Reference: `ts.transpileModule(src, { compilerOptions: { target: ESNext, module: ESNext | CommonJS, verbatimModuleSyntax: true, useDefineForClassFields: true } })`. Candidate: downlevel output from `src/index.js`. Both are evaluated and their exports compared: ESM via `vm.SourceTextModule` (the file re-execs itself under `--experimental-vm-modules` if that's not already active) with a stub linker that throws on any external import (every corpus case is self-contained, so this should never fire), CommonJS via a `new Function(module, exports, require, code)` wrapper with the mock `require` above. Comparison is a custom structural walk, `shapeOf`/`deepEqual`, not a generic deep-equal: - Records each object's own enumerable keys **in their actual order**, so property ordering set by `useDefineForClassFields` initialization order is part of the comparison. - Records each value's constructor name, so an enum or class instance can't quietly compare equal to a plain object with the same keys. - Arrays compare elementwise. `NaN` and `BigInt` get explicit handling (`JSON.stringify`, used to compare the two shapes, otherwise mishandles them). - Functions compare as an opaque `"[[function]]"` marker: cross-realm/cross-source function identity isn't meaningful here. - A `setTimeout`/`setInterval` handle (`Timeout`/`Immediate`) compares as `"[[Timeout ]]"`: its internal libuv pointers depend on process state, not on the down-leveled code's correctness, so only the delay and repeat interval (which come from the code) are compared. - If either side throws, both must throw, and `err.constructor.name` must match (not exact message text, since the two engines format messages differently for the same mistake). Reference and candidate are two separate evaluations, so anything non-deterministic must replay identically on both: `makeSandbox` seeds `Math.random` with a fixed generator (mulberry32) and pins `Date`/`Date.now()` to a clock that starts at a fixed instant and advances 1ms per read. Runtime equivalence is skipped whenever the candidate reported any error (`skipRuntime`, true once `downlevelErrors.length > 0`): an engine that reports an error for syntax it can't down-level is documented to leave that syntax in place, so its output for that case isn't valid JS and was never meant to run. The `unhandled` category exists entirely for this path: line preservation and error-position parity still apply, runtime equivalence does not. ## Self-test (`--self-test`) `judgeCase` takes engine output directly, not necessarily from the real engine, so `--self-test` feeds it deliberately broken outputs and asserts each is rejected. The main fixture is a self-referencing enum (`enum E { A = 1, B = E.A + 2 }` — self-referencing so member-init order is externally observable): - **missing reverse mapping** — caught by runtime equivalence (the exported reverse-lookup value goes missing). - **wrong init order** — caught by runtime equivalence (`B` evaluates before `A` exists). - **lost line** — every line joined with a space, caught by line-count preservation. - **extra error** — a bogus error reported at `[0,4]`, caught by the error-position check. It also runs the correct fixture through the same path first and fails loudly if that's rejected, so a bug in the judge itself (not just a missing rejection) is caught too. Several more self-tests run alongside the main one, each proving one piece of method above still works: - `holeExemptionSelfTest` — the holes/spans exemption for a cast whose own erasure guard declines to erase it, nested inside an otherwise-down-leveled namespace: accepts the real fix's output, rejects a bogus error at the wrong position in the same span, and rejects the cast error being silently dropped (caught by the runtime check, since the undetected `` text throws on evaluation). - `grammarSemanticSelfTest` — proves `semanticGrammarCode` (see "What counts as a grammar diagnostic" below) routes a grammar-only diagnostic to `grammar-rejected`, and does *not* exempt a type-error-only file from the ordinary runtime comparison. - `permissiveSelfTest` / `permissiveRealmProbe` — proves the permissive-stub linker (see "Permissive stubs" below) still rejects broken down-level output, and that built-ins inside a permissive sandbox are the real realm's intrinsics while an undeclared host global is a stub. - `mixedModuleSelfTest` — proves the mixed-module-style classification (above) accepts a candidate that reports the conflict and rejects one that reports nothing, in both streaming directions. ## Oxidase cross-check Every non-`unhandled` case also runs through `oxidase.transpile()` (`bench/node_modules/oxidase`), judged the same way except for the error check (oxidase's JS API doesn't report errors, so only line preservation and runtime equivalence apply). This is a sanity check of the judge, not of the engine: oxidase implements the same feature set independently, so broad agreement with `ts.transpileModule` through this judge is evidence the judge itself measures the right thing. A disagreement here is worth checking directly against `ts.transpileModule`'s own output before assuming either the judge or the engine is wrong — most disagreements found this way have turned out to be independent oxidase bugs, not findings about typestrip. Oxidase support is optional: `loadOxidase` returns `null` when the module isn't present, and every mode (corpus, `--dir`, `--self-test`) skips the cross-check cleanly in that case. ## Real-corpus method (`--dir`) ```sh node test/downlevel.mjs --dir [--dir ...] [--grep ] [--baseline ] [--baseline-write] [--minutes N] ``` Judges arbitrary files instead of hand-written corpus cases. Repeatable `--dir` runs several roots in one pass; `--grep` keeps only files whose raw text matches the given regex (a cheap syntactic over-approximation, used when a root is too broad to walk unfiltered); `--minutes` bounds wall-clock time per run (default 20; files past the budget are counted `budget-exceeded-skipped` rather than judged). `judgeFile` re-runs criteria 2-4 unchanged against each file, but replaces criterion 1 (runtime equivalence) with a method that doesn't need the file to have been written for this oracle: - **Binding accessor.** `collectTopLevelBindings` walks the input's own top-level statements and collects every identifier they bind (variable/destructuring names, function/class/enum names, a namespace's outermost segment, `import =` bindings, plain `import` bindings). `declare`d statements are skipped. After both outputs are generated and already measured for line/column preservation, the judge appends one statement to *each*, textually identical except for which output it's appended to: a call to a host-side `__downlevelObserve` callback with every binding read under a `typeof` guard, so a binding one side forgot to emit reads as `undefined` there instead of aborting the whole comparison with a `ReferenceError`. - **Stub linking.** A real file imports modules the judge has no file for. In `--dir` mode, an unresolved name falls back to a *permissive* stub rather than an all-`undefined` one; see "Permissive stubs" below. - **Multi-file splitting.** `splitFilenameMarkers` detects the TS test harness's `// @filename: foo.ts` convention and splits one source into one segment per marker, each judged independently (named `path::foo.ts`). A segment whose filename isn't `.ts`/`.tsx`/`.mts`/`.cts`, or is a `.d.ts`, is dropped; a file where every segment drops is `multi-file-skipped`. A segment that references another segment by relative import gets it through the stub linker, i.e. as a stub, an approximation rather than real cross-file resolution. - **Grammar TS rejects.** `grammarRejectedCode` checks the parsed `SourceFile`'s own `parseDiagnostics` for a code in the 1000-1999 syntax range. TS's parser still produces a best-effort AST for these (error recovery), so without this check such a file would fall through to the rest of the pipeline and produce a meaningless comparison; it's bucketed `grammar-rejected` instead. - **Reference-throws classification.** Many real fixtures are deliberately type-invalid programs that `ts.transpileModule` happily emits (it doesn't type-check) but that throw at evaluation for reasons unrelated to down-leveling (TDZ, calling something that isn't a function, dead code). If the reference evaluation throws, the file is bucketed `reference-throws` and the runtime comparison is skipped: there's no trustworthy ground truth to compare against. If the candidate throws while the reference didn't, that's an ordinary `fail`. - **Console silencing.** A handful of real fixtures call `console.log` as a side effect of the code under test. `evaluateObserved` wraps each evaluation in `withSilencedConsole` so that noise doesn't reach the judge's own output. Each `--dir` run reports buckets: `pass` (bindings matched), `pass-no-bindings` (no top-level binding to observe, still scored on criteria 2-4), `construct-pass`/`fail` (from the construct-level fallback, below), `pass-unhandled` (candidate reported an error inside a down-leveled construct), `fail` (an actual disagreement — the only bucket that can regress a baseline), `reference-throws`, `grammar-rejected`, `invalid-cjs`, `multi-file-skipped`, `evaluation-timed-out`, `batch-timed-out` (a batch subprocess that hung was killed, see below), `budget-exceeded-skipped`, and `oracle-error` (the judge itself failed to parse or evaluate the comparison, not the candidate). A run over tens of thousands of files is chunked into subprocess batches of 300 (`BATCH_SIZE`): each `vm.SourceTextModule` compiled from a distinct source never seems to be released by V8's module registry within one process, so a long single-process run reliably hits `--max-old-space-size`. `spawnSync`'s own `timeout` (`batchTimeoutMs`) guarantees a hung batch still terminates the sweep; its files land in `batch-timed-out` rather than vanishing from the totals. ### Known-failure ratchet `--baseline ` reads a sorted list of case identifiers (a file path, or `path::segment.ts` for a split multi-file case, relative to the repo root) that are allowed to stay in the `fail` bucket without failing the run; anything in `fail` that isn't on the list is "fresh" and fails the run. `--baseline-write` regenerates the file from the current run. Baseline identifiers are relative so they survive different checkout locations. `pnpm test:downlevel:typescript` runs `--dir` against several `vendor/TypeScript` conformance directories plus a broad `--grep`-filtered sweep of `compiler`, against `test/baselines/downlevel-typescript.txt`. `pnpm test:downlevel:oss` runs `scripts/oss-downlevel-sweep.mjs`, which calls `test/downlevel.mjs --dir` once per `vendor/oss/` with the same kind of `--grep` filter, against one shared baseline, `test/baselines/downlevel-oss.txt`; `corpus/oss-downlevel-progress.json` is written after every project so an interrupted sweep still leaves readable results. ## Permissive stubs On real application code, the whole-module binding comparison gives almost no signal with an all-`undefined` stub: a real module imports real dependencies and reads real globals, and reading a property off `undefined` or calling it throws immediately, at module top level, before any down-leveled construct runs. In `--dir` mode, an unresolved name instead falls back to a *permissive* stub: a recursive `Proxy` (`permissiveStub`) that's callable, constructible, iterable (empty), coercible to a primitive (`""`, or `0` for a numeric hint), never a thenable, and whose every string-keyed property access returns another permissive stub (memoized per key). **What is stubbed**, `--dir` mode only: - Every ESM import from any specifier: each named import the file declares, plus `default`, is a permissive stub (`namespace` imports get the `vm.SyntheticModule`'s namespace object, whose listed members are stubs). - Every CJS `require(id)` whose `id` isn't one of the fixed `MOCK_MODULES`. - Every member of the sandbox's `process` object other than the explicit ones (`env`, `argv`, `platform`, `version`, `exit`, `exitCode`, `nextTick`, `on`, `once`, `removeListener`, `stdout.write`, `stderr.write`). - Every string-keyed global name ordinary lookup doesn't find (`document`, `window`, `navigator`, `self`, `__dirname`, ...), whether read bare or as `globalThis.name`. **What is not stubbed:** the context realm's own intrinsics (`Object`, `Array`, `Math`, `JSON`, `Promise`, the `Error` family, `Date`, `RegExp`, ...), the explicit sandbox members (`console`, timers, `URL`, `TextEncoder`/`TextDecoder`, `structuredClone`, `Buffer`, `process` itself), members of `Object.prototype` read as globals, and anything the evaluated code itself declares. **How.** `installPermissiveGlobalFallback` replaces the context global's prototype with a `Proxy` over the realm's `Object.prototype`. The global's own properties (every intrinsic) resolve first; only a name that misses there and on `Object.prototype` reaches the stub. Wrapping the sandbox object itself in a `Proxy` instead (the first version of this mechanism) stubbed the intrinsics too, since Node's contextify interceptor consults the sandbox before the realm's own globals; see `permissiveRealmProbe` (part of `--self-test`) for the check that catches a regression here. Why this can't create a false pass, provided stubs only stand in for names nothing else defines: a permissive stub's `typeof` is always `"function"`, and `shapeOf` collapses every function to `"[[function]]"` before recursing into its properties, so no content a stub fabricates ever reaches the comparison. **Candidate-only global reads fail.** The fallback records every name it answers, per evaluation (`globalMisses`). After the bindings compare equal, the judge fails the file if the candidate's set has a name the reference's lacks (`runtime: candidate read undeclared global(s) the reference didn't`). Without the fallback, each such read would have thrown a `ReferenceError` on the candidate side only. This catches an unqualified reference to a namespace member (downlevel mode reads `a` where `tsc` reads `N.a`) and a top-level binding the candidate dropped. The reverse, a name only the reference reads, is not an error. ## Construct-level fallback Whole-module evaluation still fails for plenty of real files: a top-level side effect calls a real dependency's API in a way no stub can fake meaningfully, or throws for a reason unrelated to down-leveling. When `judgeFile`'s runtime check lands in `reference-throws`, `judgeConstructsFallback` tries each top-level `enum` and each top-level parameter-property `class` in the file in isolation instead: - `extractConstructUnit` extracts the file's own declarations (enum/namespace/class/function/variable/import/type/interface, via `isDeclarationStatement`) up to and including the construct, dropping every top-level *expression statement* (the main throw source this routes around), and appends a probe statement: the construct's own binding for an enum, or a probe-constructed instance for a parameter-property class (`probeArgLiteral` picks a type-guessed literal per constructor parameter — `"probe"` for `string`, `1` for `number` or anything untyped, `[]` for an array type). - `judgeConstructUnit` down-levels the extracted unit directly with the engine and compares the probe binding through the same permissive-stub binding-accessor path (`judgeRuntimeByBindings`) whole-file judging uses. This is deliberately narrower than "the construct and its transitive dependencies": a sibling declared *after* the construct isn't included, and nothing pulled in this way is itself re-verified. Namespaces are **not** covered by this fallback: it's a known gap, left unattempted, since a namespace's live-binding merge semantics are exactly the kind of subtlety worth more attention than extraction machinery for a shape this fallback can't verify with confidence anyway. Coverage is reported honestly as its own bucket, `construct-pass`, rather than conflated with whole-module `pass`; a file with no qualifying top-level construct returns `null` from `judgeConstructsFallback`, and `judgeFile` falls back to `reference-throws` for it. ## What counts as a grammar diagnostic `grammarRejectedCode` only ever sees `ts.createSourceFile`'s own `parseDiagnostics`, pure parse-time errors. TypeScript's checker raises a second, larger category of "grammar" diagnostics after parsing, during binding and checking — a structural rule that decides purely from AST shape (node kind, parent chain, modifier set, declaration order) rather than from an inferred type, but happens to use the checker's ordinary `error()` call instead of a parse-time one. A file that fails only one of these is the same kind of "TS itself rejects this input" case `grammarRejectedCode` already carves out: matching downlevel mode's output against `tsc`'s there isn't a meaningful contract either. `GRAMMAR_SEMANTIC_CODES` (`test/downlevel.mjs`) is derived mechanically from the TypeScript compiler's own source, not by listing codes a known-bad file happens to emit: 1. Every `Diagnostics.*` identifier passed to one of the checker's five `grammarError*` helpers (`grammarErrorOnNode`, `grammarErrorOnFirstToken`, `grammarErrorAtPos`, `grammarErrorOnNodeSkippedOn`, `grammarErrorAfterFirstToken`, plus one program-level `grammarErrorOnNode` call for import-attribute resolution-mode), resolved to its numeric code, **minus** any identifier that also appears anywhere else in the checker outside one of those five calls — since that means the same code can also be raised by an ordinary, non-grammar check elsewhere. 2. A handful of codes added by hand, each because its only call site(s) decide purely from AST shape with no call into the checker's type-relation machinery, despite using the checker's ordinary `error()` helper rather than a `grammarError*` one (e.g. `1061` Enum_member_must_have_initializer, `2451` Cannot_redeclare_block_scoped_variable_0, `2332` this_cannot_be_referenced_in_current_location). 3. A further set of codes present after steps 1-2 are deliberately dropped: codes gated by compiler options this judge doesn't control the same way a real project's tsconfig would (`experimentalDecorators`, the import-attributes/assertions cluster, the dynamic-import-only-module cluster), and codes whose condition calls into the checker's type-relation machinery despite the `grammarError*` wrapper. Target-gated codes (`Private_identifiers_are_only_available_when_targeting_ECMAScript_2015_and_higher` and similar) are kept: `semanticGrammarCode` always uses `ts.ScriptTarget.ESNext`, the most permissive target, so these can never fire spuriously. `semanticGrammarCode` builds a full `Program` for the source (no lib, no type-reference resolution) and returns the first semantic/declaration diagnostic whose code is in `GRAMMAR_SEMANTIC_CODES`, or `null`. It honors a handful of `// @option: value` directives (`testFileDirectives`) from the TypeScript conformance test harness's own leading-comment convention — `target`, `module`, `experimentalDecorators`, `isolatedModules`, `verbatimModuleSyntax`, `useDefineForClassFields` — narrowly, only for the options the remaining code set could ever be sensitive to, so a test file that opts into a relevant setting still doesn't misfire. `judgeFile` and `judgeConstructUnit` both call it after `grammarRejectedCode` finds nothing, and a hit routes the file to the `grammar-rejected` bucket, same as a parse-time grammar error. ## Known judge limitations These are open method limitations, not engine bugs. An open engine finding belongs in `corpus/downlevel-findings.jsonl`; a known failure on real code belongs in `test/baselines/downlevel-typescript.txt` or `test/baselines/downlevel-oss.txt`, not here. - **Mixed-file evaluation against a correct reference.** The judge derives the right *classification* for a mixed-module-style file (above) from the AST, but evaluating such a file's candidate output still needs a real CJS/ESM interop loader this judge doesn't have. A multi-file-split segment can hit a related gap, since a split segment's own module kind is also decided file-by-file. - **The `--grep` restriction is a heuristic, not an exact AST-derived count.** It can over-match (a comment or string containing a keyword like "enum") and, in principle, under-match a file whose only non-erasable construct uses formatting the regex doesn't anticipate. - **`construct-pass` coverage is real but narrow.** It only ever fires for a file with a qualifying top-level enum or parameter-property class to extract; most `reference-throws` files have neither (a nested construct, a namespace, or a down-leveled construct that depends on a sibling declared after it), so most of the `reference-throws` bucket's engine behavior remains unverified by this judge, not just unverified by whole-module evaluation. - **Permissive stubs trade specificity for coverage.** A real bug whose only symptom is "this value differs from what the real dependency would have returned" is invisible by construction: both sides get the same opaque stub. A clean `--dir` run is not a claim that downlevel mode behaves correctly against real third-party module content, only that it behaves correctly on the down-leveled syntax itself. - **A stub can still stand in for a value the candidate computes wrong.** The candidate-only global-read check catches a name the candidate reads that the reference doesn't. It can't catch the candidate reading a *member* of a stub the reference didn't read, since stubs don't log member access, nor a stub passed where the reference passes a real value. - **Reprint-sensitive values are false failures.** `Function#toString` and `Error#stack` both expose the emitted text, which `tsc` reprints differently. These files stay in the baselines as known failures; an engine fix can't remove them. - **Grammar errors on otherwise-invalid input remain a contract question.** A file rejected only by a checker-level grammar diagnostic still reaches the runtime comparison if `semanticGrammarCode` doesn't cover that specific code; whether downlevel mode's output should match `tsc`'s there at all isn't settled by this judge.