The downlevel-mode oracle #

test/downlevel.mjs is the judge for downlevel mode: it calls transformSync(src, { mode: "downlevel", onError }) from src/index.js and checks the result against the oracle, ts.transpileModule. It is independent of the engine: it only exercises the wrapper's public API.

node test/downlevel.mjs               # run corpus/downlevel.jsonl
node test/downlevel.mjs enum          # filter by case name or category substring
node test/downlevel.mjs --self-test   # judge-itself tests, feeding fabricated engine output
node test/downlevel.mjs --dir <path> [--dir <path> ...] [--grep <regex>] [--baseline <file>] [--baseline-write] [--minutes N]

This doc describes the judge's method: what it checks, how it derives ground truth, and where its own rules live. For results, open findings, and known failures, see corpus/downlevel-findings.jsonl and test/baselines/*.txt instead of this file — this file only changes when the method changes.

Cases #

corpus/downlevel.jsonl holds one JSON object per line: {name, category, src}. Every case's src is a self-contained TypeScript program ending in export const out = ... (or, for import =/export = cases, export = ...), exposing whatever the case wants checked.

Category	What it covers
enum	enum/const enum down-leveling
namespace	instantiated-namespace down-leveling, merging, ambient members
paramprop	constructor parameter properties
importeq	import x = require(...) / import x = A.B
exporteq	export = ...
importeq-exporteq	both together
cast	legacy <T>x cast erasure (downlevel mode erases these; erase mode still reports them)
mixed	combinations, generics, decorators, ASI-sensitive layouts
mixedmodule	a file mixing CommonJS-only syntax with value-level ESM in one file (see "Mixed module styles in one file" below)
unhandled	syntax downlevel mode still can't erase (e.g. x as T + 1's precedence) — these pin the position-parity contract, not runtime behavior

For judgeCase (the per-case function test/downlevel.mjs shares between corpus mode and --self-test), every non-unhandled case is checked against four criteria:

1. Runtime equivalence against ts.transpileModule.
2. Line preservation: same line count as the input, and every line outside a down-leveled construct is byte-identical to erase mode's own output for that line.
3. Erasable parity: with no down-leveled constructs at all, downlevel output equals erase output exactly.
4. Errors: downlevel mode reports no error inside a down-leveled construct, and reports the same errors (by position) as erase mode everywhere else.

Module kind #

A case needs module: CommonJS instead of ESNext exactly when it contains an import x = require(...) or export = ... statement anywhere in the source (needsCommonJS walks the full AST, not just top-level statements). ts.transpileModule silently mishandles both of these under module: ESNext, even with verbatimModuleSyntax: true (no diagnostic, wrong output), so the judge must pick the module kind itself rather than trust a fixed one.

import = cases that target an external module use two fixed mocks, MOCK_MODULES in test/downlevel.mjs (./math, ./greeter), registered with both the reference's and the candidate's require, so the comparison stays fair without touching the real filesystem.

judgeCase also refuses a CommonJS case the type checker itself rejects for module syntax: invalidCjsCase re-checks the source under module: CommonJS + verbatimModuleSyntax, and a TS1202/TS1203/TS1286/TS1287 diagnostic there marks the case invalid case rather than judged (unless the file is a mixed-module-style case, see below, which is a different, expected shape of the same checker rule).

Module kind decided from ambient code #

needsCommonJS and moduleKindReports (below) both walk the AST through forEachNonAmbient, which skips everything nested inside a declare-flagged ModuleDeclaration (declare module "x" {}, declare global {}, declare namespace N {}). Ambient content is never emitted at runtime, so an import =/export = inside one doesn't make the enclosing file need CommonJS, and a plain export {} inside one doesn't make it ESM either. Both functions share this rule so a file's module kind is decided the same way everywhere it's decided.

Down-leveled span detection #

Line preservation (criterion 2) and the error filter (criterion 4) both need to know which character ranges of the input are "a down-leveled construct." This is the judge's own rule (downleveledSpans in test/downlevel.mjs), syntactic rather than checker-driven:

• enum/const enum, unless declare — the full span of the EnumDeclaration.
• namespace/module, unless declare and unless non-instantiated (body contains only interface/type declarations or nested non-instantiated namespaces, recursively) — the full span of the ModuleDeclaration. namespace A.B.C { ... } parses as nested ModuleDeclarations where each one's body is the next one, not a ModuleBlock directly, so isInstantiatedModuleBody unwraps that chain before looking for the real body.
• import x = require(...) and export = ... — the statement's own span, unless it's a mixed-module-style construct downlevel mode reports instead of converting (see below), in which case it is not a down-leveled span.
• A class whose constructor (the implementation, not a bodyless overload signature — node.members.find((m) => ts.isConstructorDeclaration(m) && m.body)) has any parameter with public/private/protected/readonly/override — the whole class declaration's span, not just the constructor or the touched parameters.
• A legacy <T>x cast (ts.isTypeAssertionExpression) — just its own <Type> prefix, unless erasing it there would be unsafe (castEraseIsUnsafe, see "Legacy cast erasure" below), in which case it's a hole instead of a span: a position the judge already knows downlevel mode must report an error at, same as erase mode does.

The whole-class choice for parameter properties is deliberately conservative. tsc's actual insertion point for the synthesized field assignments is itself part of what criterion 1 (runtime equivalence) exists to catch; predicting that exact line here would make line preservation sensitive to an implementation's chosen downlevel strategy, which this oracle must not be shaped by.

declare enum and non-instantiated namespace/declare namespace/declare module are deliberately not marked down-leveled: they're erasable, so both erase mode and downlevel mode should blank them identically, checked by the ordinary line-preservation path.

Legacy cast erasure #

Downlevel mode erases legacy <T>x type-assertion casts (erase mode still reports them as unsupported, unchanged). The cast and its type blank to spaces, exactly like a type annotation; the operand is left in place and continues through the ordinary expression grammar, so precedence, postfix chaining, nesting, and the existing generic-arrow-vs-cast disambiguation all fall out of the existing expression parser.

The ASI carve-out. A return/throw/non-starred-yield's own expression is a restricted production: no line terminator is allowed between the keyword and the expression's first token, enforced by the JS engine that runs the erased output. If that first token is a legacy cast's <, and a line terminator appears anywhere between it and the cast's operand, erasing the cast would blank away the only thing sitting between the keyword and that line terminator — retroactively triggering ASI on the keyword and silently dropping the operand. The engine's own erasure guard detects this and falls back to reporting the cast exactly as erase mode does, instead of erasing it unsafely.

castEraseIsUnsafe mirrors this from the AST, independently of the engine: ctx is threaded through the visitor ("restricted" for a return/throw/non-starred-yield operand, "arrowBody" for an arrow's concise body, "stmtStart" for an expression statement's own start, or null) rather than read off node.parent, and propagates both down into a cast chain's own operand (<A><B>{...} only ever flags the innermost <B>) and through whatever construct is built on top of a cast at one of these positions (<any>{} && x, <any>{} ? a : b, and similar — none of these change what the engine's own one-token lookahead sees).

Two related positions carry the same hazard for a different reason: an arrow's concise body (() => { id: "x" } parses as a block with a labeled statement, not an object literal) and expression-statement start ({/function/class there parse as a block/function/class declaration). castEraseIsUnsafe checks the operand's own first token (via firstTokenNode, which descends through whatever postfix chain the cast's operand builds on top of) against ObjectLiteralExpression (both positions) or FunctionExpression/ClassExpression (statement start only).

A yield*'s operand is never restricted this way: the grammar's "no line terminator" check is satisfied at the * itself, not re-checked between * and the expression, so a cast there is always safe to erase. See cast-yield-star-no-hazard in corpus/downlevel.jsonl.

A cast's down-leveled span is only its own <Type> prefix, not the whole cast-plus-operand span: the operand's text is always copied through unchanged in either mode, so it never needs excluding from line comparison, and narrowing it keeps an outer safe cast's span from ever swallowing an inner unsafe one in a chain like <A><B>{...}.

Mixed module styles in one file #

A real file can mix plain import/export with export =/import x = require(). ts.transpileModule under module: CommonJS rewrites every import to a require() call, so its output is self-consistently CommonJS; downlevel mode's contract is narrower — it down-levels import =/export = but leaves plain import/export exactly as written. A file mixing both therefore can't run under either module system once the CommonJS-only construct is converted, so downlevel mode reports each CommonJS-only construct through onError and leaves it unconverted, the same way erase mode reports every construct it can't erase.

The judge derives the same classification from the TS AST, independently of the engine:

• isEsmModuleDecl — a value-level ESM import/export declaration: one that survives type erasure (not import type/export type, not an all-type-only clause), excluding anything reached through a namespace body (a namespace member export of the same shape is not a module-kind signal).
• isCjsOnlyConstruct — import x = require(...) (not a plain namespace alias import x = A.B) or export =.
• moduleKindReports — walks the file (via forEachNonAmbient) tracking whether it has committed to ESM (mkEsm), CommonJS (mkCjs), or both (mixed). Once mixed, it records which construct(s) downlevel mode is expected to report: every further CommonJS-only construct (cjsReports), plus — since streaming means the engine can't un-emit an earlier conversion — the later ESM declaration when CommonJS-only syntax came first (esmReports), substituting for the CommonJS construct that already streamed.

downleveledSpans excludes every construct in cjsReports from its spans (it's reported, not down-leveled), and judgeCase's expected-errors list adds one synthetic error per node in esmReports (sorted back into position order, since the engine always emits errors in source order). A mixed file whose candidate reports the conflicting construct(s) lands in pass/pass-unhandled (skip-runtime, since any reported error skips the runtime comparison); it still fails if the candidate reports nothing. mixedModuleSelfTest (part of --self-test) proves this in both streaming directions.

Runtime equivalence #

Reference: ts.transpileModule(src, { compilerOptions: { target: ESNext, module: ESNext | CommonJS, verbatimModuleSyntax: true, useDefineForClassFields: true } }). Candidate: downlevel output from src/index.js. Both are evaluated and their exports compared: ESM via vm.SourceTextModule (the file re-execs itself under --experimental-vm-modules if that's not already active) with a stub linker that throws on any external import (every corpus case is self-contained, so this should never fire), CommonJS via a new Function(module, exports, require, code) wrapper with the mock require above.

Comparison is a custom structural walk, shapeOf/deepEqual, not a generic deep-equal:

• Records each object's own enumerable keys in their actual order, so property ordering set by useDefineForClassFields initialization order is part of the comparison.
• Records each value's constructor name, so an enum or class instance can't quietly compare equal to a plain object with the same keys.
• Arrays compare elementwise. NaN and BigInt get explicit handling (JSON.stringify, used to compare the two shapes, otherwise mishandles them).
• Functions compare as an opaque "[[function]]" marker: cross-realm/cross-source function identity isn't meaningful here.
• A setTimeout/setInterval handle (Timeout/Immediate) compares as "[[Timeout <delay> <repeat>]]": its internal libuv pointers depend on process state, not on the down-leveled code's correctness, so only the delay and repeat interval (which come from the code) are compared.
• If either side throws, both must throw, and err.constructor.name must match (not exact message text, since the two engines format messages differently for the same mistake).

Reference and candidate are two separate evaluations, so anything non-deterministic must replay identically on both: makeSandbox seeds Math.random with a fixed generator (mulberry32) and pins Date/Date.now() to a clock that starts at a fixed instant and advances 1ms per read.

Runtime equivalence is skipped whenever the candidate reported any error (skipRuntime, true once downlevelErrors.length > 0): an engine that reports an error for syntax it can't down-level is documented to leave that syntax in place, so its output for that case isn't valid JS and was never meant to run. The unhandled category exists entirely for this path: line preservation and error-position parity still apply, runtime equivalence does not.

Self-test (--self-test) #

judgeCase takes engine output directly, not necessarily from the real engine, so --self-test feeds it deliberately broken outputs and asserts each is rejected. The main fixture is a self-referencing enum (enum E { A = 1, B = E.A + 2 } — self-referencing so member-init order is externally observable):

• missing reverse mapping — caught by runtime equivalence (the exported reverse-lookup value goes missing).
• wrong init order — caught by runtime equivalence (B evaluates before A exists).
• lost line — every line joined with a space, caught by line-count preservation.
• extra error — a bogus error reported at [0,4], caught by the error-position check.

It also runs the correct fixture through the same path first and fails loudly if that's rejected, so a bug in the judge itself (not just a missing rejection) is caught too.

Several more self-tests run alongside the main one, each proving one piece of method above still works:

• holeExemptionSelfTest — the holes/spans exemption for a cast whose own erasure guard declines to erase it, nested inside an otherwise-down-leveled namespace: accepts the real fix's output, rejects a bogus error at the wrong position in the same span, and rejects the cast error being silently dropped (caught by the runtime check, since the undetected <any> text throws on evaluation).
• grammarSemanticSelfTest — proves semanticGrammarCode (see "What counts as a grammar diagnostic" below) routes a grammar-only diagnostic to grammar-rejected, and does not exempt a type-error-only file from the ordinary runtime comparison.
• permissiveSelfTest / permissiveRealmProbe — proves the permissive-stub linker (see "Permissive stubs" below) still rejects broken down-level output, and that built-ins inside a permissive sandbox are the real realm's intrinsics while an undeclared host global is a stub.
• mixedModuleSelfTest — proves the mixed-module-style classification (above) accepts a candidate that reports the conflict and rejects one that reports nothing, in both streaming directions.

Oxidase cross-check #

Every non-unhandled case also runs through oxidase.transpile() (bench/node_modules/oxidase), judged the same way except for the error check (oxidase's JS API doesn't report errors, so only line preservation and runtime equivalence apply). This is a sanity check of the judge, not of the engine: oxidase implements the same feature set independently, so broad agreement with ts.transpileModule through this judge is evidence the judge itself measures the right thing. A disagreement here is worth checking directly against ts.transpileModule's own output before assuming either the judge or the engine is wrong — most disagreements found this way have turned out to be independent oxidase bugs, not findings about typestrip. Oxidase support is optional: loadOxidase returns null when the module isn't present, and every mode (corpus, --dir, --self-test) skips the cross-check cleanly in that case.

Real-corpus method (--dir) #

node test/downlevel.mjs --dir <path> [--dir <path> ...] [--grep <regex>] [--baseline <file>] [--baseline-write] [--minutes N]

Judges arbitrary files instead of hand-written corpus cases. Repeatable --dir runs several roots in one pass; --grep keeps only files whose raw text matches the given regex (a cheap syntactic over-approximation, used when a root is too broad to walk unfiltered); --minutes bounds wall-clock time per run (default 20; files past the budget are counted budget-exceeded-skipped rather than judged).

judgeFile re-runs criteria 2-4 unchanged against each file, but replaces criterion 1 (runtime equivalence) with a method that doesn't need the file to have been written for this oracle:

• Binding accessor. collectTopLevelBindings walks the input's own top-level statements and collects every identifier they bind (variable/destructuring names, function/class/enum names, a namespace's outermost segment, import = bindings, plain import bindings). declared statements are skipped. After both outputs are generated and already measured for line/column preservation, the judge appends one statement to each, textually identical except for which output it's appended to: a call to a host-side __downlevelObserve callback with every binding read under a typeof guard, so a binding one side forgot to emit reads as undefined there instead of aborting the whole comparison with a ReferenceError.
• Stub linking. A real file imports modules the judge has no file for. In --dir mode, an unresolved name falls back to a permissive stub rather than an all-undefined one; see "Permissive stubs" below.
• Multi-file splitting. splitFilenameMarkers detects the TS test harness's // @filename: foo.ts convention and splits one source into one segment per marker, each judged independently (named path::foo.ts). A segment whose filename isn't .ts/.tsx/.mts/.cts, or is a .d.ts, is dropped; a file where every segment drops is multi-file-skipped. A segment that references another segment by relative import gets it through the stub linker, i.e. as a stub, an approximation rather than real cross-file resolution.
• Grammar TS rejects. grammarRejectedCode checks the parsed SourceFile's own parseDiagnostics for a code in the 1000-1999 syntax range. TS's parser still produces a best-effort AST for these (error recovery), so without this check such a file would fall through to the rest of the pipeline and produce a meaningless comparison; it's bucketed grammar-rejected instead.
• Reference-throws classification. Many real fixtures are deliberately type-invalid programs that ts.transpileModule happily emits (it doesn't type-check) but that throw at evaluation for reasons unrelated to down-leveling (TDZ, calling something that isn't a function, dead code). If the reference evaluation throws, the file is bucketed reference-throws and the runtime comparison is skipped: there's no trustworthy ground truth to compare against. If the candidate throws while the reference didn't, that's an ordinary fail.
• Console silencing. A handful of real fixtures call console.log as a side effect of the code under test. evaluateObserved wraps each evaluation in withSilencedConsole so that noise doesn't reach the judge's own output.

Each --dir run reports buckets: pass (bindings matched), pass-no-bindings (no top-level binding to observe, still scored on criteria 2-4), construct-pass/fail (from the construct-level fallback, below), pass-unhandled (candidate reported an error inside a down-leveled construct), fail (an actual disagreement — the only bucket that can regress a baseline), reference-throws, grammar-rejected, invalid-cjs, multi-file-skipped, evaluation-timed-out, batch-timed-out (a batch subprocess that hung was killed, see below), budget-exceeded-skipped, and oracle-error (the judge itself failed to parse or evaluate the comparison, not the candidate).

A run over tens of thousands of files is chunked into subprocess batches of 300 (BATCH_SIZE): each vm.SourceTextModule compiled from a distinct source never seems to be released by V8's module registry within one process, so a long single-process run reliably hits --max-old-space-size. spawnSync's own timeout (batchTimeoutMs) guarantees a hung batch still terminates the sweep; its files land in batch-timed-out rather than vanishing from the totals.

Known-failure ratchet #

--baseline <file> reads a sorted list of case identifiers (a file path, or path::segment.ts for a split multi-file case, relative to the repo root) that are allowed to stay in the fail bucket without failing the run; anything in fail that isn't on the list is "fresh" and fails the run. --baseline-write regenerates the file from the current run. Baseline identifiers are relative so they survive different checkout locations.

pnpm test:downlevel:typescript runs --dir against several vendor/TypeScript conformance directories plus a broad --grep-filtered sweep of compiler, against test/baselines/downlevel-typescript.txt. pnpm test:downlevel:oss runs scripts/oss-downlevel-sweep.mjs, which calls test/downlevel.mjs --dir once per vendor/oss/<project> with the same kind of --grep filter, against one shared baseline, test/baselines/downlevel-oss.txt; corpus/oss-downlevel-progress.json is written after every project so an interrupted sweep still leaves readable results.

Permissive stubs #

On real application code, the whole-module binding comparison gives almost no signal with an all-undefined stub: a real module imports real dependencies and reads real globals, and reading a property off undefined or calling it throws immediately, at module top level, before any down-leveled construct runs. In --dir mode, an unresolved name instead falls back to a permissive stub: a recursive Proxy (permissiveStub) that's callable, constructible, iterable (empty), coercible to a primitive ("", or 0 for a numeric hint), never a thenable, and whose every string-keyed property access returns another permissive stub (memoized per key).

What is stubbed, --dir mode only:

• Every ESM import from any specifier: each named import the file declares, plus default, is a permissive stub (namespace imports get the vm.SyntheticModule's namespace object, whose listed members are stubs).
• Every CJS require(id) whose id isn't one of the fixed MOCK_MODULES.
• Every member of the sandbox's process object other than the explicit ones (env, argv, platform, version, exit, exitCode, nextTick, on, once, removeListener, stdout.write, stderr.write).
• Every string-keyed global name ordinary lookup doesn't find (document, window, navigator, self, __dirname, ...), whether read bare or as globalThis.name.

What is not stubbed: the context realm's own intrinsics (Object, Array, Math, JSON, Promise, the Error family, Date, RegExp, ...), the explicit sandbox members (console, timers, URL, TextEncoder/TextDecoder, structuredClone, Buffer, process itself), members of Object.prototype read as globals, and anything the evaluated code itself declares.

How. installPermissiveGlobalFallback replaces the context global's prototype with a Proxy over the realm's Object.prototype. The global's own properties (every intrinsic) resolve first; only a name that misses there and on Object.prototype reaches the stub. Wrapping the sandbox object itself in a Proxy instead (the first version of this mechanism) stubbed the intrinsics too, since Node's contextify interceptor consults the sandbox before the realm's own globals; see permissiveRealmProbe (part of --self-test) for the check that catches a regression here.

Why this can't create a false pass, provided stubs only stand in for names nothing else defines: a permissive stub's typeof is always "function", and shapeOf collapses every function to "[[function]]" before recursing into its properties, so no content a stub fabricates ever reaches the comparison.

Candidate-only global reads fail. The fallback records every name it answers, per evaluation (globalMisses). After the bindings compare equal, the judge fails the file if the candidate's set has a name the reference's lacks (runtime: candidate read undeclared global(s) the reference didn't). Without the fallback, each such read would have thrown a ReferenceError on the candidate side only. This catches an unqualified reference to a namespace member (downlevel mode reads a where tsc reads N.a) and a top-level binding the candidate dropped. The reverse, a name only the reference reads, is not an error.

Construct-level fallback #

Whole-module evaluation still fails for plenty of real files: a top-level side effect calls a real dependency's API in a way no stub can fake meaningfully, or throws for a reason unrelated to down-leveling. When judgeFile's runtime check lands in reference-throws, judgeConstructsFallback tries each top-level enum and each top-level parameter-property class in the file in isolation instead:

• extractConstructUnit extracts the file's own declarations (enum/namespace/class/function/variable/import/type/interface, via isDeclarationStatement) up to and including the construct, dropping every top-level expression statement (the main throw source this routes around), and appends a probe statement: the construct's own binding for an enum, or a probe-constructed instance for a parameter-property class (probeArgLiteral picks a type-guessed literal per constructor parameter — "probe" for string, 1 for number or anything untyped, [] for an array type).
• judgeConstructUnit down-levels the extracted unit directly with the engine and compares the probe binding through the same permissive-stub binding-accessor path (judgeRuntimeByBindings) whole-file judging uses.

This is deliberately narrower than "the construct and its transitive dependencies": a sibling declared after the construct isn't included, and nothing pulled in this way is itself re-verified. Namespaces are not covered by this fallback: it's a known gap, left unattempted, since a namespace's live-binding merge semantics are exactly the kind of subtlety worth more attention than extraction machinery for a shape this fallback can't verify with confidence anyway. Coverage is reported honestly as its own bucket, construct-pass, rather than conflated with whole-module pass; a file with no qualifying top-level construct returns null from judgeConstructsFallback, and judgeFile falls back to reference-throws for it.

What counts as a grammar diagnostic #

grammarRejectedCode only ever sees ts.createSourceFile's own parseDiagnostics, pure parse-time errors. TypeScript's checker raises a second, larger category of "grammar" diagnostics after parsing, during binding and checking — a structural rule that decides purely from AST shape (node kind, parent chain, modifier set, declaration order) rather than from an inferred type, but happens to use the checker's ordinary error() call instead of a parse-time one. A file that fails only one of these is the same kind of "TS itself rejects this input" case grammarRejectedCode already carves out: matching downlevel mode's output against tsc's there isn't a meaningful contract either.

GRAMMAR_SEMANTIC_CODES (test/downlevel.mjs) is derived mechanically from the TypeScript compiler's own source, not by listing codes a known-bad file happens to emit:

1. Every Diagnostics.* identifier passed to one of the checker's five grammarError* helpers (grammarErrorOnNode, grammarErrorOnFirstToken, grammarErrorAtPos, grammarErrorOnNodeSkippedOn, grammarErrorAfterFirstToken, plus one program-level grammarErrorOnNode call for import-attribute resolution-mode), resolved to its numeric code, minus any identifier that also appears anywhere else in the checker outside one of those five calls — since that means the same code can also be raised by an ordinary, non-grammar check elsewhere.
2. A handful of codes added by hand, each because its only call site(s) decide purely from AST shape with no call into the checker's type-relation machinery, despite using the checker's ordinary error() helper rather than a grammarError* one (e.g. 1061 Enum_member_must_have_initializer, 2451 Cannot_redeclare_block_scoped_variable_0, 2332 this_cannot_be_referenced_in_current_location).
3. A further set of codes present after steps 1-2 are deliberately dropped: codes gated by compiler options this judge doesn't control the same way a real project's tsconfig would (experimentalDecorators, the import-attributes/assertions cluster, the dynamic-import-only-module cluster), and codes whose condition calls into the checker's type-relation machinery despite the grammarError* wrapper.

Target-gated codes (Private_identifiers_are_only_available_when_targeting_ECMAScript_2015_and_higher and similar) are kept: semanticGrammarCode always uses ts.ScriptTarget.ESNext, the most permissive target, so these can never fire spuriously.

semanticGrammarCode builds a full Program for the source (no lib, no type-reference resolution) and returns the first semantic/declaration diagnostic whose code is in GRAMMAR_SEMANTIC_CODES, or null. It honors a handful of // @option: value directives (testFileDirectives) from the TypeScript conformance test harness's own leading-comment convention — target, module, experimentalDecorators, isolatedModules, verbatimModuleSyntax, useDefineForClassFields — narrowly, only for the options the remaining code set could ever be sensitive to, so a test file that opts into a relevant setting still doesn't misfire. judgeFile and judgeConstructUnit both call it after grammarRejectedCode finds nothing, and a hit routes the file to the grammar-rejected bucket, same as a parse-time grammar error.

Known judge limitations #

These are open method limitations, not engine bugs. An open engine finding belongs in corpus/downlevel-findings.jsonl; a known failure on real code belongs in test/baselines/downlevel-typescript.txt or test/baselines/downlevel-oss.txt, not here.

• Mixed-file evaluation against a correct reference. The judge derives the right classification for a mixed-module-style file (above) from the AST, but evaluating such a file's candidate output still needs a real CJS/ESM interop loader this judge doesn't have. A multi-file-split segment can hit a related gap, since a split segment's own module kind is also decided file-by-file.
• The --grep restriction is a heuristic, not an exact AST-derived count. It can over-match (a comment or string containing a keyword like "enum") and, in principle, under-match a file whose only non-erasable construct uses formatting the regex doesn't anticipate.
• construct-pass coverage is real but narrow. It only ever fires for a file with a qualifying top-level enum or parameter-property class to extract; most reference-throws files have neither (a nested construct, a namespace, or a down-leveled construct that depends on a sibling declared after it), so most of the reference-throws bucket's engine behavior remains unverified by this judge, not just unverified by whole-module evaluation.
• Permissive stubs trade specificity for coverage. A real bug whose only symptom is "this value differs from what the real dependency would have returned" is invisible by construction: both sides get the same opaque stub. A clean --dir run is not a claim that downlevel mode behaves correctly against real third-party module content, only that it behaves correctly on the down-leveled syntax itself.
• A stub can still stand in for a value the candidate computes wrong. The candidate-only global-read check catches a name the candidate reads that the reference doesn't. It can't catch the candidate reading a member of a stub the reference didn't read, since stubs don't log member access, nor a stub passed where the reference passes a real value.
• Reprint-sensitive values are false failures. Function#toString and Error#stack both expose the emitted text, which tsc reprints differently. These files stay in the baselines as known failures; an engine fix can't remove them.
• Grammar errors on otherwise-invalid input remain a contract question. A file rejected only by a checker-level grammar diagnostic still reaches the runtime comparison if semanticGrammarCode doesn't cover that specific code; whether downlevel mode's output should match tsc's there at all isn't settled by this judge.