Architecture #
A contributor's map of typestrip: how a call gets from JavaScript into the engine and back, how the engine is built, and why the build and the code look the way they do.
Terms follow glossary.md.
Repository layout #
| Path | Contents |
|---|---|
src/ |
The wrapper: core.js (platform-neutral), index.js (Node entry), browser.js (browser/worker/Deno/workerd entry). |
engine/ |
The engine's Zena source and the generated WebAssembly Text data segments it is merged with. |
scripts/build.sh |
Compiles and links build/erase.wasm and build/downlevel.wasm. |
scripts/gen/ |
Generators for engine/keywords.{zena,wat}, engine/chartable.{zena,wat}, engine/downlevel-text.wat, and shared memory layout constants (layout.mjs). |
scripts/debug/ |
Single-input debug tools (try.mjs, minimize.mjs, bisect.mjs, and others). They load build/erase.wasm by default; bisect.mjs and classify.mjs also read build/downlevel-debug.wasm's dbg() probe. See CONTRIBUTING.md. |
build/ |
Build output. Not checked in. |
test/ |
erase.mjs and downlevel.mjs are the judges; the rest cover the wrapper and types. |
corpus/ |
Small checked-in test inputs, findings, and regressions. |
vendor/ |
Fetched sources: ts-blank-space (the erase-mode oracle), TypeScript 6's test suites, corpus inputs, and the OSS corpus. Not checked in. |
bench/ |
Benchmarks against other tools. See benchmarks.md. |
docs/ |
Reference docs. |
engine/ files #
| File | Role |
|---|---|
strip.zena |
The ABI entry points and the stripping machine: regions, frame stack, emitter, anchors, errors, the statement/expression/class state machine. |
lex.zena |
The byte lexer. |
types.zena |
The type recognizer. |
keywords.zena, keywords.wat |
Keyword ids, keyword sets, and the perfect-hash lookup table. Generated. |
chartable.zena, chartable.wat |
The byte-class table for the lexer. Generated. |
downlevel.zena |
Downlevel mode's emitters: enums, namespaces, parameter properties, import x =, export =. |
downlevel-text.wat |
The text templates downlevel.zena emits, as a data segment. Generated. |
jsx.zena |
Downlevel mode's JSX output for the classic runtime: factory names, comment pragmas, text whitespace, string escaping, HTML entities. |
jsx-entities.wat |
TypeScript's HTML entity table for jsx.zena, as a data segment. Generated by scripts/gen/jsx-entities.mjs. |
hooks-erase.zena, hooks-downlevel.zena |
The two implementations of the hook seam between strip.zena and downlevel.zena and jsx.zena (see "Downlevel mode in the engine" below). |
main-erase.zena, main-downlevel.zena |
The two shipping entry points: the ABI exports, nothing else. |
debug.zena |
main-downlevel.zena plus the dbg() probe, built as build/downlevel-debug.wasm. |
rt.wat |
SIMD scan loops and memory.copy, called from Zena via @external("rt", ...). |
How a call flows #
transform/transformSync(src/core.js) validatemodeand resolveonError, then either process a whole string or open a stream.- The first call for a given engine slot (0 erase, 1 downlevel) compiles that engine's
.wasmmodule. Node reads it from disk; the browser entry streams it fromfetch. Compilation is cached per slot, shared between the sync and async paths. - Opening a stream acquires an idle
WebAssembly.Instancefor that slot, or creates one (an instance holds its own linear memory and globals, so it serves one stream at a time), and callsinit(mode). - Each chunk goes through
inbuf/push. A string chunk is UTF-8 encoded straight into the instance's memory withTextEncoder.encodeInto. AUint8Arraychunk is copied in directly, with no decode/encode round trip, as long as it is valid UTF-8 for the whole stream; the first invalid chunk falls back to the decode path for the rest of that stream. pushreturns a byte count; the wrapper reads that many bytes fromoutbuf()and UTF-8 decodes them back to a string (except on the raw-bytes path, which still decodes output, since the ABI's output is always UTF-8 text).- Errors are read from
errorsPtr()/errorCount()as(start, end)i32 pairs in UTF-16 units and passed toonError, or collected to throwUnsupportedSyntaxErrorif there is none. close()releases the instance back to its slot's idle pool, which keeps every released instance, so a later stream reuses one without paying instantiation again and repeated bursts of N concurrent streams settle on N instances. The most recently released instance is handed out first. Trimming is lazy: once a slot holds more thanIDLE = 4idle instances, an unref'dTRIM_MS = 1000timer runs. Each tick keeps enough idle instances to reach the highest concurrency seen since the previous tick again, and never fewer thanIDLE. It drops the least recently used rest, but always keeps the first instance. One to two seconds after the last stream opens, a pool is back toIDLEand the timer stops.
A whole-string call is a stream under the hood: wholeString opens one, feeds the string
in 64 KiB slices so memory stays at the streaming bound instead of growing to three bytes
per input character, and closes it.
The engine #
One forward pass, frame stack, resumable streaming #
The engine is a pushdown automaton, not a recursive-descent parser. Recursive descent
cannot suspend mid-construct without re-parsing from a checkpoint when input runs out,
which is quadratic for a construct spanning many chunks. Instead strip.zena keeps an
explicit stack of 16-byte frames ([state, a, b, c]) in linear memory, growable with
growRegion. step() dispatches on the top frame's state through an ordered if chain
(ordered by frequency: S_EXPR, S_POSTFIX, S_BINARY are the hot states). A frame
below the top is read with belowSt/belowA/belowB, computed as an offset from fp,
the top frame's own address (not a separate stack-pointer-plus-index calculation).
Streaming correctness rests on one rule.
Scan-then-commit #
Every step first does read-only lookahead: it marks a token position (let m = tb),
scans ahead any number of tokens including a whole type-recognizer call, and if the
lexer runs out of input (more == 1) it calls back(m) and returns having changed
nothing. Only after lookahead succeeds does the step commit: it advances the lexer by
exactly one token (adv()), or by exactly the tokens the lookahead already verified.
Because a step either fully commits or fully backs out, the lexer failing on the very
next token after a commit still leaves every frame consistent, and the next push simply
resumes lexing from pos. Frame fields must only ever be set from a token the lookahead
verified, never from a stale field: pushFrame conventions like c = 1 meaning "refresh
a from the current token on entry" exist specifically so a frame can be pushed before
its position is known to be safe.
push retries a step that ran out of input only once the input buffer has grown by at
least as many bytes as the failed attempt scanned ahead (retryAt), not on every chunk.
A single-chunk-at-a-time retry is quadratic in the construct's size across many chunks;
the geometric retry schedule makes it linear.
Lexer #
lex.zena is a streaming byte lexer over UTF-8 input in linear memory. Positions are
stream offsets, not buffer-relative addresses (delta converts between them). When a
token touches the end of buffered input before fin, scan() sets more, leaves pos
at the token's pre-trivia start, and produces nothing, so the same scan can be retried
after more bytes arrive. The lexer caches its last result per position (LEXC, a
256-entry direct-mapped table keyed by tb & 255) because lookahead ends in a reseek
back to the mark token at least as often as it advances past it; scan() checks the
cache before re-lexing.
Context-dependent rescans (rescanRegex, rescanTemplate, rescanGreater,
rescanLess) are driven by the parser, not the lexer, because only the parser knows
whether / starts a regex or a division, or whether >> should split.
Type recognizer #
types.zena only finds where a type ends; it never emits. Every function takes the
lexer's current token, consumes a type (or type parameter list, type argument list,
heritage clause, object type body), and leaves the lexer on the first token after it with
typeEnd at the end of the last token consumed. A false return means either the input
doesn't match a type, or the lexer ran out of input (callers check more). The parser
then blanks [start, typeEnd) as one edit.
Because every input is valid TypeScript, the recognizer resolves cover-grammar
ambiguity positionally instead of by speculation: inside (...), a : after a complete
assignment expression is a type annotation, and ? followed by :, ,, ) or = is
an optional marker, so arrow parameters and parenthesized expressions share one parse.
The one real speculation is < after an operand: the engine scans a type argument list,
then applies TypeScript's own "can this follow a type argument list in an expression"
rule, falling back to less-than; the scan is bounded by the argument list itself, so
there is no rollback cost. <T>( at an operand's start is decided by a balanced skip of
the parentheses followed by =>/: (generic arrow) or not (legacy cast).
Keywords and byte classes #
keywords.zena's keywordAt(p, n) is a perfect hash: one i64 multiply
(w0 * KW_MUL >> 56) into a 256-slot table gives a candidate keyword id, which is then
verified against the keyword's own stored bytes and length. This replaces what was, on
an older design, a chain of if (n == N && bytes == X) return K comparisons per
keyword. Keyword membership tests (keywordIn, keywordInHi) and token-type membership
tests (tokenIn) are precomputed i64 bitmasks (KS_*/TS_*), generated from named
lists in scripts/gen/keywords.mjs, instead of long || chains; a mask test is one
and against a constant.
chartable.zena's CLASS table is a 256-byte array of bit flags
(ID_START, DIGIT, ID_CHAR = ID_START | DIGIT, WS, NL). One getU8(CLASS + c)
load and a mask classifies a byte, replacing range-check chains for identifier start/
continue, digit, and the whitespace/line-terminator if-chain in the lexer's hot loops.
Both tables are data segments (engine/keywords.wat, engine/chartable.wat),
written by scripts/gen/keywords.mjs and scripts/gen/chartable.mjs and merged into
every build by wasm-merge (see "Build pipeline"). Zena itself cannot declare a data
segment; before this, both tables were code (compare chains, or array.new_fixed
constants built in the start function), which was both bigger and less compressible.
Statement heads #
A statement's first token decides whether it is a declaration, and TypeScript's own
isStartOfDeclaration plus parseModifiers do this with one lookahead rather than a
same-line rule per contextual keyword. The engine follows the same shape: for the tokens
in the KS_HEAD mask (async, interface, type, module, namespace, declare,
const, enum, export, import, the member/statement modifiers, global, using,
await), one lookahead decides hdKind (the declaration keyword, or HD_NONE/HD_AT)
and hdFlags (HD_DECLARE, HD_EXPORT), with hdMods recording the modifier tokens
already consumed. declare, abstract, and the other ASI-sensitive modifiers need their
next token on the same line; interface/type need an identifier on the same line;
module/namespace need an identifier or string; export looks past an optional type.
Once a statement's head is decided it stays decided (the frame is "committed"), because
TypeScript does not re-run isDeclaration after the first modifier token:
declare type\nX = 1 is a type alias even though declare alone, at end of line, would
otherwise look like a plain identifier statement. From hdKind/hdFlags the engine
picks ts-blank-space's action: blank the whole statement (interface, type alias, most
declare forms), blank only the removed modifiers (class, function with a body), or keep
everything (variables, enums, imports, exports).
JSX #
With init's jsx bit set, hExpr decides < at an operand start with TypeScript's .tsx rule
(jsxArrowAhead): a generic arrow for <T,, <T = or <T extends X (each optionally after
const), JSX otherwise. Each element or fragment gets one S_JSX frame, [S_JSX, flags, phase, 0],
whose phase walks the opening tag, its attributes, its children, and the closing tag. A {...}
container pushes an ordinary S_SEQ above the frame, so any expression works inside it.
JSX children cannot go through scan(): whitespace and comment-like text inside an element are text,
not trivia. lexChild() lexes a child token at pos with no trivia skipping: text up to the next {
or <, or one of <, </, {. It never touches the token replay cache, since the cache holds the
ordinary token at the same offset. childLex records that the current token came from lexChild(),
so when a child token runs out of input, push() re-lexes it the same way on the next chunk. A child
lookahead restores its mark with seek(m) and a retry, never back(), which would re-lex the mark
normally. Inside a tag, rescanJsxName() widens a name over - and rescanJsxString() rescans an
attribute string with no escapes, the way rescanRegex() works for regular expressions.
Erase mode copies the JSX through and blanks only type arguments on a tag. Downlevel mode sets the
frame's J_XF flag and writes the classic runtime through the jsx* hooks into jsx.zena. That flag
is clear inside blanked or verbatim output. In the erase build, jsxOn() returns false and every other
jsx* hook is empty, so that build never writes JSX. Downlevel mode reads the file's @jsx and @jsxFrag
comment pragmas from the first token's leading trivia, in hTop, while that trivia is still buffered.
Downlevel mode in the engine #
Hooks, and why the erase build never links downlevel.zena #
strip.zena imports ./hooks.zena, never ./downlevel.zena directly. engine/hooks.zena
does not exist in the repository; scripts/build.sh copies either
engine/hooks-erase.zena (every hook a no-op stub) or engine/hooks-downlevel.zena
(every hook forwards to downlevel.zena) to that path before each compile. Because the
import graph is decided by which file is copied in, not by a runtime mode == 1 check,
build/erase.wasm's compile never reaches downlevel.zena at all, and wasm-opt's dead
code elimination has nothing to chase there. An earlier version of this guard forced
mode == 1 to a literal false at every call site and relied on wasm-opt to fold the
dead branches; that only saved part of the cost, because about 19 of strip.zena's
downlevel call sites read state that only downlevel.zena's own gated hooks ever set, so
they aren't behind a foldable mode == 1 test. Swapping the whole hook target removes
the module from the build's graph regardless of those sites.
The hook functions themselves (init, enumStart, enumStep, namespaceStart,
namespaceEnd, exportModifier, dropPending, hookSp, blockHook, functionHead,
className, varFlags, declNames, importEquals, exportAssign, classOpen,
classEnd, paramProp, ctorBody, castErase, ambientVar, moduleEsmSeen,
moduleCjsSeen, and the jsx* hooks into jsx.zena) are called from about twenty sites in strip.zena, each behind
mode == 1 or a field only downlevel mode ever sets.
What downlevel mode changes #
Downlevel mode down-levels enums, instantiated namespaces, parameter properties,
import x = require(), and export =, matching what ts.transpileModule emits for
them. Every inserted construct is free of line breaks, so line numbers are preserved
everywhere, and columns are preserved outside the converted construct itself.
- An enum becomes
var E;(function(E){...}).call(E||(E={}),E);, with the source's own braces as the IIFE body, sothisinside is the enum object. Members with bindable names are also bound as locals so later initializers can reference them bare. Whether a member gets a reverse mapping follows TypeScript's own evaluator: a syntactic string gets none, a statically known number gets one, a value constant to TypeScript but runtime-typed is decided bytypeof, and anything opaque to TypeScript gets one. - A namespace uses the same IIFE shape; dotted names nest, and exports are written after
their statement (
this.x = x;forconst/function/class/import aliases, an enumerable accessor forlet/varso the binding stays live in both directions). A namespace whose body has no values is restored from the retained input and re-blanked, which gives exactly erase mode's output for it. - A class with parameter properties anchors its body's
{; field declarations are inserted at the constructor's body start (or after the first top-levelsuper(...)in a derived class). import x = require()becomesconst x = require();import x = A.Bbecomes avaralias;export =becomesmodule.exports =.
Because Zena compiles a string literal to a wasm:js-string import, and the engine has
zero imports, every downlevel text fragment (var, the IIFE scaffolding, the per-member
templates) is instead a single packed data segment, engine/downlevel-text.wat, read by
offset. scripts/gen/downlevel-text.mjs regenerates both that .wat file and the
generated span of downlevel.zena between its GENERATED markers from one set of
templates; edit the templates there, not the generated Zena text.
ABI #
The ABI is the export contract between the wrapper and an engine module. Both
build/erase.wasm and build/downlevel.wasm implement it identically; the wrapper
cannot tell them apart except by the slot it loaded.
| Export | Signature | Meaning |
|---|---|---|
memory |
— | The module's own linear memory, auto-exported. |
init |
(mode: i32) -> void |
Starts a new input. Bit 0 of mode is 0 for erase and 1 for downlevel; the wrapper sends 1 only to build/downlevel.wasm. Bit 1 turns on JSX. |
pragma |
(len: i32, fragLen: i32) -> void |
Sets downlevel mode's JSX factories: len bytes of UTF-8 for the element factory, then fragLen for the fragment factory, written at inbuf(len + fragLen)'s pointer. Required after init when downlevel mode has JSX on. Erase mode ignores it. |
inbuf |
(cap: i32) -> i32 |
Returns a pointer where the host may write up to cap bytes of the next input chunk, growing memory first if needed. Valid until the next push. |
push |
(len: i32, fin: i32) -> i32 |
Consumes len bytes written at inbuf's last pointer. fin is 1 on the last chunk. Returns the number of output bytes now ready at outbuf(), or -1 if the input cannot be parsed (the stream is then dead until the next init). |
outbuf |
() -> i32 |
Pointer to the bytes the last push made ready. Valid until the next push or init. |
errorCount |
() -> i32 |
Number of (start, end) error pairs ready at errorsPtr(). |
errorsPtr |
() -> i32 |
Pointer to the error pairs, as consecutive i32s in UTF-16 units of the whole input. |
Every export is an i32 function or the memory; nothing else crosses the boundary. This
is also why --closed-world is a valid wasm-opt flag for this program (see "Build
pipeline"): no caller outside the module can reach an internal function or type.
Output contract #
Positions are reported, and in erase mode preserved, in UTF-16 units, because that is what JavaScript source positions are measured in.
Erase mode. Output has exactly the same UTF-16 length as the input. Every kept unit
is identical to the input; every changed unit is a space, or (to keep ASI or arrow-
function semantics across a blanked range) ;, (, or ). Line terminators are never
changed. So when a blanked range held multi-byte UTF-8, the output has fewer bytes than
that range of input, even though the UTF-16 length matches. This is judged against
ts-blank-space by test/erase.mjs: same length and kept units, a valid parse wherever the
oracle's is valid, an equal JS AST, and equal error start positions.
Downlevel mode. Lines are preserved everywhere; columns are preserved everywhere
outside a down-leveled construct, inside which only the rule "no inserted line breaks"
holds. This is judged against ts.transpileModule by test/downlevel.mjs, by evaluating
both outputs and comparing runtime behavior, not by text or AST equality (down-leveled
code is new JavaScript ts-blank-space never emits).
UTF-8 and UTF-16 #
Both directions across the ABI are UTF-8 bytes. The wrapper never sends a buffer that
splits a multi-byte code point across two push calls: TextEncoder.encodeInto only
writes whole characters for a string chunk, and the raw-bytes path
(src/core.js's pushBytes) holds back an incomplete trailing sequence for the next
chunk itself. A lone surrogate in a string chunk is encoded by encodeInto as U+FFFD (3
bytes, still one UTF-16 unit), so positions still line up; erase mode additionally
restores the original lone surrogate positionally in its output, since its contract
requires exact unit identity for kept text, while downlevel mode's output is not
positional and keeps U+FFFD as written.
Build pipeline #
./scripts/build.sh
- Compile.
zena-cli -O s build engine/main-erase.zena(andmain-downlevel.zena,debug.zena) produces a.wasmper entry point, withengine/hooks.zenacopied in first fromhooks-erase.zenaorhooks-downlevel.zenaas appropriate.zena -O 2produces a byte-identical module to-O sfor this program; every size and speed difference in this pipeline comes fromwasm-opt, not Zena's own optimization level. - Generate.
scripts/gen/keywords.mjs,scripts/gen/chartable.mjs, andscripts/gen/downlevel-text.mjsare run ahead of time (not bybuild.shitself) to produce the generated.zenasources and their matching.watdata-segment modules, all placed at fixed addresses fromscripts/gen/layout.mjs. - Merge.
wasm-asassembles each.watdata module plusengine/rt.wat(the SIMD scan loops andmemory.copythat Zena has no intrinsic for) to.wasm.wasm-mergelinks each into the compiled engine as an active data segment or direct call on the engine's own imported memory; this works because Zena cannot declare a data segment or call anything outside its own module except through@externalimports, so merging the "rt" imports into real functions and the table modules in as data is exactly what is needed.rt.wat's own exports are then removed (--remove-exports) since they only existed to satisfy the import. - Optimize.
wasm-optwith:--enable-simd --enable-gc --enable-reference-types --enable-bulk-memory --enable-sign-ext --enable-mutable-globals --enable-nontrapping-float-to-int: only the features Zena itself already emits.--all-featureswould letwasm-optintroduce ones no shipping engine implements yet (custom-descriptorexacttypes, which Node 24 rejects). SIMD is required forrt.wat'sv128loops; fixed-width SIMD ships in every engine that ships WasmGC.--closed-world --gufa-optimizing: valid because both artifacts export only i32 functions plus memory (see "ABI"), so no external caller can reach an internal function or type; this unlocks GUFA's whole-program type and value inference.--low-memory-unused: letsoptimize-added-constantsfoldpointer + constantinto a load or store offset. That fold is exact only when the sum cannot wrap past 4 GiB into the lowest 1 KiB of memory; the engine's addresses are always small positive sums, and the generated layout keeps all static data above 1 KiB (scripts/gen/layout.mjs), so the flag's precondition always holds here.--one-caller-inline-max-function-size=150: caps single-caller inlining. Without the cap,-O2's default inlining builds one function too large for V8's Liftoff baseline compiler to keep; TurboFan then optimizes it once calls accumulate, which is most of the RSS a short-lived engine instance pays (11.16 MiB delta after 20,000 calls uncapped doubles to roughly 21 MiB; see "Size and speed decisions"). Capped at 150, raw size grows by about 150 B and gzip drops by about 40 B relative to no cap, for a flat RSS curve.-O2 --converge --precompute-propagate --simplify-locals-nostructure -O2 --converge, run twice:-O2rather than-Oztrades about 1% more gzip for 8-20% more throughput (see "Size and speed decisions"). The--precompute-propagate/--simplify-locals-nostructurepass between the two-O2runs is a second constant-propagation and local-simplification sweep over code-O2's own inlining just produced, which finds a few more foldable values that a single pass misses.
scripts/build.sh prints raw and gzip sizes for both artifacts at the end, as a quick
regression check; it does not assert a budget.
Size and speed decisions #
Current sizes: erase.wasm 32,776 B raw / 13,648 B gzip -9; downlevel.wasm 52,059 B /
22,205 B. JSX added 1,026 B gzip to erase.wasm (the parser) and 4,367 B to downlevel.wasm (the
parser, jsx.zena, and the 2,181 B entity table), with no throughput change with jsx off.
Where the bytes go (erase.wasm, after the architectural and -O2 passes) #
| Area | Share |
|---|---|
strip.zena (statement/expression machine, output, anchors, errors) |
About 64% of code. Control flow: roughly 900 ifs, 1,000 calls, 2,100 reads of lexer/frame state. |
types.zena (type recognizer) |
About 15% of code. One recursive-descent function per TypeScript type production; scanPrimary alone is about 1.1 KB. |
lex.zena (lexer, scans) |
Smaller; includes the inlined keyword lookup. |
keywords.zena code |
About 21 B; the lookup itself is now a data segment plus roughly 200 B of code inside scan. |
| Static data (keyword hash table, keyword text, punctuator table, byte-class table) | About 1.06 KB. |
| ABI entry points | About 600 B. |
| Globals, types, exports, header | About 270-440 B; roughly 54 mutable globals at about 5 B each. |
downlevel.wasm adds downlevel.zena's hook targets and helpers (enum/namespace
emitters, the name table, about 9 KB) and its own text-template data segment (about
1.7 KB) on top of the same strip.zena/types.zena/lex.zena cost.
Kept #
- Tables as data, not code. Zena has no data-segment literal: a string becomes a
wasm:js-stringimport (which breaks the zero-import ABI), and an array literal becomesarray.new_fixedconstants built in the start function, both bigger and far less compressible than a raw byte table. Every table the engine needs (keyword hash table and text, punctuator map, byte-class table, downlevel text templates) is instead generated as a.watmodule andwasm-merged in as an active data segment. - Perfect-hash keywords. Replacing an 84-entry
if (n == N && bytes == X)chain with onei64multiply into a 256-slot table cutkeywords.zena's code from about 2.3 KB to about 21 B, at the cost of roughly 1.1 KB of data; net smaller and faster. - Frame addressing from
fp. The frame stack address isfp(the top frame's own address) plus a small constant offset, notstackBase + (sp - 1) * 16 + krecomputed from two globals at every one of roughly 200 call sites. --low-memory-unusedplus data above 1 KiB. Lets constant-offset folding apply everywhere addresses are used, for free, because the precondition (no address can wrap) is kept true by construction.- Keyword and token sets as bitmasks. A
KS_*/TS_*mask test replaces a chain of 4 or more||terms with oneand; worthwhile once a chain is long enough or repeated, not for every short one-off chain (see "Tried and rejected"). --closed-world --gufa-optimizing. Both artifacts export only i32 functions and memory, so closed-world analysis is exactly correct, not an approximation; GUFA's whole-program inference then finds foldable values a plain-O2pass cannot.- The 150-byte one-caller inline cap. See "Build pipeline"; keeps post-tier-up RSS flat (about 11 MiB at 20,000 calls) instead of doubling (about 21 MiB uncapped), for a cost of about 150 B raw / a 40 B gzip saving relative to no cap at all.
-O2over-Ozfor bothwasm-optpasses. About +1,982 B raw / +118 B gzip (+1.0%) onerase.wasm, and about +2,369 B raw / +196 B gzip (+1.1%) ondownlevel.wasm, for roughly +8-12% (lib.dom.d.ts) to +6-20% (a many-small-files OSS corpus) median throughput onerase.wasm, and roughly +7-16%/+10-19% ondownlevel.wasm. RSS after tier-up is unaffected, because the inline cap (not the-Olevel) is what bounds it. A roughly 1% gzip cost for consistent double-digit throughput clears the bar for "keep it" on this codebase.- SIMD scan loops (
rt.wat).find3(next occurrence of one of three bytes, 16 bytes per step) speeds up line/block comments, strings, and template text scanning.blank16(16-ASCII-byte-at-a-time space fill, stopping at a line terminator or a pending;) speeds up the emitter's blank path. Together they cost about 629 B raw / 296 B gzip onerase.wasmand roughly double throughput on comment- and whitespace- heavy input (lib.dom.d.tsroughly +77% in one measured run). SIMD requires--enable-simdat every build step, which every WasmGC-capable engine already ships. */pair scan for block comments (findClose). Once a block comment has seen a line terminator, the lexer stops only at*immediately followed by/(two overlapping 16-byte loads,i8x16.eqeach,v128.and, then a bitmask), instead offind3stopping at every*, which in JSDoc is once per line. Before the first line terminator it still usesfind3, because the comment'snlflag needs that stop. Costs +151 B raw / +45 B gzip (+0.36%) onerase.wasmand +153 B / +62 B (+0.35%) ondownlevel.wasm. Medians of 6 alternating rounds:lib.dom.d.ts158.0 to 165.9 MB/s (+5.0%) erase and 177.3 to 187.5 (+5.8%) downlevel, faster in every round; the corpus 119.6 to 122.1 (+2.1%) and 121.9 to 127.2 (+4.3%); the 20MB OSS sample flat (69.9 to 70.0, 69.2 to 69.5, 5 rounds), since runtime code has far fewer long comments. Line comments, strings, and template text already scan withfind3, so they got no new helper.- Lexer token-replay cache (
LEXC). A 256-entry direct-mapped cache of the lastscan()result at each stream position, because lookahead ends in areseekback to its mark at least as often as it advances past it (scan()was measured running about 1.54 times per distinct token). Costs a small amount of data and code for a double- digit throughput gain on both corpora. - A pool trimmed by a timer, not by a fixed cap.
src/core.jsused to drop every instance released past 4 idle per slot, so each round of N > 4 concurrent streams re-instantiated N - 4 engines. Instantiation itself is cheap (about 40 µs), but at 64 interleaved streams that churn cost 10-15% of the time: about 237 instantiations per 256 streams, plus the GC and RSS of the dropped memories (2,857 dead instances awaiting GC at the end of one burst, and peak RSS growth up to +46 MiB against +16 MiB). Keeping instances until a quiet timer tick removes all of it. Against the fixed cap, the time for 256 round-robin streams of 4 KiB chunks falls from 75-78 to 68-70 ms at 8 streams and from 83-85 to 69-72 ms at 64 streams (bench/profile/concurrency.mjs). After the burst settles, the pool holds the same external memory as before (3.0 MiB). One-stream throughput, cold load, and first call are unchanged within noise. Cost: about 164 B gzip of wrapper, and nothing in either engine. - Raw-bytes wrapper fast path.
src/core.js'spushBytescopies aUint8Arraychunk straight intoinbuf, skipping the decode-then-encode round trip a string chunk needs, as long as the chunk validates as UTF-8; it falls back to the decode path for the rest of a stream on the first invalid chunk. Costs wrapper size (bundled+minified+ gzip moved from about 1,958 B to about 2,402 B) for roughly 10-30% faster throughput on byte input, string input unaffected.
Tried and rejected #
| Idea | Result |
|---|---|
-O3/-O4, or raising the inline cap to 300 instead of capping at 150 |
More raw and gzip size for no consistent extra throughput over -O2 at cap 150, and (for -O3/-O4/no-cap) the same doubled post-tier-up RSS the cap exists to prevent. |
Dropping the one-caller inline cap entirely (nocap-O2) |
Faster (roughly +12-20% vs +6-12%) for about +216 B gzip, but doubles post-tier-up RSS (about 21 MiB vs about 11 MiB delta after 20,000 calls). Rejected for any long-lived engine instance; -O2 with the cap dominates it. |
| A table-driven (bytecode-interpreted) type recognizer | Raw size dropped (27,825 to 27,396 B) but gzip and brotli both grew, because the straight-line if (!nx()) return false;-shaped code it replaced already compresses better than denser bytecode plus its interpreter loop (about 300 B fixed interpreter cost). |
Moving downlevel.zena's tables from a GC array to linear memory |
Bounded win of about 20 B raw / 131 B gzip at best; a linear-memory version still needs its own growth logic, which would have to join strip.zena's region layout and cost erase.wasm too even though erase.wasm never uses these tables. |
| Single-site 3- and 4-term ` | |
SIMD identifier-tail scan (idEnd) |
Measured flat throughput against the scalar byte-class loop once the lexer replay cache and -O2 were already in place; dropped for a smaller, simpler rt.wat. |
SIMD whitespace-run scan (skip2) |
Slower: real indentation runs are short enough that the call and setup cost outweighs the vectorized loop. |
Forcing mode == 1 to a compile-time false instead of swapping the hook file |
Only removed part of downlevel.zena's code from erase.wasm, because about 19 call sites read state only the real hooks ever set and aren't behind a foldable mode == 1 guard. Superseded by the hook-file swap, which removes the whole module from the import graph. |
One engine instance per mode that parks and swaps each stream's state by handle (exp/shared-instance) |
1-6% faster than the timer-trimmed pool at 8-64 interleaved streams (67-68 ms against 68-72 ms), but +6.6% gzip on erase.wasm and +7.4% on downlevel.wasm. It also holds more memory: about 10 MiB more RSS growth at one stream, and 14.8 MiB of external memory after a 64-stream burst against 3.0 MiB, because the one memory keeps the parked states' high-water. |
Holding instances past the idle minimum through a WeakRef, so GC trims them |
The same speed as the timer, for about 45 B gzip instead of 164 B. But trimming then depends on GC timing. In an async run under allocation pressure, a major GC dropped the spares mid-load and 73 instances were created again in one measurement. An idle process that runs no major GC keeps a burst's memory indefinitely. |
| Pre-instantiating 4 instances per mode at compile | No change at steady state, since the pool fills on the first burst anyway, and the idle memory is paid before any second stream needs it. |
Starting linear memory at 4 pages instead of 1 (each instance grows to 4 on init) |
No measurable change at any concurrency: memory.grow on a fresh instance is a few microseconds. |
--inline-functions-with-loops |
No consistent size or speed change over the flags already used; dropped as not worth the extra flag. |