Zena compiler issues #

Zena compiler bugs and limitations this project hit, kept here until each is filed upstream at elematic/zena. The early entries came from a port of sucrase's tokenizer that this repo no longer has, so their examples are standalone. Some entries cite notes/size-log.tsv and other notes that have since been removed. git log -- notes/ has them.

Assigning to a field of a namespace-import's member crashes the compiler #

import * as s from './mod.zena' gives a read-only record s. Reading s.something is fine, and so is reading-then-writing a primitive exported var through it is fine for reads, but writing to a field of an object reached through that record crashes the compiler (-O 0 and -O s both):

// c.zena
export class State { var pos: i32 = 0; }
export let state: State = new State();

// main.zena
import * as s from './c.zena';
export function bump(): void {
  s.state.pos = s.state.pos + 1;   // crash
}

Exception caught in CLI compiler!
Error message: zir emit: ref.null type

The same assignment through a named import of the same binding compiles fine:

import { state } from './c.zena';
export function bump(): void { state.pos = state.pos + 1; } // fine

An earlier, broader theory (assigning to any cross-module var, even a bare i32, crashes -- see the original repro below) turned out to be a special case of this: a bare exported var has no fields to assign through a namespace record, so the record itself gets reassigned, which is exactly what "read-only" forbids, and the compiler panics there instead of diagnosing it.

// a.zena
export var pos: i32 = 0;

// main.zena
import * as a from './a.zena';
export function bump(): void { a.pos = a.pos + 1; }   // crash, same error

Workaround: use named imports (import { state } from './x.zena') for any binding whose fields -- or itself, if a bare var -- get assigned from outside its own module, instead of import * as x. Applied throughout zena/: every module that needs to mutate tokenizer/state.zena's shared State singleton imports state (and the helper functions) by name rather than as a namespace.

Ternary ?: is not valid expression syntax #

Not a bug, just a syntax gap worth noting for anyone porting more JS/TS: Zena has no C-style ternary. cond ? a : b is a parse error; use if (cond) a else b as an expression. Hit repeatedly porting tokenizer/index.ts's inline ? : expressions (e.g. code === charCodes.verticalBar ? tt.logicalOR : tt.logicalAND).

No bitwise-NOT (~) unary operator #

Only ! and unary - are listed as unary operators; ~x is a parse error. Used -8 in place of ~7 for the alignment mask in state.zena's align8 (two's-complement ~7 == -8, and masking with it has the same effect).

A bare return expr; as a match case's only statement needs braces #

case _: return (-1, -1); (no braces) is a parse error inside a match used as a statement; wrapping it as case _: { return (-1, -1); } compiles. Every other case in the same match already used a { ... } block, so this may just be a block-vs-bare-expression case-body ambiguity rather than a bug.

Namespace-importing a module that exports a class crashes the compiler #

Found porting the traverser. Even a pure read through the namespace crashes, as long as the imported module exports a class:

// r1.zena
export let A: i32 = 5;
export class K {
  var pos: i32 = 0;
}

// r2.zena
import * as r from './r1.zena';
export function main(x: i32): i32 {
  return x + r.A;   // crash
}

Exception caught in CLI compiler!
Error message: zir emit: ref.null type

Dropping export from class K, or replacing the class with an exported function, compiles and runs fine (main(1) == 6). This is likely the same root cause as the entry above: the namespace record gets a field for the exported class (or the instance-typed let state) and emitting its ref.null initializer fails.

Workaround: namespace imports (import * as tt) only for modules that export nothing but constants and functions (types.zena, keywords.zena, charcodes.zena); everything from tokenizer/state.zena is imported by name.

A namespace import of a function module costs a trampoline per export #

Found wiring the phase 2 emitter. import * as erase from '../erase.zena' (a module that exports only functions) and calling erase.statementBegin() compiles and runs correctly, but every exported function of erase.zena gets a $name$trampoline wrapper plus globals for the namespace record, whether or not it is called through the record. With about 60 exports that was 24KB of strip.wasm (148,219 bytes against 124,156 with named imports).

Workaround: named imports, aliased so the call sites still read as hooks: import { statementBegin as eraseStatementBegin } from '../erase.zena'.

A namespace import of a constants-only module also pays per import site, not just per export #

The entry above's workaround note said namespace imports were still fine "for modules that export nothing but constants and functions (types.zena, keywords.zena, charcodes.zena)". That turned out to be wrong: a constants-only module gets the same treatment as a function module, just with an extra layer. Each exported let becomes a zero-arg getter function ((structref) -> i32, reading one field of a backing data struct), the module gets a vtable struct of those getters, and every file that writes import * as k from './consts.zena' gets its own fresh copy of the backing data struct and the {structref, vtable} existential pair that wraps it -- not a shared global reused across importers. The cost scales with (number of importing files) x (number of exports), not just the export count.

Minimal repro, consts.zena exporting 5 lets, imported via namespace by two tiny files vs. the same two files importing by name:

// consts.zena
export let a: i32 = 1;
export let b: i32 = 2;
export let c: i32 = 3;
export let d: i32 = 4;
export let e: i32 = 5;

// ns1.zena / ns2.zena (namespace form)
import * as k from './consts.zena';
export function f1(): i32 { return k.a + k.b; }

// named1.zena / named2.zena (named form)
import { a, b } from './consts.zena';
export function f1(): i32 { return a + b; }

zena -O s build on a main.zena that imports f1/f2 from each pair: 560 bytes (namespace) vs. 75 bytes (named). wasm-tools print on the namespace build shows why: one getter function per export (struct.get 4 N inside a (structref) -> i32 function), a vtable struct of those getters (type 1), a data struct holding the real values (type 4), and an existential pair wrapping them (type 2, {structref, (ref 1)}). ns1.zena and ns2.zena each get their own struct.new 4

• struct.new 2 in the global initializers -- the data struct is rebuilt per importer instead of being one shared global.

Found while shrinking strip.wasm: zena/tokenizer/types.zena (113 TokenType constants) and keywords.zena (41 ContextualKeyword constants) are each namespace-imported (import * as tt / import * as ck) by around ten files across erase.zena, plugins/, traverser/ and tokenizer/. Converting just one small file (plugins/types.zena, 4 constants used) from import * as tt to named imports dropped strip.wasm by 1,432 bytes on its own; converting every remaining namespace import of types.zena/keywords.zena/ charcodes.zena is the largest single lever found in this pass (see notes/size-log.tsv).

Workaround: named imports for every module, including constants-only ones. There is no case where import * as x is cheaper than import { the names actually used } from './x.zena' once more than one file imports it.

A bare case label is always a type pattern, even for an imported or local let #

Hit converting the namespace imports above to named imports. zena/tokenizer/index.zena's getTokenFromCode switched on a charcode with match (code) { case cc.numberSign: ...; case cc.dot: ...; ... }. Once cc.numberSign became the named import numberSign, the same case numberSign: failed to compile:

'numberSign' names a type, but only classes can be matched by name.
Write 'let numberSign' to bind the matched value to a new variable.

Minimized, with no imports at all -- a bare top-level let fails the same way as its own case label:

let foo: i32 = 5;
export function test(x: i32): i32 {
  match (x) {
    case foo: { return 1; }   // error: 'foo' names a type
    case _: { return 0; }
  }
}

So the rule isn't about imports: a bare identifier in case position is always resolved as a type pattern (matching an instance of a class named that), never as a value-equality check against a constant, no matter where the constant is declared. case (foo): and case foo + 0: were tried as workarounds; the first still errors the same way (parens don't change the pattern grammar) and the second is a parse error (case takes a pattern, not an arbitrary expression). Only a qualified name (cc.numberSign, a member-access expression) is unambiguously a value in that position, which is presumably why every pre-existing match in this codebase already switched on namespace-qualified names rather than local constants -- this limitation was apparently known and designed around from the start, just never written down.

Workaround: don't put a bare constant in a case label. Either keep it qualified behind a namespace import (defeats the point if the whole reason to drop the namespace is size), or, as done in getTokenFromCode, rewrite the match as an if (code == x) { ...; return; } chain. The latter is what this pass did, since it was the one spot this surfaced and losing the namespace import there was worth ~52KB of strip.wasm (see notes/size-log.tsv).

zena -O s leaves most of its own size on the table; wasm-opt -Oz finds it #

Measured on engine/main.zena's strip-only build (build/lite.zena.wasm before wasm-opt, 44,459 bytes; build/lite.wasm after, 29,204 bytes at first with plain -Oz --converge, before the --closed-world --gufa-optimizing and RSS-cap flags in notes/size-log.tsv's later rows). wasm-opt -Oz --converge alone removes 15,255 bytes (34.3%) that zena -O s does not. Isolating each transformation by running it alone against the pre-wasm-opt binary (so these numbers don't compound the way the real -Oz --converge pipeline's iteration does, but do rank which single transformation zena's own -O s is most missing), ranked by bytes:

1. Single-caller inlining, --inlining-optimizing alone: 8,300 bytes (18.7%). The largest single lever by far. zena -O s compiles every function as written and never inlines across a function boundary, even when a function has exactly one call site -- which strip.zena's per-state handlers (hStmt, hExpr, hBinary, ...) mostly are, called from the one step() dispatcher. Inlining collapses wasm-opt --metrics' function count from 143 to 80, removing call/return overhead and per-function prologue duplication, and opens up constant propagation across what used to be a call boundary.
2. Branch and block folding, --merge-blocks --remove-unused-brs --remove-unused-names --code-folding together: 3,640 bytes (8.2%). zena's match on integers lowers to a compare chain (see the "integer match compiling to compare chains" entry above), and its lack of a ternary means every inline conditional is its own if/else block (see the "no ternary" entry). Both produce more block and branch nesting than the logic needs; these passes fold shared branch targets and remove blocks and labels nothing jumps to.
3. Local variable reduction, --simplify-locals --coalesce-locals --reorder-locals together: 2,984 bytes (6.7%). wasm-opt --metrics shows [vars] (locals beyond parameters) dropping from 1,119 to 173 over the full pipeline. zena -O s gives each let/var binding and each lookahead's saved position its own local slot and never reuses one once its last read is behind it; these passes merge locals whose live ranges don't overlap and renumber what's left for a smaller index encoding.
4. Peephole simplification, --optimize-instructions alone: 2,690 bytes (6.1%). Folds comparison and arithmetic idioms zena emits literally, including the if (cond) a else b-as-expression shape every inline conditional takes (no ternary) and the two's-complement -8 for ~7 idiom (no bitwise-NOT) from the entries above.
5. Dead code and module-element pruning, --dce / --remove-unused-module-elements / --vacuum, each alone: 1,081 bytes (2.4%). zena -O s does not prune functions, globals or exports left unreferenced after whole-program analysis (as opposed to syntactically dead code in one function, which it does appear to skip already, since plain --vacuum alone finds the same 1,081 bytes as --dce).

Two further transformations only pay off with --closed-world, which wasm-opt cannot infer on its own and zena has no equivalent flag for: --gufa-optimizing's whole-program value and type inference finds a few more constant folds once it can see every call site to every internal function (worth roughly 60-90 bytes gzip on this module, which is almost entirely linear-memory i32 logic rather than GC objects -- --type-merging/--type-ssa alone changed nothing, since there's barely any GC type use for them to act on). zena itself could plausibly ship the --closed-world inference internally for any module whose only exports are scalar functions and memory, without waiting for a downstream wasm-opt pass, since the compiler already knows its own export list.

One more finding is a runtime cost, not a size one, worth flagging separately: -Oz's default single-caller inlining has no cap, so it keeps collapsing strip.zena's mostly-linear call chain into one function until, on this module, that function is about 14KB of wasm bytecode -- too big for V8's Liftoff baseline tier, so V8 promotes it to TurboFan on first call. That optimization is the majority of the RSS a short-lived process pays to strip one small file: 7.9MiB over bare node, against 1.7-1.8MiB with --one-caller-inline-max-function-size=150 capping the inlining (see notes/size-log.tsv's lite:one-caller-inline-cap-rss row). zena's own largest function at -O s (no inlining at all) is about 3.2KB, so this is entirely a wasm-opt-introduced cost, not one inherent to the source.

Size findings from the architectural size pass (notes/size-architecture.md) #

None of these is a correctness bug. Each one cost measurable bytes in build/lite.wasm.

• No data segments. User code has no way to put bytes in a data segment. With --target js, string literals become wasm:js-string imports. FixedArray<T> literals become array.new_fixed of constants in the start function, which costs about 1.3x to 1.4x the bytes for i64-packed text and compresses badly. array.new_data and array.init_data exist as intrinsics but are not reachable for a user-defined segment. The workaround is generated WAT modules that wasm-merge links in (scripts/build.sh).
• No memory.copy, memory.fill or i32.load16_u intrinsics. Memory.getU16 is two getU8 calls, and a memmove is a hand-written loop. The workaround is engine/rt.wat, an imported function that is merged in and inlined.
• Hoisting Memory.default adds a null check. let mem = Memory.default; emits (drop (ref.as_non_null (global.get $Memory))) in that function, which wasm-opt cannot remove, and it keeps the otherwise unused Memory struct global and its start-function struct.new alive. A direct Memory.default.getX(p) lowers to the bare load. The hoist-memory-default-* rows in size-log.tsv, measured on the older engine, no longer hold.
• The inliner copies small straight-line bodies at every site, at -O s and under -g. codegen/ir/inline.zena inlines any straight-line callee of 12 or fewer instructions (straightBudget) whatever the call count, and -g's help text says it turns inlining off. types.zena's nx() was copied into about 100 sites, about 500 B. There is no @noinline; the workaround is to return through a branch.
• if chains and match on integers never become br_table. Neither binary has one. step's 35-state dispatch is a linear compare chain.
• Constant module lets with a non-literal initializer are not folded. let M: i64 = (1 as i64) << (T_X as i64) | ... becomes a mutable global set in the start function. A tiny pure helper such as bit(t) = 1 << t is neither inlined nor folded at constant call sites. The workaround is generated hex literals (scripts/gen/keywords.mjs).
• Duplicate function types in the rec group. lite.wasm has two (func) types ($5, $10) in one rec group, and closed-world type merging leaves both.

Findings from the second size pass #

• / on two i32s is a float division. n / 10 with n: i32 has type f32 (a return of it from an i32 function is a type error), and (n / 10) as i32 lowers to f32.convert_i32_s, f32.div and i32.trunc_sat_f32_s, which is inexact above 2^24. % does lower to i32.rem_s. This pass found no integer division operator. engine/downlevel.zena had a repeated-subtraction divBy; txDecimal now divides by ten with the exact reciprocal multiply in i64, n * 0xCCCCCCCD >> 35.