A native-speed rewrite of the roblox-ts compiler in golang
rotor roadmap.md
49 kB · 349 lines
Markdown

rotor Roadmap #

rotor is a native Go reimplementation of the roblox-ts compiler (rbxtsc), built on typescript-go. The contract: byte-identical Luau output vs roblox-ts 3.0.0 (header-normalized), the same @rbxts/* ecosystem, the same CLI — at roughly 10x the speed.

  • Design spec: docs/superpowers/specs/2026-06-05-rotor-design.md
  • Detailed plans: docs/superpowers/plans/
  • Porting source of truth: research digests in docs/superpowers/research/ (authority order: reference/ > digest > plan > tests)
  • Acceptance test: the real randomness game project compiles byte-identical to rbxtsc 3.0.0

Legend: ✅ done · 🚧 in progress · ⬜ not started

Status at a glance #

Phase Scope Status
0 Foundation — repo, tsgo mirror, checker spike ✅
1 Luau AST + renderer (port of @roblox-ts/luau-ast 2.0.0) ✅
2 Transformer core + differential harness vs real rbxtsc ✅
2b Functions, destructuring, for-of (arrays), switch, loop closures ✅
3a Imports, module resolution, Rojo resolver, new + constructor macros ✅
3b Macro tables, optional chaining, full iteration, pnpm/baseUrl resolution ✅
3c JSX, classes, decorators, spread, async, try, enums, namespaces ✅
4 Project layer — emit layout, watch, incremental, full CLI, plugin sidecar ✅
5 Conformance — upstream behavioral suite, diagnostics corpus, acceptance ✅
v1.0 — drop-in rbxtsc replacement ✅

Measured progress: 43/43 differential fixtures and 44/44 conformance goldens are byte-identical to real rbxtsc 3.0.0; the vendored behavioral suite passes under Lune (460 passed, 0 failed, 0 skipped on June 7, 2026); the full vendored diagnostics corpus passes without skips; and the real randomness project compares byte-for-byte across out/ + include/ (95/95 files, zero divergent, zero blocked).


Phase 0 — Foundation ✅ #

Plan: docs/superpowers/plans/2026-06-06-rotor-phase0-phase1.md. De-risks the core bet: driving tsgo's real TypeChecker from Go.

Phase 1 — Luau AST + Renderer ✅ #

Plan: same file as Phase 0. Pure code, no checker; fully unit-tested; byte-exact formatting.

Phase 2 — Transformer Core + Differential Harness ✅ #

Plan: docs/superpowers/plans/2026-06-06-rotor-phase2.md. First byte-identical Luau; the harness (real rbxtsc goldens via tools/oracle/oracle.ps1, manifest-gated diff tests) becomes the backbone for every later phase.

Phase 2b — Functions, Destructuring, for-of, switch ✅ #

Plan: docs/superpowers/plans/2026-06-07-rotor-phase2b.md.

Phase 3a — Imports, Module Resolution, NewExpression ✅ #

Plan: docs/superpowers/plans/2026-06-07-rotor-phase3a.md. Unblocked the 96% of real-world files that were import-blocked.

Phase 3b — Resolution Gaps, Macro Tables, Optional Chaining, Iteration ✅ #

Plan: docs/superpowers/plans/2026-06-08-rotor-phase3b.md. Digest: phase3b-digest.md.

Phase 3c — JSX, Classes, Spread, Async, Try, Enums ✅ #

Plan: docs/superpowers/plans/2026-06-07-rotor-phase3c.md. Digests: phase3c-jsx-digest.md, phase3c-classes-digest.md, phase3c-async-try-enums-digest.md. Breaks the JSX wall (33 randomness files) and completes the language surface.

Phase 4 — Project Layer ✅ #

Plan: docs/superpowers/plans/2026-06-07-rotor-phase4.md. Digest: phase4-project-digest.md. Everything that makes rotor a usable CLI tool rather than a compile library.

    • cachedvfs host FS — wraps the compiler host (build + rotor check) so module resolution's repeated Stat/FileExists/Realpath/DirectoryExists probes are memoized for the pass (was ~40% of warm time in GetFileAttributesEx syscalls; every checker group otherwise re-stats the same node_modules/@rbxts paths). Safe: a build never mutates its source tree mid-pass. Same wrapper tsgo's own project host uses.
    • Rojo searchDirectory — uses os.ReadDir entry kinds instead of an os.Stat per child ×2; only reparse points (junctions/symlinks) fall back to a following stat, so pnpm installs stay correct.

Phase 5 — Conformance ✅ #

The 1:1 proof at full scale.

v1.0 — Drop-in replacement ✅ #

Verified locally on June 7, 2026 with:

  • go test ./... -count=1
  • go test ./internal/conformance -count=1 with ROTOR_ROJO_PATH, ROTOR_LUNE_PATH, and ROTOR_RANDOMNESS_PATH
  • bun test in tools/sidecar (after bun install --no-save; npm test remains available as a fallback)
  • go test ./internal/compile -run TestTransformersFixture -count=1 — real-package transformer coverage (Flamework + rbxts-transform-env; requires bun install --no-save in testdata/transformers/project)

Continuously verified in CI (June 8, 2026): .github/workflows/ci.yml runs gofmt + go vet + go build ./... + full go test ./... on every push to master and PR via the shared .github/actions/setup composite action — which provisions Go, Bun (the canonical fixture installer; npm fallback was the source of flaky installs) and Node (for the transformer sidecar). CI also runs the sidecar JS suite (node --test in tools/sidecar) and installs testdata/transformers/project so the real-package transformer test runs. CI and release.yml set ROTOR_REQUIRE_TRANSFORMERS_FIXTURE=1 so a missing fixture install fails the run instead of silently skipping the real-package test. rojo/lune are intentionally not installed in CI, so the lune-executed runtime suite skips there; it runs locally via aftman install and is exercised by the byte-parity differential/diagnostics tests regardless. release.yml reuses the same setup, runs the tests, then publishes CLI binaries via GoReleaser.

Linux validation (June 10, 2026, golang:1.26-bookworm container + Node 22 + npm installs from a clean clone): sidecar JS suite, go vet, full go test ./..., sidecar/transformer tests under -race, and a release-style (CGO_ENABLED=0 -trimpath -ldflags "-s -w -X main.version=...") rotor build of the transformers fixture — including embedded-sidecar extraction to ~/.cache/rotor — all green. All six GoReleaser targets cross-compile; goreleaser check validates the config. Note: releases up to v1.0.2 predate the embedded sidecar — their transformer-plugin support only worked from a repo checkout; the first release after June 10, 2026 carries the fix.

Success criteria (from the design spec):

  1. Byte-identical output (header-normalized) vs rbxtsc 3.0.0 on the upstream corpus and randomness
  2. The vendored behavioral suite passes under Lune and the vendored diagnostics corpus matches upstream expectations
  3. rotor build and rotor build -w work as drop-in rbxtsc replacements with the same npm packages
  4. Measured wins: ≥5x cold build, near-instant watch rebuilds

Out of scope for v1: Playground/VirtualProject, --writeTransformedFiles, devlink

Post-v1 Follow-up #

Watch v2 + QOL + hardening (June 11, 2026; spec: docs/superpowers/specs/2026-06-11-rotor-watch-v2-qol-design.md):

v1.2.1 — transformer/render/resolution fixes (June 12, 2026):

Remaining:

  • Keep one warm Node sidecar session across build -w rebuilds instead of respawning per polling cycle — landed June 10, 2026 (transformer-plugin hardening; see Phase 4 sidecar entry)
  • Retire the package-level TransformStatement func var now that the transform surface is feature-complete
  • Fold the remaining root-only extends-chain validation/sanitizer limitation into the project-options pipeline
  • Warm sidecar .d.ts staleness: stamp-diffed changedFiles cover the .ts/.tsx compile surface; an edited ambient .d.ts is invisible to plugins (not to rotor) until the watch session restarts

Luau toolchain — v2 direction 🚧 #

MVPs of all five sub-projects shipped (2026-06-12): the Luau front-end (lexer + parser + faithful unparse, 405/405 corpus), rotor minify, rotor bundle (Lune-verified to still run), rotor dev, and rotor pack (project → rbxm/rbxmx model or self-reconstructing Luau script, azalea/wax style, Lune-verified to run). Remaining items below each sub-project are depth/coverage follow-ups (readable generator, variable renaming, rojo require-mode + aliases, native pack tree). The TS→Luau compiler is unchanged.

Spec: docs/superpowers/specs/2026-06-12-rotor-luau-toolchain-design.md. Expands rotor from a rbxtsc-parity TS→Luau compiler into an all-in-one Luau toolchain — a require-resolving bundler (still runnable, darklua-style), a minifier, and a rotor dev watch+serve loop. References: darklua + full_moon. The existing byte-parity compiler is untouched; this is additive. Four sub-projects, each with its own plan.

Sub-project A — Luau front-end (lexer + trivia-preserving CST + parser + generators; the critical path)

  • [~] A.3 — Generators — dense (minified) serializer DONE (cst.Dense, in internal/luau/cst/dense.go): block-aware, drops trivia, minimal whitespace (exact boundary re-lex), ; inserted for the Lua call-ambiguity; the faithful retain_lines serializer is cst.Unparse (A.2). GATE: 405/405 corpus files minify to valid Luau with byte-identical significant tokens, 61.9% of original size. readable (pretty) generator still pending. (Generators live in cst, not a separate gen package, since they need the unexported tree visitor.)

Sub-project B — rotor dev (watch cwd + incremental build + supervise rojo serve; independent of A)

Sub-project C — Minifier (rotor minify; depends on A)

Sub-project D — Bundler (rotor bundle; depends on A + internal/rojo)

Sub-project E — rotor pack (project → distributable artifact, azalea/wax style; uses rojo build for the instance tree)

v3 — Cloud toolchain (June 12, 2026) #

Spec: docs/superpowers/specs/2026-06-12-rotor-cloud-toolchain-design.md. Expands rotor into the all-in-one Roblox toolchain: Open Cloud asset sync (asphalt-style), declarative deployment (mantle-style IaC engine), a typed rotor.config.ts evaluated natively (esbuild transpile + goja eval — no Node), and project scaffolding. Built by parallel subagents on shared foundations; all packages httptest/fake-covered, no network in tests.

v1.4.0 — released June 12, 2026 #

v1.5.0 — deploy resource expansion (June 12, 2026) #

v2.0.0 — TOML config, asset macro, more macros (June 13, 2026) #

Spec: docs/superpowers/specs/2026-06-12-rotor-2.0-design.md. Major version; breaking config change.

v2.1 — diagnostics code frames + DX + minifier (June 14, 2026) ✅ #

Spec: docs/superpowers/specs/2026-06-14-unified-code-frame-diagnostics-design.md. Plans 1–3 under docs/superpowers/plans/.

Plan 1 — internal/diagframe renderer + Luau wiring ✅

Plan 2 — TS location plumbing + build/watch frames ✅

Plan 3 — init adopt-existing + doctor synergy ✅

Minifier + toolchain features ✅

Deferred to a future minifier pass ⬜

v2.2 — hosted schema + unified macro types (June 15, 2026) ✅ #

v2.2.1 — pack fixes (June 16, 2026) ✅ #