README.md

lichen-orchestrator #

The build/orchestration tooling for the multi-agent testing + hardening of lichen.page — kept separate from the lichen code repo (lichen.page.wip). This repo is the config, the decision journal, and the infra docs; the actual product code lives in lichen.page.wip.

Contents #

  • agents.toml — the agent-orchestrator config: the agents, the [watchdog], and the [pipeline] of testing phases (deps-upgrade → parallelize → break-it → toolchain → stress → property → mutation → break-it(final) → iroh pass → nix-service → linecov).
  • prompts/ — the per-phase agent prompts.
  • plans/ — the per-phase plans (the binding phase specs the agents work to).
  • JOURNAL.md — a running log of orchestrator decisions + actions.
  • INFRA.md — documentation of the infrastructure the build runs on (host, b1 Incus VMs, Tailscale, Tangled tooling, the pipeline engine).

Recovery #

If it goes down, SETUP.md is the bootstrap/recovery guide — from "an agent crashed" to "rebuild on a fresh host," plus which secrets to restore (they are never committed).

Engine #

The runtime is the agent-orchestrator engine (a git submodule upstream; mirrored at tangled.org/notplants-bot.bsky.social/agent-orchestrator): a tmux-session driver + watchdog that auto-advances the [pipeline], heals crashes, and detects stalls (build-aware).

Comparing compiler stages across two commits #

scripts/compare-compiler-stages.sh <BEFORE-SHA> <AFTER-SHA> [crate ...] compiles the same crate at two commits and diffs five representations rustc builds on its way to machine code, so you can ask whether a change that looks harmless in the source is still harmless once the compiler has finished with it.

# stage the command that prints it what a difference here means
1 Expanded AST -Zunpretty=expanded the code after macros ran is not the same code
2 HIR -Zunpretty=hir names resolve to different things, or desugaring differs
3 THIR -Zunpretty=thir-tree the two versions typecheck differently
4 MIR -Zunpretty=mir the control-flow graph — the behaviour — differs
5 LLVM IR --emit=llvm-ir what is handed to the code generator differs

It complements scripts/verify-test-migration.sh, which proves a test relocation is faithful at the level of source tokens. These go under that.

No nightly toolchain is installed. Four of the five are -Z flags, normally nightly-only; the script sets RUSTC_BOOTSTRAP=1 so the pinned stable rustc accepts them. That is deliberate — it means the result describes the compiler that actually builds this project, not some other one. --emit=llvm-ir is stable anyway.

The command for the R-11 test migration #

R-11 moved every inline #[cfg(test)] mod tests { … } into a #[path]-attached file. bc3070d9 is the commit before, 32934f4a is the move with no reformatting.

The script needs cargo, git, and scratch space measured in gigabytes — the THIR dump alone is ~180 MB per tree on the smallest crate here, and it builds three trees. So it must not run on the shared orchestrator box. Send it to lichen-ci. ci-test.sh -f hands a script to the remote with no arguments, so prepend them with set --:

{ echo 'set -- bc3070d9 32934f4a lichen-yrs'
  cat scripts/compare-compiler-stages.sh
} > /tmp/run.sh

CI_TEST_JOBS=4 CI_TEST_MEM_MAX=5G scripts/ci-test.sh 32934f4a -f /tmp/run.sh

Drop the crate name to compare every crate the two commits touch (for this pair that is seven: lichen-server, lichen-cms, lichen-cli, lichen-shell, lichen-flat-file, lichen-iroh, lichen-yrs). Measured cost for lichen-yrs alone, on lichen-ci with a warm cargo registry: 59 s wall and about 1.1 GB of scratch at peak. lichen-server has roughly 11x the source and nothing here has measured it — its THIR dump should be expected in the gigabytes per tree, and there are three trees. Do a multi-crate run on a VM lane rather than lichen-ci, because a targeted CI run holds ~/lpr/.base.lock for its whole duration.

There is a second pair worth running: 32934f4a → 131f0870 is rustfmt and nothing else. Anything that differs there is formatting reaching into a representation it has no business reaching.

{ echo 'set -- 32934f4a 131f0870 lichen-yrs'; cat scripts/compare-compiler-stages.sh; } > /tmp/run.sh
CI_TEST_JOBS=4 CI_TEST_MEM_MAX=5G scripts/ci-test.sh 131f0870 -f /tmp/run.sh

What the two R-11 pairs actually showed (lichen-yrs, on lichen-ci, rustc 1.97.0) #

The move, bc3070d9 -> 32934f4a.

stage raw differing lines after normalisation armed check
EXPANDED AST 123 12 passed (4 lines)
HIR 159 12 passed (4 lines)
THIR 38734 120 passed (533 lines)
MIR 2011 211 passed (8 lines)
LLVM IR 763800 511 passed (20 lines)

All 866 residual lines were classified, not sampled, and every one of them is the recorded source location of test code, in five different encodings:

stage what the residual is lines
EXPANDED AST / HIR tracing_core::metadata::Metadata::new("event …/ws.rs:332", …) 12 + 12, all
THIR Literal( lit: Spanned { node: Str(…) / Int(…) } ) 120, all
MIR interned-constant byte dumps, TestDesc fields, _5 = const "…/ws.rs" 211, all
LLVM IR @alloc_… = private unnamed_addr constant and ___CALLSITE4META 511, all

The 12 lines at the top two stages are the ones worth reading. The tracing macros bake file!() and line!() into a static Metadata at expansion time, so moving a test that logs changes what its own log lines say about themselves — src/ws.rs:332 becomes src/tests/ws.rs:58. No token- or AST-level check can see this, because the tokens debug!("…") did not change.

The reformat, 32934f4a -> 131f0870. rustfmt reindented and reflowed — joining multi-line signatures, so files got shorter and line numbers moved too, not only columns (git diff --ignore-all-space on this pair is not empty: 6 insertions, 17 deletions).

stage result
EXPANDED AST identical
HIR identical
MIR identical — but see below, this one means less than it looks
THIR 124 lines differ, all Literal(Int) nodes
LLVM IR 488 lines differ, all @alloc_… constant definitions

Do not read "MIR: identical" as "formatting does not reach MIR". assert_eq! compiles to a #[track_caller] panic, whose core::panic::Location is not materialised until codegen; MIR's only trace of a caller location is the span, and this script erases every span. So MIR here cannot distinguish a pure reindent from no change at all — "identical" is a fact about the normalisation, not about the emitted code. LLVM IR from the same run says plainly that something did move. That contrast is the most useful thing these two pairs teach: a stage can be clean because it agreed, or clean because you blinded it, and only the arming and the erasure list tell you which.

What the report certifies, and what it does not #

  • Every normalisation is printed under the stage that applied it. These dumps are noisy across checkouts — spans, DefId numbering, interned-constant ids, LLVM's debug metadata table — and a transform aggressive enough to make everything match proves nothing. A reader who cannot see what was erased cannot judge the result.

  • Wrapped values are unwrapped before erasing. rustc's printers break a long value onto the next line, so source_file prints on one line or two depending only on how long the path is. The erasures are line-oriented sed, so the wrapped form would not match — and that fails asymmetrically on exactly this comparison, since relocating a test makes its path longer: BEFORE stays on one line and is erased, AFTER wraps and is not. Measured on lichen-iroh: HIR reported 43 differing lines where the expanded AST, which erases less, reported 5; with the values unwrapped it reports 4. Note the direction — an unhandled wrap invents residual, it cannot hide a change — and note that the arming cannot catch this, because the perturbation produces a difference either way. Arming shows a stage can see a change; it says nothing about whether both sides were treated alike.

  • Each stage is armed. Before reporting a clean comparison the script flips one assert_eq! to assert_ne! in a relocated test, re-dumps, and requires the stage to see it. A stage that stays green under that edit is reported UNUSABLE, not as a pass — an empty diff from a stage that never ran looks exactly like an empty diff from a stage that ran and matched. An empty dump is likewise reported as broken.

  • Default features only. Test code behind a non-default #[cfg(feature = …)] is not compared at all. That gap is large: on lichen-server the default feature set lists 114 tests where the full set lists 312.

  • The lib target only, compiled as cargo test compiles it (--profile test --lib). Binaries and tests/ integration targets are not compared.

Trigger Lichen CI #

The CI push helper lives here; pass the product checkout explicitly:

scripts/ci-run.sh --repo /path/to/lichen.page HEAD smoke
scripts/ci-run.sh --repo /path/to/lichen.page --prune 14

It resolves commits and uses the review remote in that checkout (override the remote with CI_RUN_REMOTE). Workflow validation and the existing in-flight check still run before a test branch is pushed. Pruning deletes old test-* branches on that same selected remote.

Run the isolated relocation tests with python3 -m unittest discover -s tests -p test_ci_run.py.