- HTML 47%
- Python 30%
- Shell 20%
- Rust 3%
- Go <1%
- 2 more languages <1%
lichen-orchestrator #
The build/orchestration tooling for the multi-agent testing + hardening of
lichen.page — kept separate from the lichen
code repo (lichen.page.wip). This repo is the config, the decision journal, and the infra docs; the
actual product code lives in lichen.page.wip.
Contents #
agents.toml— the agent-orchestrator config: the agents, the[watchdog], and the[pipeline]of testing phases (deps-upgrade → parallelize → break-it → toolchain → stress → property → mutation → break-it(final) → iroh pass → nix-service → linecov).prompts/— the per-phase agent prompts.plans/— the per-phase plans (the binding phase specs the agents work to).JOURNAL.md— a running log of orchestrator decisions + actions.INFRA.md— documentation of the infrastructure the build runs on (host, b1 Incus VMs, Tailscale, Tangled tooling, the pipeline engine).
Recovery #
If it goes down, SETUP.md is the bootstrap/recovery guide — from "an agent crashed"
to "rebuild on a fresh host," plus which secrets to restore (they are never committed).
Engine #
The runtime is the agent-orchestrator engine (a git submodule upstream; mirrored at
tangled.org/notplants-bot.bsky.social/agent-orchestrator): a tmux-session driver + watchdog that
auto-advances the [pipeline], heals crashes, and detects stalls (build-aware).
Comparing compiler stages across two commits #
scripts/compare-compiler-stages.sh <BEFORE-SHA> <AFTER-SHA> [crate ...] compiles the
same crate at two commits and diffs five representations rustc builds on its way to
machine code, so you can ask whether a change that looks harmless in the source is
still harmless once the compiler has finished with it.
| # | stage | the command that prints it | what a difference here means |
|---|---|---|---|
| 1 | Expanded AST | -Zunpretty=expanded |
the code after macros ran is not the same code |
| 2 | HIR | -Zunpretty=hir |
names resolve to different things, or desugaring differs |
| 3 | THIR | -Zunpretty=thir-tree |
the two versions typecheck differently |
| 4 | MIR | -Zunpretty=mir |
the control-flow graph — the behaviour — differs |
| 5 | LLVM IR | --emit=llvm-ir |
what is handed to the code generator differs |
It complements scripts/verify-test-migration.sh, which proves a test relocation is
faithful at the level of source tokens. These go under that.
No nightly toolchain is installed. Four of the five are -Z flags, normally
nightly-only; the script sets RUSTC_BOOTSTRAP=1 so the pinned stable rustc accepts
them. That is deliberate — it means the result describes the compiler that actually
builds this project, not some other one. --emit=llvm-ir is stable anyway.
The command for the R-11 test migration #
R-11 moved every inline #[cfg(test)] mod tests { … } into a #[path]-attached file.
bc3070d9 is the commit before, 32934f4a is the move with no reformatting.
The script needs cargo, git, and scratch space measured in gigabytes — the THIR dump
alone is ~180 MB per tree on the smallest crate here, and it builds three trees. So it
must not run on the shared orchestrator box. Send it to lichen-ci. ci-test.sh -f hands
a script to the remote with no arguments, so prepend them with set --:
{ echo 'set -- bc3070d9 32934f4a lichen-yrs'
cat scripts/compare-compiler-stages.sh
} > /tmp/run.sh
CI_TEST_JOBS=4 CI_TEST_MEM_MAX=5G scripts/ci-test.sh 32934f4a -f /tmp/run.sh
Drop the crate name to compare every crate the two commits touch (for this pair that
is seven: lichen-server, lichen-cms, lichen-cli, lichen-shell, lichen-flat-file,
lichen-iroh, lichen-yrs). Measured cost for lichen-yrs alone, on lichen-ci with a warm
cargo registry: 59 s wall and about 1.1 GB of scratch at peak. lichen-server has
roughly 11x the source and nothing here has measured it — its THIR dump should be expected
in the gigabytes per tree, and there are three trees. Do a multi-crate run on a VM lane
rather than lichen-ci, because a targeted CI run holds ~/lpr/.base.lock for its whole
duration.
There is a second pair worth running: 32934f4a → 131f0870 is rustfmt and nothing else.
Anything that differs there is formatting reaching into a representation it has no
business reaching.
{ echo 'set -- 32934f4a 131f0870 lichen-yrs'; cat scripts/compare-compiler-stages.sh; } > /tmp/run.sh
CI_TEST_JOBS=4 CI_TEST_MEM_MAX=5G scripts/ci-test.sh 131f0870 -f /tmp/run.sh
What the two R-11 pairs actually showed (lichen-yrs, on lichen-ci, rustc 1.97.0) #
The move, bc3070d9 -> 32934f4a.
| stage | raw differing lines | after normalisation | armed check |
|---|---|---|---|
| EXPANDED AST | 123 | 12 | passed (4 lines) |
| HIR | 159 | 12 | passed (4 lines) |
| THIR | 38734 | 120 | passed (533 lines) |
| MIR | 2011 | 211 | passed (8 lines) |
| LLVM IR | 763800 | 511 | passed (20 lines) |
All 866 residual lines were classified, not sampled, and every one of them is the recorded source location of test code, in five different encodings:
| stage | what the residual is | lines |
|---|---|---|
| EXPANDED AST / HIR | tracing_core::metadata::Metadata::new("event …/ws.rs:332", …) |
12 + 12, all |
| THIR | Literal( lit: Spanned { node: Str(…) / Int(…) } ) |
120, all |
| MIR | interned-constant byte dumps, TestDesc fields, _5 = const "…/ws.rs" |
211, all |
| LLVM IR | @alloc_… = private unnamed_addr constant and ___CALLSITE4META |
511, all |
The 12 lines at the top two stages are the ones worth reading. The tracing macros bake
file!() and line!() into a static Metadata at expansion time, so moving a test that
logs changes what its own log lines say about themselves — src/ws.rs:332 becomes
src/tests/ws.rs:58. No token- or AST-level check can see this, because the tokens
debug!("…") did not change.
The reformat, 32934f4a -> 131f0870. rustfmt reindented and reflowed — joining
multi-line signatures, so files got shorter and line numbers moved too, not only columns
(git diff --ignore-all-space on this pair is not empty: 6 insertions, 17 deletions).
| stage | result |
|---|---|
| EXPANDED AST | identical |
| HIR | identical |
| MIR | identical — but see below, this one means less than it looks |
| THIR | 124 lines differ, all Literal(Int) nodes |
| LLVM IR | 488 lines differ, all @alloc_… constant definitions |
Do not read "MIR: identical" as "formatting does not reach MIR". assert_eq! compiles
to a #[track_caller] panic, whose core::panic::Location is not materialised until
codegen; MIR's only trace of a caller location is the span, and this script erases every
span. So MIR here cannot distinguish a pure reindent from no change at all — "identical" is
a fact about the normalisation, not about the emitted code. LLVM IR from the same run says
plainly that something did move. That contrast is the most useful thing these two pairs
teach: a stage can be clean because it agreed, or clean because you blinded it, and only
the arming and the erasure list tell you which.
What the report certifies, and what it does not #
-
Every normalisation is printed under the stage that applied it. These dumps are noisy across checkouts — spans,
DefIdnumbering, interned-constant ids, LLVM's debug metadata table — and a transform aggressive enough to make everything match proves nothing. A reader who cannot see what was erased cannot judge the result. -
Wrapped values are unwrapped before erasing. rustc's printers break a long value onto the next line, so
source_fileprints on one line or two depending only on how long the path is. The erasures are line-orientedsed, so the wrapped form would not match — and that fails asymmetrically on exactly this comparison, since relocating a test makes its path longer: BEFORE stays on one line and is erased, AFTER wraps and is not. Measured onlichen-iroh: HIR reported 43 differing lines where the expanded AST, which erases less, reported 5; with the values unwrapped it reports 4. Note the direction — an unhandled wrap invents residual, it cannot hide a change — and note that the arming cannot catch this, because the perturbation produces a difference either way. Arming shows a stage can see a change; it says nothing about whether both sides were treated alike. -
Each stage is armed. Before reporting a clean comparison the script flips one
assert_eq!toassert_ne!in a relocated test, re-dumps, and requires the stage to see it. A stage that stays green under that edit is reported UNUSABLE, not as a pass — an empty diff from a stage that never ran looks exactly like an empty diff from a stage that ran and matched. An empty dump is likewise reported as broken. -
Default features only. Test code behind a non-default
#[cfg(feature = …)]is not compared at all. That gap is large: onlichen-serverthe default feature set lists 114 tests where the full set lists 312. -
The lib target only, compiled as
cargo testcompiles it (--profile test --lib). Binaries andtests/integration targets are not compared.
Trigger Lichen CI #
The CI push helper lives here; pass the product checkout explicitly:
scripts/ci-run.sh --repo /path/to/lichen.page HEAD smoke
scripts/ci-run.sh --repo /path/to/lichen.page --prune 14
It resolves commits and uses the review remote in that checkout (override the
remote with CI_RUN_REMOTE). Workflow validation and the existing in-flight
check still run before a test branch is pushed. Pruning deletes old test-*
branches on that same selected remote.
Run the isolated relocation tests with python3 -m unittest discover -s tests -p test_ci_run.py.