small gleam coding and (not yet) persistent agent daemon with a detachable cli forked from okami.mom/albedo
albedo robot-docs testing.md
11 kB · 200 lines
Markdown
at main

Test execution #

test.sh is the pre-commit gate. With no arguments it selects Python end-to-end scenarios affected by the change and runs Go end-to-end tests for CLI changes. ./test.sh --full runs all end-to-end scenarios and the web check. Manual tests under test/manual are opt-in.

Gate and resource limits #

test/quick.sh runs formatting, type, lint, generated-code, and Go vet checks before the suites. It also builds each workspace package with warnings as errors, and the gate runs each package's own tests as a suite. The gate builds a daemon snapshot and launches suites concurrently. Its end-to-end selection and web checks depend on the changed paths or --full. See test.sh for that selection and Gleam lint for lint profiles.

The snapcompact tests in gleam test render frames through albedo-render. The daemon looks for it in priv/bin, then on PATH. The gate installs it with native/render/install.sh before its suites. When gleam test runs alone and finds no renderer, its entry point, test/albedo_test.gleam, runs that script first. The script needs cargo from nix develop. Run it again after changing native/render, because gleam test does not rebuild a renderer that is already installed.

test/resources.py calculates concurrency limits from CPU and available memory. It reserves one quarter of memory and gives half of the rest to end-to-end lanes. A lane is one test on the shared daemon, two exclusive daemons, and one Go scenario. E2E_LANE_MEMORY is what one lane adds to the gate's peak, and the plan rejects a host where one lane does not fit. ALBEDO_TEST_*_JOBS variables can override the calculated limits. These limits control concurrency, not process memory allocations.

Measure the gate before changing a cost in the plan:

  1. Set every ALBEDO_TEST_*_JOBS variable to the plan under test, such as CI's test plan: line with a different lane count.
  2. Put TMPDIR and ALBEDO_TEST_TMP under /dev/shm, as CI does.
  3. Run each suite of ./test.sh --full in its own cgroup, for example through systemd-run --user --scope, and sample each cgroup's memory.current.
  4. Compare the gate's peak at two or more lane counts.

A cgroup counts the detached daemons and the tmpfs scratch that a sum of the suite's process tree misses. Gleam lint runs in test/quick.sh before the suites start, so its peak does not add to theirs. See Gleam lint memory.

The suites start together, so the gate's peak memory is what they all hold at that moment. CI's workflow puts TMPDIR and ALBEDO_TEST_TMP under /dev/shm, so scratch files use the same memory. Do not let a test start background work that it does not check. For example, dev-proxy.test.mjs loads only the server options from vite.config.ts. With the config's plugins, Vite pre-bundles the client's dependencies when its server starts, and CI's fresh node_modules has no cache for that work.

Expected reports in gleam test #

Some tests provoke failure paths on purpose, and the daemon code they drive logs those failures. A passing run prints these lines, each from the named test:

  • listener was killed, listener did not start: port taken, and giving the listener up come from a_bounded_listener_stops_after_its_failed_restarts_test in listener_test.
  • kernel_cleanup_failed with plugin cleanup did not finish within 1.5 s comes from hard_timeout_requires_explicit_reset_test in runtime_test. Its cell holds the kernel's event loop, so plugin cleanup cannot start.
  • owner_lost with kernel ownership cleanup was not confirmed comes from tests that end a kernel's owner: owner_loss_* in owner_supervision_test and runtime_killed_with_a_queued_reload_result_retires_the_kernel_test in runtime_ownership_test. Only the owner can forget the kernel's record. The last test also prints reload for a stopped runtime: with either that reason or kernel lost before its processes could be supervised, depending on whether the driver handles the stop or the owner's death first.
  • spare warmup outcome is unknown comes from an_unknown_spare_allocation_retains_budget_without_blocking_stop_test, and kernel cleanup late-cleanup: unconfirmed from late_cleanup_success_keeps_another_generations_failure_and_retry_test, both in runtime_ownership_test.

This list covers the listener and kernel lifecycle reports. When one of them comes from a different test, or a new one appears, find the test that printed it before treating it as expected.

Fixture teardown #

test/e2e/harness.py owns test daemons and fixtures. peer_daemon(name, prepare=None) starts a separate daemon for a test. finish_test cleans up peers and registered resources. Use the shared harness for daemon scenarios. See storage migrations for upgrade coverage.

A test daemon stops when the suite that started it exits, through ALBEDO_PARENT_PID. A daemon that a scenario installs as a service drops that variable, so the scenario's own cleanup must stop it. When the gate ends, test/reap-daemons.sh waits for detached kernels to finish their grace period, then stops any process still running from the gate's snapshot and fails the gate, naming each process's ALBEDO_HOME. Fix the scenario that owns that home. Run the script on the scratch root to clear processes left by an interrupted run. That also stops the daemons of a gate still running in another checkout.

Mixed-version moves #

move_versions_test.py runs this checkout's daemon beside an older reference daemon. Both run as peer daemons. Moves are real ones, over the loopback ssh that move_test.py uses. Each source session has a transcript, kernel cells, a memory note, a work item, a vent, a delivered webhook letter, a schedule, and a workspace link.

  • A move from the reference to this daemon is the supported path. The scenario finishes the checklist, then checks the carried data and runs a cell and a turn on the target.
  • A move from this daemon to the reference must be refused cleanly. The scenario adds a column to sessions first, so the refusal does not depend on how far the pin lags. The reference predates POST /transfers/check, so the preflight must still offer the move. The move must then fail at staging with transfer_target_older and the reference's detail, leaving no receipt or staging directory there. The scenario then checks for no session on the reference, and a source session that keeps its schedule and takes input. move_test.py covers the check against a target of this build.

The rules these scenarios check are in moves between versions.

test/reference-daemon.sh pins the reference commit in REFERENCE. Albedo has no release tags, so the pin is a commit on main. The script extracts it with git archive, builds it with that commit's own test/snapshot-daemon.sh and its CLI, and prints the directory. It fetches the commit first when a shallow clone lacks it. Builds are cached under ALBEDO_REFERENCE_CACHE, which defaults to ~/.cache/albedo-reference, one directory per commit. A cold build took 37 s on a developer machine with warm Hex and Go module caches. A cached run returns at once. CI keeps the directory in its albedo-reference cache, keyed by the script, so ./test.sh --full builds it once per pin.

Bump the pin when a commit that users run lands, such as one deployed to another host, or when the reference falls back past a transfer protocol change.

  1. Set REFERENCE to the full commit hash.
  2. Run nix develop -c python3 test/e2e/run.py move_versions_test.py downgrade_test.py.
  3. Report a failure of the supported move as an incompatibility between the two builds. Do not loosen an assertion to make it pass.

reference_support.py builds the reference on first use. Its run_reference points a test daemon's next boot at the reference daemon and CLI. downgrade_test.py uses it to restart a home this checkout upgraded on the reference build, and checks that the reference reads a session this checkout wrote. A rollback across a breaking migration is not supported, so bump the pin past each new breaking migration. See downgrades.

areas.toml selects the file for changes to transfer code, adapters, migrations, and the core tables in conversation.gleam and mail.gleam. A schema change in another owner reaches it only through ./test.sh --full. Run it directly after adding a column or table that a transfer carries.

Offline Antigravity scenarios #

antigravity_test.py drives the real adapter through a private copy of the runner's sealed daemon. The fixture replaces the streaming transport's destination function and the non-streaming HTTP binding in that copy and reseals it before boot. Production endpoints and profile settings remain unchanged.

The route guard accepts only the exact inference URL, the dummy bearer token, and the fixture's marked home. The compiled destination is a numeric route on the loopback fake provider. Other streaming destinations and non-streaming provider HTTP calls are refused. The safety probe checks refusals before socket acquisition, including malformed headers.

Run nix develop -c python3 test/e2e/run.py antigravity_test.py for adapter scenarios. They cover tool execution, title-request separation, usage, restart replay, cancellation, and conditional signature fallback. Fixture signatures check preservation, not Google's cryptographic validation. OAuth, Google TLS, live quotas and acceptance of genuine signatures require separate live checks.

Manual web review #

Use the UI review kit for seeded browser reviews, scenario helpers, evidence capture, and private-resource cleanup. These manual checks are separate from the automated gate.