# existing storage migrations

## migration test inventory

Keep these tests while the corresponding old format remains supported. A
fresh-install scenario cannot exercise an upgrade from that format. Retire a
test only together with its migration or compatibility reader and an explicit
decision to stop supporting that upgrade path.

| Supported data | Implementation owner | Coverage |
| --- | --- | --- |
| Older core and extension schemas, inline images, and session preferences | Core migrations, extension-contributed migrations, and preference import | [startup scenarios](../test/e2e/migrations_test.py), [mail schema upgrade](../test/daemon/mail_migration_test.gleam), [migration ordering](../test/harness/migration_test.gleam) |
| Older transcript metadata, titles, and timestamps | Conversation columns and transcript classification | [conversation tests](../test/daemon/conversation_test.gleam), [metadata backfill](../test/daemon/transcript_sources_test.gleam) |
| Inline image records and TEXT image payloads | Image-store and cell-image migrations | [image-store tests](../test/daemon/image_store_test.gleam), [cell image readers](../test/harness/tool_image_test.gleam) |
| Legacy item-count compaction state | Rolling's lazy compatibility reader | [rolling tests](../test/harness/rolling_test.gleam) |
| Older Python namespace snapshots | Kernel snapshot reader | [kernel state scenarios](../test/e2e/kernel_state_test.py) |
| Flat provider configuration, relocated credentials, and implicit account rotation | Provider configuration and credential migration | [settings scenarios](../test/e2e/settings_test.py), [startup integration](../test/e2e/integration_test.py), [credential migration](../test/e2e/credentials_test.py), [account migration](../test/e2e/account_migration_test.py) |
| Older layouts inspected offline | Daemon storage helper compatibility reader | [storage compatibility scenarios](../test/e2e/storage_test.py) |

These are groups within feature suites, not a separate test lane. The normal
gate runs them. Compatibility with another running daemon build and provider
response shapes has its own contracts; those tests are not retired merely
because a storage migration is removed.

core implementations live under `src/albedo/daemon/migrations/`;
extension implementations live under their own `migrations/` directories and
contribute `MigrationPlugin` values. the host applies them, without importing
extension-specific migration modules.

## startup order

`server.main` first claims `ALBEDO_HOME/daemon.lock` with `BEGIN EXCLUSIVE`.
Offline cleanup in `cli/internal/storage` takes the same lock after confirmation,
before it revalidates candidates. Its Gleam helper holds ownership through file
deletion and vacuum. Daemon startup refuses while maintenance owns the home,
before it touches application storage. Neither owner commits the lock transaction,
changes its journal mode, or replaces the lock file; the CLI reads the lock's
holder to tell a stale `daemon.json` from a live daemon. Cleanup checks health for
diagnostics; only exclusive ownership permits offline mutation. The CLI checks
cancellation before sending authorization; it does not rely on asynchronous
helper termination to prevent an already-cancelled cleanup from starting.
Cancellation or death stops the helper and releases ownership. Deletions already
completed after authorization remain deleted if cancellation, later deletion,
or vacuum fails.

The client invokes `albedo-daemon storage inspect <database> <scratch-directory>`
and `albedo-daemon storage maintain <home>` through the same local executable
selection as daemon startup. `daemon/storage_cli` owns both helper modes and
dispatches before server startup. Inspection copies the database and WAL into
the client's private scratch directory, compares native file identities before
and after copying, then reads the snapshot without schema initialization or
migrations. It supports older layouts independently of the live projection.
The client removes scratch files and retains its filesystem accounting and
cleanup revalidation. Helper metadata uses the native timestamp resolution.

Maintenance exchanges newline-delimited JSON with the client. It captures
candidate identities under the home lock, reports readiness, and waits for the
client's authorization after revalidation. An EOF watcher terminates the helper
VM during authorized work, including vacuum. Go keeps the pipe open until the
helper exits and accepts a result only with a successful exit. These helper
modes do not require Python or create a daemon discovery record.

`runtime.start` installs the extension registry before `server.prepare_storage`.
Installation batches the owners' initialisers and schema migrations into one
transaction on the unpublished store. Each extension has a savepoint: an error
or raised initialiser rolls back that extension before quarantine, without
undoing successful owners. Initialisers use writer queries, not reader snapshots,
and must not publish the store or commit its transaction. Fresh core schema preparation
uses a second batch before data migrations. Existing core stores keep their
bounded transcript-recovery commits. Those migrations retain their own
transaction and backup boundaries. `store.transaction` uses a savepoint inside an
existing schema batch and keeps `BEGIN IMMEDIATE` outside one.
for each installed extension, `extension.install` first calls its table
initialiser, then applies its contributed `SchemaMigration` callbacks on the
store-owned SQLite connection, in plugin order. work contributes `cwd.apply`;
paperclips contributes `title.apply`, `scope.apply`, and `reply.apply`.
The links extension's `ledger.initialise` calls `links.apply` to create
`workspace_links`, then creates its revision tables and triggers. See
[linked workspaces](workspaces.md#linked-workspaces). The mcp extension's initialiser creates `mcp_catalogues` (see [mcp](mcp.md#saved-catalogues)); python contributes `kernel_links.apply`, the detached kernel's link tables (see [kernel](kernel.md)), which `link.create` also ensures for embedded runtimes that boot kernels without the extension. their ledgers create tables but do not run
upgrades themselves. all installed owners upgrade, even when disabled for
sessions; session selection and live reload never rerun migrations.

`server.prepare_storage` then runs before sessions start:

1. `conversation.initialise` creates the core tables, then calls
   `conversation_columns.apply`: the existing session columns first, transcript
   columns second. These columns include root-peer provenance, role, and close state.
   The same core initialization creates `operations`,
   `pending_inputs`, `submission_events`, and `continuation_markers` through
   `operations.schema`.
   `operations.initialise` upgrades historical display and continuation metadata
   to unique `(session,operation_id)` keys in one transaction, preserving their
   rowids. Admission receipts and pending inputs still require globally unique
   input identities. It then runs its existing domain-local session recovery
   (missing activity/title from transcript), then creates `sessions_activity`.
2. quota, mail, and family initialise their tables. Mail also ensures its incoming and outgoing conversation indexes after upgrading an older kind constraint.
3. `migrations.run(ledger, backup)` runs the core `image_store.run`, which first
   externalizes legacy transcript images, then converts image-table TEXT data
   to decoded BLOB. It then runs `transcript_classes.run` to classify old
   transcript rows, and `session_profiles.run`, which once fills
   `session_profiles` (each session's distinct row providers, what a profile
   rename looks up; see [settings](settings.md#provider-and-mcp-values)) from
   every transcript row and installs the triggers that keep it. That first
   start reads the whole transcript, about 7 seconds for a 3 GB store. `runtime.migrate(host, backup)` then collects installed
   extensions' `DataMigration` callbacks in registry/plugin order and applies
   them with the same backup path. python contributes `cell_images.run`, which
   requires the core image and marker tables. callbacks report named row counts;
   the server logs nonzero extension results as `migration <name>: <n> rows`.
   a failed callback stops startup and prevents later migrations from running.
4. the existing best-effort filesystem credential migration runs; then provider
   assignment from legacy configuration runs, unchanged.

schema creation is still owned by each subsystem. `work/migrations/cwd.apply`
checks `PRAGMA table_info(work)`, adds the existing reserved `__albedo_legacy__`
cwd default when absent, then creates `work_cwd_id`;
`paperclips/migrations/title.apply` adds the existing non-null empty-default title
column, `paperclips/migrations/scope.apply` drops the now-unused workspace
index left from the cwd-scoped vent era, and
`paperclips/migrations/reply.apply` adds the non-null empty-default reply
column that makes a /paperclips answer durable. `rolling/migrations/observation.apply`
drops the observation table rolling used to write and read straight back. these upgrades run immediately
after their owner's table creation, so later extension initialisers can use
the upgraded schema. embedding hosts call
`runtime.migrate` after preparing core storage and before opening sessions.

## provider configuration and credential migration

Before sessions start, `albedo_credentials.migrate` moves legacy secrets and then
runs `albedo_account_migration` under the shared settings lock. Only an existing
`config.json` without the account marker is upgraded; startup never creates
config or credential files just to mark a fresh home. Settings API config writes
set the marker, so profiles created after this upgrade are not legacy pools.
The upgrade writes `config.json`, `creds.json`, and the top-level config marker
`accountsVersion: 2` in one settings recovery journal. Interrupted publication
finishes before the next settings read. Unreadable settings are left intact.

The upgrade assigns public IDs to stored sign-in accounts and makes old implicit
pools explicit: one account becomes `accountId`, several become `accountIds`.
Existing explicit pins and lists stay unchanged. OpenAI-compatible profile keys
become deduplicated endpoint/key accounts; their profile secrets are removed.
OpenCode Go keeps its former sibling-key/account pool as an explicit list.
Alibaba profiles become OpenAI profiles on chat completions; their profile keys
and stored Alibaba keys move into `accounts.openai`, retaining stored endpoints
and limit marks. Their bindings include the former pool's environment account
IDs when those variables are set, but never store the environment values.

Later starts do not expand deliberately plain profiles. They still move newly
hand-written profile keys into key accounts, replacing an old pin or rotation
with the new key's pin.
Legacy secret-file backups and their migration notices are unchanged.

## compatibility and interruption

- `conversation_columns.apply` adds nullable `transcript.row_class`, checked
  against `user`, `image_fit`, and `other`. Live writes set it in the same
  transaction as the payload; forks copy it with the retained prefix.
  `transcript_classes.run` classifies only NULL rows in 128-row transactions;
  completed batches survive interruption. An unreadable payload, here or in
  `conversation`'s startup indexing, is healed by `conversation.heal_in`: its
  bytes move to `transcript_quarantine` (one row per sequence ID, deleted with
  its session), and the row becomes a user-role system note saying it was set
  aside, classed `user`. Startup continues. The backfill adds no migration
  marker. It runs after the image migration.
- a tool result saved after rows that are neither calls nor results (a
  capability note written while the call ran, then the result recovered after a
  restart) moves back beside its call: providers reject a call whose result does
  not follow it. `tool_results.settle` is the one rule. `conversation` applies it
  when it commits results, `session_history` applies it to the live history and to
  a transcript when it loads, and saves the moves from that load. A move keeps
  every seq and moves row contents between them, and remaps `transcript_calls`,
  `submission_events`, `continuation_markers`, `provider_requests`,
  `operations.committed_seq`, and `transcript_quarantine` with them. LCM source
  ranges are not remapped.
- the request builder pairs the same way at the request boundary:
  `tool_results.pair` gives every call a request's provider items make exactly
  one tool result right after the block that made it. A late result moves
  there, a call no result answers gets `tool_results.unpaired_result`
  ("interrupted, no output"), and a result whose call nothing carries — one a
  projection dropped, or a second answer to a call already answered — stays
  out of the request: providers reject a `tool_use` without its `tool_result`
  and a `tool_result` without its `tool_use` alike. The transcript is left
  as it is: the pairing is applied again to every request, so a session an
  older fold left unpaired sends valid history without a repair.
- `transcript_pending_class` indexes unclassified sequence IDs.
  `transcript_users` indexes `(session,seq)` for user and image-fit rows;
  `transcript_fits` indexes `(session,seq)` for image-fit rows alone. Completed
  databases have no pending entries, so subsequent startups do not decode
  transcript payloads for classification.
- the existing `migrations(name,applied_at)` table remains unchanged. the exact
  markers are `image_store` and `cell_images`, recorded with `unixepoch()`.
  transcript migration skips when its marker exists; cell migration skips when
  its marker exists, and does nothing without a cells table.
- TEXT-to-BLOB migration deliberately has no marker. it checks
  `typeof(data)='text'` on every run and converts pending rows, even when the
  `image_store` marker already exists. invalid legacy base64 fails without
  replacing that row.
- transcript and cell scans retain their original cursor order and 16-row pages;
  BLOB conversion retains its 16-row limit. each page uses the existing
  transaction helper. there is no new all-migrations transaction or transaction
  policy change. markers are written at the original completion points, and
  interrupted runs recheck remaining rows.
- core `backup.gleam` and python's cell migration preserve the two existing
  backup call patterns: transcript and BLOB migration check for an existing file before directory creation and
  `VACUUM INTO`, retaining the `image store backup failed: ` error prefix; cell
  migration ensures the directory first and copies on the first page containing
  inline images, outside its transaction. the server still supplies one shared
  `home/backups/albedo-before-image-store-<timestamp>.sqlite` path. no backup is
  overwritten and backup triggering is unchanged (the transcript candidate
  check can include tool outputs without inline images).
- live writes and transcript migration share `image_payloads.insert`, the
  unchanged decoded-BLOB insertion helper. image hashes, serialized formats,
  image validation, and legacy inline fallback reads are unchanged. the existing
  Erlang conversion and cell packing functions remain the FFI implementation.

## migration inventory

| previous implementation | current implementation |
| --- | --- |
| `conversation.initialise`: session/transcript `add_columns` | `migrations/conversation_columns.apply` |
| transcript row classification | `migrations/transcript_classes.run`, after core image upgrades |
| `images.migrate` / `migrate_legacy` / transcript pages | `migrations/image_store.run` / `migrate_legacy` / transcript pages |
| `images.migrate_blobs` / BLOB pages | `migrations/image_store.migrate_blobs` / BLOB pages |
| `images.backup_before_migration` | `migrations/backup.image_store` |
| `python/cells.migrate_images` / cell pages | `python/migrations/cell_images.run` / cell pages |
| `python/cells.backup_before_migration` | `python/migrations/cell_images.backup_before_migration` |
| `work/ledger.initialise`: cwd alteration and index | `work/migrations/cwd.apply`, contributed by work |
| `paperclips/ledger.initialise`: title addition | `paperclips/migrations/title.apply`, contributed by paperclips |
| `paperclips/ledger.initialise`: `paperclips_cwd` index creation | dropped from the schema; `paperclips/migrations/scope.apply` removes it from existing stores |
| `paperclips/ledger.initialise`: reply column | `paperclips/migrations/reply.apply`, added with the global ledger |
| `rolling.initialise`: `rolling_compaction_observation` creation | dropped from the schema; `rolling/migrations/observation.apply` removes it from existing stores |

`images.migrate` and `cells.migrate_images` are removed rather than wrapped:
the migration implementations are separate from the live domain readers and
writers. startup discovers extension upgrades through plugins, not direct calls
to their implementation modules.

## intentionally not extracted

- session recovery stays in `conversation`: it uses the domain's transcript
  decoding/title helpers and still runs at exactly the same point. core schema
  bootstrap and index creation remain with their owners.
- rolling compaction's legacy item-count-to-user-cut conversion remains lazy in
  `rolling/extension.resume`. it needs the current prepared history and source,
  validates the old fingerprint (including legacy image payload fingerprints),
  and only writes a cut when the history matches and its tail begins with a user
  message. treating this as an unconditional startup migration would change its
  timing and semantics; its projections are untouched.
- credentials migration is filesystem/auth, not SQLite. legacy configuration's
  provider assignment likewise remains in the existing server startup path.
- no zstd, compression, schema-version, offline storage upgrade,
  new dependency, or unrelated transaction fix is part of this refactor.

## durable operation admissions

`operations` owns receipt retention and pending delivery state on the shared
store connection. Session creation commits its session row and accepted receipt
in one immediate transaction. Submission admission commits its receipt and
pending input together. Transcript consumption moves pending metadata into
`submission_events`, records its committed position, and removes pending input
in the same transaction as the transcript rows. Failed statements roll back the
whole transaction, including a failed commit.

Receipts retain the original admission result. They do not reference the session
row, so deletion cancels pending inputs and leaves receipts available. Terminal
receipts expire seven days after acceptance for creation or rejection, or after
commit or cancellation for submissions. Pending receipts do not expire. Cleanup
removes at most 128 expired receipts per call. Durable display metadata in
`submission_events` remains until session deletion, including operation IDs,
display text, and echo identity. Continuations commit a `continuation_markers`
row in the same transaction beside their prompt, a system-notice transcript row
the history view leaves out. These
markers also remain until session deletion and survive receipt cleanup.
Forks copy this historical metadata with its original input IDs, timestamps,
and turn IDs at the branch's remapped transcript positions. A continuation
follows its anchor row, so only markers anchored strictly before the selected
checkpoint belong to the fork. Forks copy no admission receipts or live runs.

Acceptance guarantees stored input or its committed or cancelled outcome.
Commit means the input reached the transcript. Neither guarantee promises that
assistant generation, tools, or external effects occur exactly once.