# Folio implementation and release plan

Updated 2026-11-07. The store model below replaces the earlier write/publication lifecycle and snapshot-import design. The store, memory adapter, transport, basic public-repository client, Wasm browser binding, IndexedDB storage, and React Query adapter are implemented. Cryptographic repository verification and broader integrations remain to build. The [earlier review](implementation-plan-review.md) records the previous plan and does not certify this revision.

## 1. The result we are building

Folio is an embedded Rust database that mirrors all or part of ATProto repositories. It contains data the user can write to, within the user's repositories, and data the user cannot write to, outside those repositories. A user can have one public repository and a repository in each Space they participate in. Folio can also mirror accessible public and Space repositories belonging to other users for reading.

A repository is identified by its author's DID and optional Space URI: `RepositoryId { did, space: None }` is the author's public repository; `Some(space_uri)` is that author's repository in a Space. A shared Space spans separate author repositories, potentially on different PDSes. The authenticated account, repository author, repository host, and Space authority are distinct.

Apps read locally from selected replicas. Local record edits target only the authenticated user's repositories, with publication further limited by the app's granted collection/action permissions. Space access does not grant permission to edit another author's repository. Read-only replicas still receive storage commits when remote data is pulled.

The first product target is Pixl: read a cached gallery, then save record edits and photo bytes offline. Metadata coverage and offline blob availability are separate. Spaces, additional language bindings, and history tools come later.

## 2. Current implementation

- [folio-store](../crates/folio-store/src/lib.rs) defines persisted replicas with `load`/`commit` and independent blob caching with `get_blob`/`put_blob`.
- [folio-store-memory](../crates/folio-store-memory/src/lib.rs) is a reference adapter. Clones share storage; dropping every handle loses it. It makes no durability claim.
- [Client](../crates/folio-client/src/client.rs) receives the account DID explicitly and selects its public repository with `get_pub_repo()` or its Space repository with `get_priv_repo(space_uri)?`. Constructors and selection make no network requests; Space I/O returns an unsupported error for now. Public handles provide `get`, `list`, `put`, `delete`, local `conflicts` inspection, revision-checked `resolve`, explicit `sync`, and callback-based `sync_with`. The first `get` or `list` refreshes all loaded collections plus the requested collection at one checkpoint; cached reads and edits stay local. Sync refreshes that coherent mirror and applies the chosen conflict policy before publishing a captured edit batch with a repository commit guard. The default leaves conflicts for review. Each successful transaction receipt advances the mirror directly and preserves newer local edits. Downloads check record CIDs and require a stable remote checkpoint; they trust the discovered PDS rather than verify repository signatures or Merkle proofs.
- Blob bytes are stored separately from replica state, scoped by repository and CID. `put_blob(bytes, mime_type)` saves verified bytes locally and returns a BlobRef; `get_blob(&reference)` reads cached bytes or downloads and caches missing bytes. Sync uploads bytes referenced by pending record edits before publishing them. The browser exposes `putBlob(file)` and `getBlob(reference)` with the same behavior.
- [Transport](../crates/folio-transport/src/transport.rs) has one async `send` method using standard HTTP requests/responses and buffered byte bodies. The [native HTTP adapter](../crates/folio-transport-http/README.md) lives in a separate crate: `HttpTransport` uses reqwest and authenticates on its first request. The `xrpc` helper delegates typed request construction and response parsing to Jacquard; the client uses its generated ATProto bindings and data model.
- The [E2E test](../crates/folio/tests/e2e.rs) exercises a live public repository with the memory store: lazy collection loading, optimistic local create/update, explicit sync, independent PDS reads, and deletion through Folio. It loads host-owned credentials from the ignored `.env`; the test itself is ignored by default and runs only with `cargo test -p folio --test e2e -- --ignored`. The live round trip passed on 2026-10-12.

- [folio-wasm](../crates/folio-wasm/src/lib.rs) binds the existing Rust client to JavaScript-supplied storage, fetch, and notifications. The memory and browser stores share one serializable `Replica` representation.
- [`@folio/client`](../packages/client/README.md) opens IndexedDB-backed public replicas, lets local operations continue while sync waits on network, emits committed-change notifications, and exchanges tab hints. The host supplies authenticated fetch and can set `resolveConflict` to `"remote"`, `"local"`, or a synchronous callback when opening Folio. [`@folio/react-query`](../packages/react-query/README.md) provides query options and one subscription connection per client/cache pair; local mutations save through Folio and synchronization remains explicit.
- `devenv test` runs formatting, Clippy, Rust and browser tests, and the isolated Astro/React package check. `devenv tasks run folio:build` builds Wasm and the Bun packages; `devenv up browser` serves the interactive fixture with real IndexedDB and a mock PDS.

Add an adapter or binding when a working consumer requires it; do not introduce architecture-only crates.

## 3. Two record states

A repository holds the last remote snapshot and local edits:

```rust
struct Repo {
    revision: u64,
    snapshot: Option<Snapshot>,
    edits: BTreeMap<Key, Edit>,
}

struct Edit {
    base: Option<Data>,
    value: Option<Data>,
}
```

A record has only two editing states:

| State | Stored data | Visible value |
| --- | --- | --- |
| Mirrored | No local edit | The remote value |
| Edited | Original base and desired value | The desired value, including deletion |

An absent base means a create; an absent desired value means a deletion. Removing the entire edit means use the remote value again. Repeated local saves replace the desired value and keep the original base. Competing local saves use the last committed desired value; the store does not retain proposals or edit history.

A snapshot contains one remote checkpoint, a set of completely loaded collections, and complete record data. Every stored record has its body. Missing keys in loaded collections are absent; unloaded collections have no implied contents. A local edit requires a loaded collection and a known base, including known absence. There is no bodyless-record or whole-repository coverage variant. Loading another collection refreshes the union; records from different checkpoints are never merged.

There are no write IDs, selected-write pointers, predecessor chains, active flags, persisted request states, staging IDs, or sealed-ID sets.

## 4. Store operations

`load(repository)` returns one coherent owned `Replica`: its revision, optional complete snapshot, and local edits. Record bodies and the remote index are shared immutably. Current backends load the complete replica; query projections and pagination are deferred until a backend needs them.

`commit(repository, &replacement)` atomically replaces that replica if `replacement.revision` is still current. The store assigns the next revision and returns `Some(new_revision)`. A stale revision returns `None`; stale commits and storage errors leave persisted state unchanged. The caller's replacement is borrowed and stays unchanged. There is no separate expected revision, partial update language, or send lifecycle.

Cancellation can leave a durable backend's outcome unknown, so load again before retrying. The browser stores the serialized replica directly, with one revision source, and resolves commits only after its IndexedDB transaction completes. Schema v3 keeps blob bytes in a separate DID/CID store; upgrades from v1/v2 preserve saved records and edits. Blob cache writes do not change the replica revision.

`get_blob(repository, cid)` and `put_blob(repository, cid, bytes)` store verified content-addressed bytes independently of replica replacement. Blob cache writes do not change record state or require upload flags. Bytes remain available for pending records and previews; automatic eviction and garbage collection are not implemented.

The client constructs replacement replicas; the store does not interpret protocol commands, decode ATProto records, authorize requests, or decide conflicts. Implementations and their futures need no particular runtime or `Send` bound. Native error sources require `Send + Sync`; browser sources can retain JavaScript exceptions.

## 5. Editing and synchronization

Conflicts are derived by comparing the original base B, local desired value L, and latest remote value R. Compare canonical values/CIDs, with explicit absence. Application conflict policy runs only on genuine divergence, after convergence and unchanged-server cases have been handled.

| Comparison | Client behavior |
| --- | --- |
| L = R | The desired value is present remotely; the edit can be removed after a suitable fresh read. |
| R = B | The local edit can be sent with the remote API's supported preconditions. |
| Otherwise | Apply the configured policy; the default leaves the edit saved for manual review and stops publication. |

Returning a local edit to its original base still expresses a desired value. If the remote value changed in the meantime, preserve that intent for conflict review.

A conflict is derived from those values. `sync()` keeps manual review as the default. `sync_with(FnMut(&Conflict) -> Result<Resolution>)` chooses local, remote, or a replacement value for each divergent record:

```rust
repo.sync_with(|_| Ok(Resolution::Remote)).await?; // prefer server
repo.sync_with(|_| Ok(Resolution::Local)).await?;  // prefer pending local edit
```

The callback receives the key and base/local/remote values. `Resolution::Replace(Some(value))` supplies an application merge; `Replace(None)` deletes. All decisions are validated and committed together before any remote writes. A failing callback leaves the decisions unapplied. Ordinary offline edits whose server base is unchanged do not invoke the callback.

In JavaScript, `Folio.open({ did, resolveConflict: "remote" })` configures server preference; `"local"` configures local preference. A synchronous callback can return either choice or `{ replace: value }`, including `null` for deletion. Omitting `resolveConflict` preserves manual review. Local preference selects the pending value when synced. Timestamp-based last-write-wins belongs to the application: it must define ordering and deterministic ties for both records and deletions. Folio does not invent a shared clock or infer time from record keys.

For manual review, `conflicts()` returns key/base/local/remote values with the current store revision. `resolve(key, revision, Resolution)` removes the edit for remote preference or rebases a kept/replacement value onto the reviewed remote value. A stale revision returns `SyncError::LocalChanged`; inspect conflicts again before deciding. There is no automatic field merge or CRDT.

Local writes are serialized within one browser client, while cached reads and local writes continue during network waits. After downloading all loaded collections, sync reloads the latest local replica and reconciles those edits against the fresh snapshot. Local edits during the download therefore participate normally. If another operation loaded a collection outside the downloaded set, the pull returns a stale-state error instead of merging snapshots from different checkpoints. A short local compare-and-commit protects that replacement; the remote `swapCommit` protects publication. Sync calls serialize within one browser client; separate clients and native handles rely on revision checks. There is no retry framework.

The resolved edit batch is immutable. Transactions contain at most 200 edits; each successful receipt supplies the exact accepted checkpoint, so no confirmation download is needed. Each accepted chunk is saved before the next begins; a later failure preserves earlier acknowledged chunks and remaining edits. A known successful A→B request followed by a newer local C preserves C and rebases its remaining edit onto accepted B. Receipt installation requires the same preceding snapshot and never merges with another operation's remote snapshot. A competing commit during the short replacement can return a stale-state error; the caller decides whether to run another pass.

A failed or ambiguous request leaves its local edits saved and ends that pass with an error. A response without commit metadata cannot acknowledge edits; the next explicit pull reconciles them. Do not blindly retry a timed-out request or claim it failed remotely. Pull and compare values before deciding what to do next. This simple model deliberately does not preserve a request journal across restart: it cannot guarantee exact reconstruction of lost responses, undo while an old request remains outstanding, or protection from every delayed request. Those guarantees are deferred instead of being encoded speculatively in the store.

Download and verify a complete replacement snapshot outside the store, then commit it with a revision check. An interrupted download leaves the last usable snapshot intact. A stale commit requires rereading and revalidating the candidate; do not retry an old download against a new revision without checking it. Incoming snapshots must fit in memory initially. Chunked durable import is deferred until a real backend needs it.

## 6. Authentication and repository access

Authentication supplies an account DID and an authorized request capability. The host application owns login, credentials, refresh, signing, HTTP, and proxies. Transport carries requests and responses across that boundary; one implementation may route multiple hosts only with destination-appropriate authorization. Credentials do not belong in the store.

`Transport::send(Request<Vec<u8>>) -> Result<Response<Vec<u8>>, Self::Error>` carries HTTP status, headers, and bytes. The native adapter also preserves HTTP version. HTTP 4xx/5xx responses remain responses for the client to interpret. An error means no complete response was returned; it does not prove the server rejected a write. The trait imposes no async runtime or `Send` requirement on implementations and futures; native host error sources must be `Send + Sync`, while Wasm sources can remain local. The native implementation uses reqwest and returns its concrete `folio_transport_http::Error` enum, retaining the HTTP or authentication source.

`HttpTransport::new(did, handle, password)?` synchronously stores credentials. Its first request discovers a `did:plc` account's PDS and caches its session. It scopes its token to that PDS's origin, disables redirects and automatic retries, and sets a 30-second timeout. The host can call `invalidate_authorization().await` to discard the session across clones and authenticate again on the next request; in-flight requests keep their existing token. Invalidation does not replay a request. OAuth remains with the host. `Client::new(did, store, transport)` owns the account identity and lazily discovers the repository host; repository coordinates are not exposed by the transport. The [E2E example](../crates/folio/examples/e2e.rs) demonstrates construction and the local read/edit/sync flow. Bodies are fully buffered initially; streaming remains deferred.

Broader replica support must resolve each repository author's trusted identity and host separately and reject local record edits or blob uploads targeting another author's repository. The current client selects only the configured account's repositories; other authors' replicas remain future work. Ownership does not bypass the app's granted permissions. Loss of network access or permission is not evidence of remote deletion.

Public repository support comes first. Space support must be based on a tested host contract for discovery, credentials, snapshot shape, and supported writes. Repository identity already separates authors and Spaces; it does not claim permission or protocol support. Do not assume Space APIs provide the same conditional writes as public repositories.

## 7. Implementation order

1. **Public repository verification.** The basic read/edit/sync flow against a trusted PDS is implemented and live-tested. Add cryptographic repository verification before treating mirrors as independently verified data.
2. **Durable storage.** Browser/Wasm and IndexedDB are implemented. Tests cover completed transactions, reopen, failed commits, competing revisions, and account/database isolation. Chromium verification covers offline edits, reload recovery, explicit mock-PDS sync, and cross-tab UI refresh. The first backend replaces the complete replica in one transaction; larger collections and browser quota/eviction behavior need further workload testing.
3. **Basic offline record editing.** Local create/update/delete, one original-base/desired-value edit per key, and a finite guarded sync pass are implemented. Sync preserves newer local edits and applies a callback policy or reports conflicts for manual review. Conflict inspection exposes base/local/remote values, and manual resolution requires the reviewed store revision.
4. **Pixl read integration.** The React Query package and standalone React fixture are implemented; Pixl migration remains. Use one client and subscription connection for the gallery. Replace direct DOM changes/reloads with local reads and updates. Test pagination and offline cached metadata before adding uploads.
5. **Offline photo bytes.** Separate durable blob storage, cache-first downloads, and upload-before-record publication are implemented. Pixl still needs to use these APIs for file selection and previews, generating keys/timestamps once. Records do not download every blob automatically; an offline application shell and cache lifecycle policy remain application work.
6. **Broader replicas.** Test other authors' public data, then experimental Space replicas. Keep other authors read-only and isolate identical keys by author/Space. Add incremental pulls or chunked snapshot storage only when measured workloads require them.
7. **Additional consumers.** Add native SQLite, Node, or Swift bindings as actual consumers need them. Reuse the tested data model; packaging and durable runtime behavior need their own checks.

No release number constitutes a completed milestone. Do not advertise durable saves before a persistent backend exists, or safe automatic recovery of ambiguous writes under this model.

## 8. Verification

The memory adapter's tests cover complete replica replacement, stale atomic commits, repository isolation, detached loads, competing commits, borrowed inputs staying unchanged, and revision exhaustion. The Store trait test guards support for non-`Send` adapters.

The end-to-end test in `crates/folio/tests/` now runs through public APIs against the account's existing public repository. It checks optimistic local reads, synchronized creates and updates, and synchronized deletion with independent PDS reads. The core mock tests cover coherent collection refresh, publication workset races, lost and missing receipts, newer local edits during acknowledgement, snapshot/CAS races, whole-batch conflict rejection, and recovery after later-chunk failure. Browser package tests now cover reopening durable edits, offline cached reads/mutations, validation failures, and query cache invalidation. No separate test crate is needed.

The browser checks use a mock PDS and do not establish live browser authorization, Pixl integration, an offline application shell, or cryptographic protocol verification. Blob tests cover persisted bytes and publication ordering; they do not establish a complete offline photo application. A memory-adapter test does not establish durable recovery or browser behavior. The live test establishes this password-authenticated public-repository round trip; it does not establish those broader guarantees.

## 9. Deliberate omissions

No retained alternative edits, automatic local history, publication state machine, persisted attempt journal, import lifecycle, public multi-record transaction API, automatic field merge, or CRDT. No promise of exactly-once remote effects, full uncertain-request recovery, network-wide repository discovery, or browser background execution.

Keep ordinary ATProto records and the current crate boundaries. Prefer a working local read/edit/sync example over adding machinery for outcomes the first client does not yet handle. Add complexity only when a demonstrated consumer requirement outweighs its cost.