Folio implementation and release plan #

Updated 2026-10-24. The store model below replaces the earlier write/publication lifecycle and snapshot-import design. The store, memory adapter, transport, basic public-repository client, Wasm browser binding, IndexedDB storage, and React Query adapter are implemented. Cryptographic repository verification and broader integrations remain to build. The earlier review records the previous plan and does not certify this revision.

1. The result we are building #

Folio is an embedded Rust database that mirrors all or part of ATProto repositories. It contains data the user can write to, within the user's repositories, and data the user cannot write to, outside those repositories. A user can have one public repository and a repository in each Space they participate in. Folio can also mirror accessible public and Space repositories belonging to other users for reading.

A repository is identified by its author's DID and optional Space URI: RepositoryId { did, space: None } is the author's public repository; Some(space_uri) is that author's repository in a Space. A shared Space spans separate author repositories, potentially on different PDSes. The authenticated account, repository author, repository host, and Space authority are distinct.

Apps read locally from selected replicas. Local record edits target only the authenticated user's repositories, with publication further limited by the app's granted collection/action permissions. Space access does not grant permission to edit another author's repository. Read-only replicas still receive storage commits when remote data is pulled.

The first product target is Pixl: read a cached gallery, then save record edits offline, then add offline photo bytes. Metadata coverage and offline blob availability are separate. Spaces, additional language bindings, and history tools come later.

2. Current implementation #

• folio-store defines the persisted data and two async methods: read and commit.

• folio-store-memory is a reference adapter. Clones share storage; dropping every handle loses it. It makes no durability claim.

• Client receives the account DID explicitly and selects its public repository with get_pub_repo() or its Space repository with get_priv_repo(space_uri)?. Space I/O returns an unsupported error for now. Public handles provide get, list, put, delete, and explicit sync. The first get or list downloads the requested collection; cached reads and edits stay local. Sync refreshes loaded collections, pushes eligible edits with a repository commit guard, and pulls to confirm while preserving newer local edits. Downloads check record CIDs and require a stable remote checkpoint; they trust the discovered PDS rather than verify repository signatures or Merkle proofs.

• Transport has one async send method using standard HTTP requests/responses and buffered byte bodies. The native HTTP adapter lives in a separate crate: HttpTransport uses reqwest and authenticates on its first request. The xrpc helper delegates typed request construction and response parsing to Jacquard; the client uses its generated ATProto bindings and data model.

• The E2E test exercises a live public repository with the memory store: lazy collection loading, optimistic local create/update, explicit sync, independent PDS reads, and deletion through Folio. It loads host-owned credentials from the ignored .env and runs only when explicitly requested. The live round trip passed on 2026-10-12.

• folio-wasm binds the existing Rust client to JavaScript-supplied storage, fetch, and notifications. The memory and browser stores share serializable Replica state and its read/commit logic.

• @folio/client opens IndexedDB-backed public replicas, serializes engine operations, emits committed-change notifications, and exchanges tab hints. The host supplies authenticated fetch. @folio/react-query provides query options and one subscription connection per client/cache pair; local mutations save through Folio and synchronization remains explicit.

• cargo make build runs wasm-pack and the Bun workspace builds. cargo make test checks types and exercises the real Wasm engine with fake IndexedDB and a CID-correct mock PDS.

Add an adapter or binding when a working consumer requires it; do not introduce architecture-only crates.

3. Two record states #

A repository holds the last remote snapshot and local edits:

struct Repo {
    revision: u64,
    snapshot: Option<Snapshot>,
    edits: BTreeMap<Key, Edit>,
}

struct Edit {
    base: Option<Data>,
    value: Option<Data>,
}

A record has only two editing states:

State	Stored data	Visible value
Mirrored	No local edit	The remote value
Edited	Original base and desired value	The desired value, including deletion

An absent base means a create; an absent desired value means a deletion. Removing the entire edit means use the remote value again. Repeated local saves replace the desired value and keep the original base. Competing local saves use the last committed desired value; the store does not retain proposals or edit history.

A snapshot contains the remote checkpoint, coverage, and an immutable shared record index. Missing coverage, known absence, and an indexed record whose body has not been downloaded remain distinct facts about remote data. They are not additional editing states. A local edit requires a known base, including known absence; unknown data is not an absent base.

There are no write IDs, selected-write pointers, predecessor chains, active flags, persisted request states, staging IDs, or sealed-ID sets.

4. Store operations #

read(repository, query) returns a coherent revision, snapshot, records, and continuation key. Point reads preserve requested order, including duplicate keys. Scans use lexical (collection, rkey) order. Collection selection and edited_only are scan filters. Deletions remain visible in edited-record scans.

commit(repository, expected_revision, commit) replaces supplied data atomically:

struct Commit {
    edits: BTreeMap<Key, Option<Edit>>,
    snapshot: Option<Snapshot>,
}

An omitted key is unchanged. An edit entry of None removes that local edit. An omitted snapshot leaves remote data unchanged. A supplied snapshot replaces the complete remote snapshot; it does not implicitly clear local edits. A client can replace a snapshot and adjust edits in the same commit.

A successful commit returns Some(new_revision). A stale revision returns None and changes nothing. Storage errors also change nothing. Cancellation can leave a durable backend's outcome unknown, so read again before retrying. These are storage/concurrency facts, not a domain error vocabulary.

The client constructs the replacement data. The store does not interpret create/select/prepare/accept/settle commands, decode ATProto data, verify repositories, authorize remote requests, or decide conflicts. The caller validates input at those boundaries. Implementations and their futures do not require Send or a particular async runtime.

5. Editing and synchronization #

Conflicts are derived by comparing the original base B, local desired value L, and latest remote value R. Compare canonical values/CIDs, with explicit absence. Do not compare timestamps or infer changes from unknown or unmaterialized data.

Comparison	Client behavior
L = R	The desired value is present remotely; the edit can be removed after a suitable fresh read.
L = B	No remaining local change; the edit can be removed.
R = B	The local edit can be sent with the remote API's supported preconditions.
Otherwise	Show base/local/remote for review; leave the edit saved and pause sending that record.

A conflict is a comparison result, not a persisted class or third editing state. Keeping remote removes the edit. Keeping local or supplying a replacement creates a new base/value pair against the reviewed remote value. Review must use current data. There is no generic field merge or CRDT.

The first client serializes synchronization per repository. Before sending, it captures the edit and relevant remote preconditions in memory. A response must reread local state and use a revision-checked commit; it must not blindly clear an edit saved while that request ran. For a known successful A→B request followed by a newer local C, preserve C and rebase its remaining edit onto confirmed B. These are client behaviors still to implement, not storage states.

A failed or ambiguous request leaves the local edit saved and ends that send pass with an error. Do not blindly retry a timed-out request or claim it failed remotely. Pull and compare values before deciding what to do next. This simple model deliberately does not preserve a request journal across restart: it cannot guarantee exact reconstruction of lost responses, undo while an old request remains outstanding, or protection from every delayed request. Those guarantees are deferred instead of being encoded speculatively in the store.

Download and verify a complete replacement snapshot outside the store, then commit it with a revision check. An interrupted download leaves the last usable snapshot intact. A stale commit requires rereading and revalidating the candidate; do not retry an old download against a new revision without checking it. Incoming snapshots must fit in memory initially. Chunked durable import is deferred until a real backend needs it.

6. Authentication and repository access #

Authentication supplies an account DID and an authorized request capability. The host application owns login, credentials, refresh, signing, HTTP, and proxies. Transport carries requests and responses across that boundary; one implementation may route multiple hosts only with destination-appropriate authorization. Credentials do not belong in the store.

Transport::send(Request<Vec<u8>>) -> Result<Response<Vec<u8>>, Self::Error> carries HTTP status, headers, and bytes. The native adapter also preserves HTTP version. HTTP 4xx/5xx responses remain responses for the client to interpret. An error means no complete response was returned; it does not prove the server rejected a write. The trait imposes no async runtime or Send requirement. The native implementation uses reqwest, with std::io::Error preserving the underlying HTTP or authentication error.

HttpTransport::new(did, handle, password)? synchronously stores credentials. Its first request discovers a did:plc account's PDS and authenticates once. It scopes its token to that PDS's origin, disables redirects and automatic retries, sets a 30-second timeout, and leaves refresh/OAuth to the host. Client::new(did, store, transport) owns the account identity and lazily discovers the repository host; repository coordinates are not exposed by the transport. The E2E example demonstrates construction and the local read/edit/sync flow. Bodies are fully buffered initially; streaming remains deferred.

The client resolves each repository author's trusted identity and host separately. It rejects local record edits and blob uploads targeting another author's repository. Ownership does not bypass the app's granted permissions. Loss of network access or permission is not evidence of remote deletion.

Public repository support comes first. Space support must be based on a tested host contract for discovery, credentials, snapshot shape, and supported writes. Repository identity already separates authors and Spaces; it does not claim permission or protocol support. Do not assume Space APIs provide the same conditional writes as public repositories.

7. Implementation order #

1. Public repository verification. The basic read/edit/sync flow against a trusted PDS is implemented and live-tested. Add cryptographic repository verification before treating mirrors as independently verified data.
2. Durable storage. Browser/Wasm and IndexedDB are implemented. Tests cover completed transactions, reopen, failed commits, competing revisions, and account/database isolation. Chromium verification covers offline edits, reload recovery, explicit mock-PDS sync, and cross-tab UI refresh. The first backend replaces the complete replica in one transaction; larger collections and browser quota/eviction behavior need further workload testing.
3. Basic offline record editing. Implement local create/update/delete using one edit per key. Keep the original base through repeated saves, use last committed local save semantics, and expose derived remote conflicts. Add a finite sync pass using supported remote preconditions and preserve newer local edits on responses.
4. Pixl read integration. The React Query package and standalone React fixture are implemented; Pixl migration remains. Use one client and subscription connection for the gallery. Replace direct DOM changes/reloads with local reads and updates. Test pagination and offline cached metadata before adding uploads.
5. Offline photo bytes. Persist blobs separately, retain dependencies of current edits, upload needed bytes before referencing records, and show local previews. Generate keys/timestamps once. Add an app-owned offline shell; cached records alone do not provide offline pixels or navigation.
6. Broader replicas. Test other authors' public data, then experimental Space replicas. Keep other authors read-only and isolate identical keys by author/Space. Add incremental pulls or chunked snapshot storage only when measured workloads require them.
7. Additional consumers. Add native SQLite, Node, or Swift bindings as actual consumers need them. Reuse the tested data model; packaging and durable runtime behavior need their own checks.

No release number constitutes a completed milestone. Do not advertise durable saves before a persistent backend exists, or safe automatic recovery of ambiguous writes under this model.

8. Verification #

The memory adapter's tests cover replacing/removing edits, explicit creates/deletions, complete snapshot replacement, preserved edits during pulls, stale atomic commits, scan filters/order/coverage, repository isolation, detached reads, competing commits, and unchanged data on revision exhaustion. The Store trait test guards support for non-Send adapters.

The end-to-end test in crates/folio/tests/ now runs through public APIs against the account's existing public repository. It checks optimistic local reads, synchronized creates and updates, and synchronized deletion with independent PDS reads. A mock transport also tests failed sends, ordinary conflicts, and newer local edits during confirmation. Browser package tests now cover reopening durable edits, offline cached reads/mutations, validation failures, and cache invalidation during an overlapping read. No separate test crate is needed.

The browser checks use a mock PDS and do not establish live browser authorization, Pixl integration, an offline application shell, offline blob bytes, or cryptographic protocol verification. A memory-adapter test does not establish durable recovery or browser behavior. The live test establishes this password-authenticated public-repository round trip; it does not establish those broader guarantees.

9. Deliberate omissions #

No retained alternative edits, automatic local history, publication state machine, persisted attempt journal, import lifecycle, public multi-record transaction API, automatic field merge, or CRDT. No promise of exactly-once remote effects, full uncertain-request recovery, network-wide repository discovery, or browser background execution.

Keep ordinary ATProto records and the current crate boundaries. Prefer a working local read/edit/sync example over adding machinery for outcomes the first client does not yet handle. Add complexity only when a demonstrated consumer requirement outweighs its cost.