fork of hydrant
hydrant docs specs db-layer.md
14 kB · 317 lines
Markdown

# Spec: db layer restructuring Status: draft Tracking: beads epic (see `bd show`, filed alongside this spec) ## Motivation The db layer works, but its correctness rests on conventions that every callsite must independently remember. Concretely: - `Db` (`src/db/mod.rs:48`) exposes every keyspace as a public `Keyspace` field plus `pub inner: Arc<Database>`. Any module can write raw bytes to any keyspace with hand-built keys; `db.inner.batch()` appears in 27+ files. - Four access styles coexist: half-dead static async helpers (`Db::get`, with `insert`/`remove`/`contains_key` marked `#[allow(dead_code)]`), direct sync calls inside `spawn_blocking` closures, direct sync calls on dedicated blocking threads, and free functions taking `(batch, db, ...)`. - Key formats live in free functions returning `Vec<u8>` (`src/db/keys/`) with prefix constants scattered per module — `db/filter.rs` maintains its own `SEP`/prefix constants separate from `keys::SEP`. The authoritative documentation of key formats is comments (duplicated again in AGENTS.md). - Serialization helpers exist (`ser_repo_state` etc.) but ~31 files call `rmp_serde` directly, several deserializing `RepoState` raw and bypassing the helpers (`backfill/worker/process.rs:58`, `backfill/manager.rs:122`, `backfill/worker/task.rs:187`). None of these are hypothetical risks: 1. **The record-import invariant is duplicated three times.** A record write must keep in sync: `records` ks, `blocks` ks, per-collection counts, backlinks (cfg), a `StoredEvent`, and a jetstream event (cfg). This is implemented once properly in `ops::apply_commit` (via `RecordEmitter`), then re-implemented by hand in `backfill/sparse.rs` (`persist_sparse_backfill` builds `StoredEvent` + event key + `jetstream::stage_event` manually, and carries its own cfg'd backlinks block) and twice more in `backfill/worker/process.rs`. Divergence between these is silent index drift — missing events, missing backlinks, wrong counts. 2. **Three manual registries…