oppi.li
1# Carstore
2
3Store a zillion users of PDS-like repo, with more limited operations (mainly: firehose in, firehose out).
4
5## [ScyllaStore](scylla.go)
6
7Blocks stored in ScyllaDB.
8User and PDS metadata stored in gorm (PostgreSQL or sqlite3).
9
10## [FileCarStore](bs.go)
11
12Store 'car slices' from PDS source subscribeRepo firehose streams to filesystem.
13Store metadata to gorm postgresql (or sqlite3).
14Periodic compaction of car slices into fewer larger car slices.
15User and PDS metadata stored in gorm (PostgreSQL or sqlite3).
16FileCarStore was the first production carstore and used through at least 2024-11.
17
18## [SQLiteStore](sqlite_store.go)
19
20Experimental/demo.
21Blocks stored in trivial local sqlite3 schema.
22Minimal reference implementation from which fancy scalable/performant implementations may be derived.
23
24```sql
25CREATE TABLE IF NOT EXISTS blocks (uid int, cid blob, rev varchar, root blob, block blob, PRIMARY KEY(uid,cid))
26CREATE INDEX IF NOT EXISTS blocx_by_rev ON blocks (uid, rev DESC)
27
28INSERT INTO blocks (uid, cid, rev, root, block) VALUES (?, ?, ?, ?, ?) ON CONFLICT (uid,cid) DO UPDATE SET rev=excluded.rev, root=excluded.root, block=excluded.block
29
30SELECT rev, root FROM blocks WHERE uid = ? ORDER BY rev DESC LIMIT 1
31
32SELECT cid,rev,root,block FROM blocks WHERE uid = ? AND rev > ? ORDER BY rev DESC
33
34DELETE FROM blocks WHERE uid = ?
35
36SELECT rev, root FROM blocks WHERE uid = ? AND cid = ? LIMIT 1
37
38SELECT block FROM blocks WHERE uid = ? AND cid = ? LIMIT 1
39
40SELECT length(block) FROM blocks WHERE uid = ? AND cid = ? LIMIT 1
41```