this repo has no description
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

wip

+2404 -559
+615 -559
rewrite.md
··· 1 - # Redesigning docstore architecture layers 1 + # Docstore Query Engine Architecture 2 + 3 + A layered query engine for IndexedDB with support for compound indexes, spread (denormalized) indexes, joins, and aggregations. 2 4 3 - ## Current Architecture Analysis 5 + > **Implementation Status Legend:** 6 + > 7 + > - ✅ = Fully implemented 8 + > - 🟡 = Partially implemented 9 + > - ❌ = Not yet implemented 4 10 5 - What you've built is impressive — it's essentially a **relational query engine on top of IndexedDB**. The key components: 11 + --- 6 12 7 - 1. **Schema Layer** (`z.object` + `StorageRef`) — Zod-based type definitions with special handling for blob references 8 - 2. **Index Definition** (`IndexBuilder`, `IndexDef`) — Declarative compound/spread indexes with replication 9 - 3. **Denormalization Engine** (`ingest()`) — Cartesian product expansion for spread indexes into separate IDB stores 10 - 4. **Statistics** (`StatsManager`, `HyperLogLog`) — Cardinality estimation for query planning 11 - 5. **Query Planning** (`QueryPlanner`) — Cost-based index selection, intersection detection, self-join handling 12 - 6. **Execution** (`rows()`, `docs()`, `rowsPage()`) — Batched iteration, keyset pagination, intersection execution 13 - 7. **Joins** (in `docstore-joins.ts`) — Hash join, left join, nested loop for cross-store queries 13 + ## Query DSL ❌ 14 14 15 - ### Pain Points I See 15 + Three top-level combinators build queries from stores: 16 16 17 - 1. **Monolithic file** — 4000+ lines, hard to reason about individual concerns 18 - 2. **Schema and storage tightly coupled** — Can't easily swap IDB for another backend 19 - 3. **Query DSL is implicit** — `QuerySpec` is just data, no builder ergonomics 20 - 4. **Cross-store joins are separate** — `docstore-joins.ts` is disconnected from the planner 21 - 5. **Observables vs AsyncGenerators** — Two streaming models that don't integrate 22 - 6. **No transaction abstraction** — Raw IDB transactions leak through 17 + > **Status:** The high-level DSL (`where()`, `aggregate()`, `join()` combinators) is **not implemented**. Instead, queries are built imperatively using `QuerySpec` objects and executed via `QueryPlanner.plan()` + `rows()`/`docs()`. 23 18 24 - --- 19 + ### `where(source, filter)` 🟡 25 20 26 - ## Proposed Layered Architecture 21 + Filters a store or query. Returns `Query<T>`. 27 22 28 - Here's a modular redesign that separates concerns cleanly: 23 + > **Status:** Filtering exists but via `QuerySpec.where` (array of `WhereClause`) and `QuerySpec.matches` (array of `SpreadMatch`), not as a combinator function. Supports `=`, `!=`, `<`, `>`, `<=`, `>=`, `in` operators. Boolean `and`/`or` combinators are **not implemented** - all clauses are implicitly AND'd. 29 24 25 + ```typescript 26 + // Simple predicate 27 + where(entries, ['feedId', '=', 'f1']) 28 + 29 + // Spread match (all fields must match same array element) 30 + where(entries, ['...tags', 'contains', {tag: 'javascript'}]) 31 + 32 + // AND 33 + where(entries, { 34 + and: [ 35 + ['feedId', '=', 'f1'], 36 + ['readState', '=', 'unread'], 37 + ], 38 + }) 39 + 40 + // OR 41 + where(entries, { 42 + or: [ 43 + ['...tags', 'contains', {tag: 'javascript'}], 44 + ['...tags', 'contains', {tag: 'typescript'}], 45 + ], 46 + }) 47 + 48 + // Nested boolean logic 49 + where(entries, { 50 + and: [ 51 + ['feedId', '=', 'f1'], 52 + { 53 + or: [ 54 + ['readState', '=', 'unread'], 55 + ['starred', '=', true], 56 + ], 57 + }, 58 + ], 59 + }) 60 + 61 + // Chaining is implicit AND 62 + where(where(entries, ['feedId', '=', 'f1']), ['readState', '=', 'unread']) 30 63 ``` 31 - ┌─────────────────────────────────────────────────────────────────────────────┐ 32 - │ QUERY DSL (Layer 6) │ 33 - │ │ 34 - │ q.from('entries') │ 35 - │ .where('feedId', '=', 'f1') │ 36 - │ .where('...tags.tag', '=', 'javascript') │ 37 - │ .join('feeds', 'feedId', 'id') │ 38 - │ .select('title', 'feeds.name') │ 39 - │ .orderBy('publishedAt', 'desc') │ 40 - │ .limit(50) │ 41 - │ │ 42 - │ Compiles to → QueryAST │ 43 - └──────────────────────────────────────┬──────────────────────────────────────┘ 44 - 45 - ┌──────────────────────────────────────▼──────────────────────────────────────┐ 46 - │ CROSS-STORE PLANNER (Layer 5) │ 47 - │ │ 48 - │ • Resolves joins across stores │ 49 - │ • Picks join strategy (hash, merge, nested-loop) │ 50 - │ • Estimates cardinalities across stores │ 51 - │ • Produces JoinPlan (tree of store-level plans) │ 52 - │ │ 53 - │ Input: QueryAST + Catalog │ 54 - │ Output: JoinPlan { left: StorePlan, right: StorePlan, strategy, ... } │ 55 - └──────────────────────────────────────┬──────────────────────────────────────┘ 56 - 57 - ┌──────────────────────────────────────▼──────────────────────────────────────┐ 58 - │ STORE-LEVEL PLANNER (Layer 4) │ 59 - │ │ 60 - │ • Single-store index selection │ 61 - │ • Intersection detection (self-join on spread) │ 62 - │ • Predicate pushdown to key ranges │ 63 - │ • Covering index detection │ 64 - │ • Cost estimation from stats │ 65 - │ │ 66 - │ Input: StorePredicate[] + IndexCatalog + Stats │ 67 - │ Output: StorePlan { source, index, keyRange, filter, intersection?, ... } │ 68 - └──────────────────────────────────────┬──────────────────────────────────────┘ 69 - 70 - ┌──────────────────────────────────────▼──────────────────────────────────────┐ 71 - │ EXECUTOR (Layer 3) │ 72 - │ │ 73 - │ Volcano-style streaming operators: │ 74 - │ • ScanOp — iterate index with key range │ 75 - │ • FilterOp — apply residual predicates │ 76 - │ • ProjectOp — select fields │ 77 - │ • SortOp — in-memory sort when needed │ 78 - │ • LimitOp — early termination │ 79 - │ • HashJoinOp — build/probe join │ 80 - │ • MergeJoinOp — sorted merge join │ 81 - │ • IntersectOp — set intersection on doc IDs │ 82 - │ • LookupOp — point lookup by key │ 83 - │ │ 84 - │ All operators: AsyncGenerator<Row> → AsyncGenerator<Row> │ 85 - └──────────────────────────────────────┬──────────────────────────────────────┘ 86 - 87 - ┌──────────────────────────────────────▼──────────────────────────────────────┐ 88 - │ INDEX STORE (Layer 2) │ 89 - │ │ 90 - │ IndexStore { │ 91 - │ schema: SchemaDefinition │ 92 - │ indexes: IndexDefinition[] │ 93 - │ denormalizedStores: DenormalizedStore[] │ 94 - │ │ 95 - │ put(doc): Promise<void> │ 96 - │ get(key): Promise<Doc | undefined> │ 97 - │ delete(key): Promise<void> │ 98 - │ scan(index, range, direction): AsyncGenerator<Row> │ 99 - │ count(index, range): Promise<number> │ 100 - │ } │ 101 - │ │ 102 - │ • Manages primary + denormalized stores │ 103 - │ • Handles ingest/denormalization on write │ 104 - │ • Maintains statistics │ 105 - │ • Abstracts IDB transactions │ 106 - └──────────────────────────────────────┬──────────────────────────────────────┘ 107 - 108 - ┌──────────────────────────────────────▼──────────────────────────────────────┐ 109 - │ SCHEMA (Layer 1) │ 110 - │ │ 111 - │ SchemaDefinition { │ 112 - │ name: string │ 113 - │ fields: Map<string, FieldType> │ 114 - │ primaryKey: string │ 115 - │ storageRefs: Map<string, StorageRefSchema> │ 116 - │ } │ 117 - │ │ 118 - │ IndexDefinition { │ 119 - │ name: string │ 120 - │ keys: IndexKey[] // supports '...spread.path' notation │ 121 - │ replicate: string[] │ 122 - │ sparse: boolean │ 123 - │ spreadRoots: Set<string> │ 124 - │ } │ 125 - │ │ 126 - │ • Pure data definitions, no behavior │ 127 - │ • Zod integration for validation │ 128 - │ • Type inference utilities │ 129 - └──────────────────────────────────────┬──────────────────────────────────────┘ 130 - 131 - ┌──────────────────────────────────────▼──────────────────────────────────────┐ 132 - │ STORAGE ADAPTER (Layer 0) │ 133 - │ │ 134 - │ interface StorageAdapter { │ 135 - │ transaction(stores: string[], mode: 'r' | 'rw'): Transaction │ 136 - │ } │ 137 - │ │ 138 - │ interface Transaction { │ 139 - │ store(name: string): ObjectStore │ 140 - │ commit(): Promise<void> │ 141 - │ } │ 142 - │ │ 143 - │ interface ObjectStore { │ 144 - │ get(key): Promise<unknown> │ 145 - │ put(value): Promise<void> │ 146 - │ delete(key): Promise<void> │ 147 - │ getAll(range?, count?): Promise<unknown[]> │ 148 - │ getAllKeys(range?, count?): Promise<IDBValidKey[]> │ 149 - │ count(range?): Promise<number> │ 150 - │ index(name): Index │ 151 - │ } │ 152 - │ │ 153 - │ Implementations: │ 154 - │ • IDBStorageAdapter (browser) │ 155 - │ • SQLiteStorageAdapter (server/testing) │ 156 - │ • MemoryStorageAdapter (unit tests) │ 157 - └─────────────────────────────────────────────────────────────────────────────┘ 64 + 65 + Filter type: 66 + 67 + ```typescript 68 + type Filter<T> = 69 + | [keyof T | string, Op, unknown] // predicate (string for spread paths) 70 + | {and: Filter<T>[]} 71 + | {or: Filter<T>[]} 72 + 73 + type Op = '=' | '!=' | '<' | '>' | '<=' | '>=' | 'contains' 158 74 ``` 159 75 160 - --- 76 + > **Status:** Current implementation uses `WhereClause` interface: `{key: string, op: CompareOp, value: unknown}`. Supports `=`, `!=`, `<`, `>`, `<=`, `>=`, `in` (but not `contains`). No boolean `and`/`or` combinators yet. For spread matching, use `SpreadMatch` which handles the "all fields must match same array element" case. 77 + > 78 + > **Design Note — Boolean Combinators:** 79 + > The `or` combinator requires special handling since IDB can only scan one index at a time. Two approaches: 80 + > 81 + > 1. **Union of plans**: Execute each OR branch as a separate query plan, collect doc IDs, deduplicate. Works well when branches use different indexes. 82 + > 2. **CNF normalization**: Convert to conjunctive normal form, then use `IN` queries for small disjunctions on the same key (e.g., `status = 'a' OR status = 'b'` becomes `status IN ['a', 'b']`, already supported via cursor hopping). 83 + > 84 + > For nested boolean logic, represent as a tree in `QuerySpec`, then flatten during planning. The planner already handles `IN` efficiently via `plan.inValues`. 85 + 86 + ### `aggregate(source, groupKeys, aggregateFn)` 🟡 87 + 88 + Groups and aggregates a store or query. Returns `Query<GroupedRow>`. 89 + 90 + > **Status:** Aggregation exists via `collectIndexValues()` and `collectKeyPairs()` for tag-cloud/faceted-search use cases. General `aggregate()` combinator with fold functions (`count()`, `sum()`, `topN()`, etc.) is **not implemented**. 91 + > 92 + > **Design Note — General Aggregation:** 93 + > The existing `collectIndexValues()` is essentially `aggregate(source, [keyField], count())`. To generalize: 94 + > 95 + > 1. Define `AggregateFn<T, Acc, Out>` as shown above - this is a standard fold interface. 96 + > 2. `aggregate(source, groupKeys, aggFn)` would: 97 + > - Plan index selection favoring indexes that cover `groupKeys` (enables streaming aggregation without collecting all rows) 98 + > - Stream rows via `rows()`, accumulate into `Map<groupKey, Acc>` 99 + > - For `topN()`, maintain a bounded heap per group 100 + > 3. When groupKeys match index prefix, can use IDB's natural ordering to detect group boundaries and emit incrementally. 101 + > 4. Built-in aggregators: `count()`, `sum(field)`, `min/max(field)`, `collect()`, `topN(n, cmp)` - all are just `{init, step, finalize}` objects. 102 + 103 + ```typescript 104 + // Count entries per feed 105 + aggregate(entries, ['feedId'], count()) 106 + // -> Query<{ feedId: string, count: number }> 161 107 162 - ## Layer Details 108 + // Multiple group keys 109 + aggregate(entries, ['feedId', 'readState'], count()) 110 + // -> Query<{ feedId: string, readState: string, count: number }> 163 111 164 - ### Layer 0: Storage Adapter 112 + // Different aggregators 113 + aggregate(entries, ['feedId'], sum('duration')) 114 + aggregate(entries, ['feedId'], max('publishedAt')) 115 + aggregate( 116 + entries, 117 + ['feedId'], 118 + topN(3, (a, b) => b.publishedAt - a.publishedAt), 119 + ) 120 + ``` 165 121 166 - **Goal:** Abstract away IDB specifics so we can test with in-memory storage or port to SQLite. 122 + Aggregator functions are folds: 167 123 168 124 ```typescript 169 - // storage-adapter.ts 170 - interface StorageAdapter { 171 - transaction(stores: string[], mode: 'readonly' | 'readwrite'): Transaction 172 - install(schemas: StoreSchema[]): Promise<void> 125 + type AggregateFn<T, Acc, Out = Acc> = { 126 + init: () => Acc 127 + step: (acc: Acc, row: T) => Acc 128 + finalize?: (acc: Acc) => Out 173 129 } 174 130 175 - interface Transaction { 176 - store(name: string): ObjectStore 177 - done: Promise<void> // resolves when committed 178 - } 131 + // Built-ins 132 + count() // -> number 133 + sum(field) // -> number 134 + ;(min(field), max(field)) // -> T[field] | undefined 135 + collect() // -> T[] 136 + topN(n, compareFn) // -> T[] 137 + ``` 179 138 180 - interface ObjectStore { 181 - get(key: IDBValidKey): Promise<unknown> 182 - put(value: unknown): Promise<void> 183 - delete(key: IDBValidKey): Promise<void> 184 - getAll(range?: KeyRange, count?: number): Promise<unknown[]> 185 - getAllKeys(range?: KeyRange, count?: number): Promise<IDBValidKey[]> 186 - count(range?: KeyRange): Promise<number> 187 - index(name: string): Index 188 - } 139 + ### `join(left, right, template)` 🟡 140 + 141 + Joins two stores/queries. Returns `Query<TemplateOutput>`. 189 142 190 - interface Index extends Omit<ObjectStore, 'put' | 'delete' | 'index'> {} 143 + > **Status:** Joins exist in `docstore-joins.ts` as pure in-memory functions: `hashJoin()`, `leftJoin()`, `hashJoinAggregate()`, `nestedLoopJoin()`, `nestedLoopLeftJoin()`. These work on already-materialized arrays, not as query combinators integrated with the planner. 144 + > 145 + > **Design Note — Planner-Integrated Joins:** 146 + > The current join functions work but require manual orchestration. To integrate with the planner: 147 + > 148 + > 1. `join()` combinator returns a `JoinQuery<L, R, Out>` that captures both sources and key extractors 149 + > 2. Planner produces a `JoinPlan` with strategy selection: 150 + > - **Hash join** (current): When right side is small enough to fit in memory. Build hash map from right, probe with left. 151 + > - **Nested loop with batched lookup**: When left is much smaller than right. Use `getMany()` for batched fetches (already used in `docs()`) 152 + > - **Merge join**: When both sides are ordered by join key (requires index). Stream both sides, merge on matching keys. 153 + > 3. Key insight: The existing `docs()` function IS a nested loop join (index rows → primary store). Generalizing this pattern to arbitrary stores is straightforward. 191 154 192 - // Key range abstraction (maps to IDBKeyRange) 193 - type KeyRange = 194 - | {type: 'only'; value: unknown} 195 - | {type: 'bound'; lower?: unknown; upper?: unknown; lowerOpen?: boolean; upperOpen?: boolean} 196 - | {type: 'lowerBound'; value: unknown; open?: boolean} 197 - | {type: 'upperBound'; value: unknown; open?: boolean} 155 + ```typescript 156 + join( 157 + [feeds, 'id'], // left source + join key 158 + [entries, 'feedId'], // right source + join key 159 + (f, e) => ({ 160 + // template callback 161 + ...f.row(), 162 + entries: e.row(), 163 + }), 164 + ) 165 + // -> Query<Feed & { entries: Entry }> 198 166 ``` 199 167 200 - ### Layer 1: Schema 201 - 202 - **Goal:** Pure data definitions, fully decoupled from storage. 168 + The template callback receives extractors for each side: 203 169 204 170 ```typescript 205 - // schema/types.ts 206 - interface SchemaDefinition<S extends FieldShape = FieldShape> { 207 - name: string 208 - fields: S 209 - primaryKey: keyof S & string 210 - storageRefs: Map<string, StorageRefSchema> 171 + interface Extractor<T> { 172 + row(): T // full document 173 + field<K extends keyof T>(k: K): T[K] // single field 211 174 } 175 + ``` 212 176 213 - interface IndexDefinition<S extends FieldShape = FieldShape> { 214 - name: string 215 - keys: IndexKey<S>[] 216 - replicate: (keyof S & string)[] 217 - sparse: boolean 218 - spreadRoots: Set<string> 219 - } 177 + #### Aggregated joins ❌ 220 178 221 - // schema/builder.ts 222 - function defineStore<S extends FieldShape>( 223 - name: string, 224 - options: { schema: S; primaryKey?: keyof S; indexes?: IndexBuilder<S>[] } 225 - ): SchemaDefinition<S> 179 + The template can include aggregates, producing per-left-row aggregate values: 226 180 227 - function defineIndex<S extends FieldShape>(name?: string): IndexBuilder<S> 181 + > **Status:** Not implemented. Use `hashJoinAggregate()` from `docstore-joins.ts` for similar functionality on materialized arrays. 182 + > 183 + > **Design Note — Aggregate Joins:** 184 + > `hashJoinAggregate()` already implements the core algorithm. To integrate with the DSL: 185 + > 186 + > 1. Template callback returns an object with `aggregate()` calls for specific fields 187 + > 2. Planner detects these aggregate markers and collects them into an `AggregateSpec[]` 188 + > 3. Execution builds a single hash map from right side, but instead of emitting per-match, accumulates into aggregator per left row 189 + > 4. **Batching optimization**: When multiple aggregates share the same grouping, compute all in a single pass over the right side. The template `{unreadCount: aggregate(...), latestUnread: aggregate(...)}` becomes one scan with two accumulators. 228 190 229 - // schema/paths.ts 230 - // Type-level path extraction (the IndexablePath magic) 231 - type IndexKey<S> = /* ... recursive path extraction ... */ 191 + ```typescript 192 + join([feeds, 'id'], [where(entries, ['readState', '=', 'unread']), 'feedId'], (f, e) => ({ 193 + ...f.row(), 194 + unreadCount: aggregate(e, ['feedId'], count()), 195 + latestUnread: aggregate(e, ['feedId'], topN(1, byPublishedDesc)), 196 + })) 197 + // -> Query<Feed & { unreadCount: number, latestUnread: Entry[] }> 232 198 ``` 233 199 234 - ### Layer 2: Index Store 200 + The planner detects aggregate-joins and batches multiple aggregates into a single pass over the right side. 235 201 236 - **Goal:** Encapsulate all storage operations for a single store + its denormalized indexes. 202 + **Constraint:** Cannot mix `e.row()` and `aggregate(e, ...)` in the same template—that would produce ambiguous output multiplicity. 237 203 238 - ```typescript 239 - // store/index-store.ts 240 - class IndexStore<S extends FieldShape> { 241 - readonly schema: SchemaDefinition<S> 242 - readonly indexes: IndexDefinition<S>[] 243 - readonly stats: StatsManager 204 + --- 244 205 245 - constructor(adapter: StorageAdapter, schema: SchemaDefinition<S>, indexes: IndexDefinition<S>[]) 206 + ## Ordering, Pagination, Execution ✅ 246 207 247 - // CRUD 248 - async put(doc: Infer<S>): Promise<void> 249 - async putMany(docs: Infer<S>[]): Promise<void> 250 - async get(key: IDBValidKey): Promise<Infer<S> | undefined> 251 - async getMany(keys: IDBValidKey[]): Promise<Map<IDBValidKey, Infer<S>>> 252 - async delete(key: IDBValidKey): Promise<void> 208 + Queries are inert until executed. Ordering and limits are specified at execution time: 253 209 254 - // Scanning (used by executor) 255 - scan(options: ScanOptions): AsyncGenerator<Row> 256 - countRows(options: ScanOptions): Promise<number> 210 + > **Status:** Implemented. `rows()` and `docs()` are async generators. `rowsPage()` supports cursor-based pagination. `collect()`, `take()`, `skip()` helpers exist. Ordering via `QuerySpec.orderBy` and `RowsOptions.orderBy`. 257 211 258 - // Stats access (used by planner) 259 - getStats(): StoreStats 260 - getCardinality(field: string): Promise<number> 212 + ```typescript 213 + // Streaming execution 214 + for await (const row of query.rows({orderBy: ['publishedAt', 'desc'], limit: 50})) { 215 + console.log(row) 261 216 } 262 217 263 - interface ScanOptions { 264 - store: 'primary' | string // primary or denormalized store name 265 - index?: string // null = full scan 266 - range?: KeyRange 267 - direction?: 'next' | 'prev' 218 + // Collect all 219 + const rows = await query.rows({orderBy: ['publishedAt', 'desc']}).collect() 220 + 221 + // First match 222 + const first = await query.rows({limit: 1}).first() 223 + 224 + // Hydrate to full documents (lookup join to primary store) 225 + for await (const doc of query.docs({orderBy: ['publishedAt', 'desc']})) { 226 + console.log(doc) 268 227 } 269 228 ``` 270 229 271 - ### Layer 3: Executor 230 + - `rows()` returns what the chosen index provides (may be denormalized rows with `_docId` + replicated fields) 231 + - `docs()` always returns full documents from the primary store via lookup join 272 232 273 - **Goal:** Streaming operators that compose to execute query plans. 233 + ### Keyset pagination ✅ 274 234 275 235 ```typescript 276 - // executor/operators.ts 236 + const page1 = await query.rows({orderBy: ['publishedAt', 'desc'], limit: 20}).collect() 237 + const cursor = page1.cursor // opaque cursor from last row 277 238 278 - // Base type: all operators are async generators 279 - type RowStream = AsyncGenerator<Row, void, undefined> 239 + const page2 = await query.rows({orderBy: ['publishedAt', 'desc'], limit: 20, after: cursor}).collect() 240 + ``` 280 241 281 - // Scan operator - entry point from storage 282 - function scan(store: IndexStore, options: ScanOptions): RowStream 242 + > **Status:** Implemented via `rowsPage()` returning `PageResult<T>` with `cursor: PageCursor | null`. Cursor contains `indexKeyValues`, `primaryKey`, and `direction`. Serialization via `serializeCursor()`/`deserializeCursor()`. Batched iteration via `iterateRows()` uses keyset pagination internally. 283 243 284 - // Filter - apply residual predicates 285 - function filter(source: RowStream, predicates: Predicate[]): RowStream 244 + --- 286 245 287 - // Project - select fields 288 - function project(source: RowStream, fields: string[]): RowStream 246 + ## Planner ✅ 289 247 290 - // Sort - materialize and sort (when index order doesn't match) 291 - function sort(source: RowStream, key: string, direction: 'asc' | 'desc'): RowStream 248 + The planner transforms a `Query` into an `ExecutionPlan`. Two layers: 249 + 250 + > **Status:** Implemented as `QueryPlanner` class. Takes `QuerySpec`, produces `QueryPlan`. Includes index selection, predicate pushdown, intersection planning, and cost-based decisions using stats. 251 + 252 + ### Store-Level Planner ✅ 253 + 254 + Optimizes queries against a single store: 292 255 293 - // Limit - early termination 294 - function limit(source: RowStream, n: number): RowStream 256 + 1. **Index selection** — picks the index that best satisfies filters + orderBy ✅ 257 + 2. **Predicate pushdown** — converts equality/range predicates to IDB key ranges ✅ (`KeyRangeSpec`, compound keys supported) 258 + 3. **Intersection detection** — for spread indexes with multiple `contains` predicates, plans set intersection on `_docId` ✅ (`IntersectionPlan` with chained `.next`) 259 + 4. **Residual filters** — predicates that can't be pushed down become post-scan filters ✅ (`plan.filter`) 260 + 5. **Available fields** — tracks which fields are present in the chosen index's rows ✅ (`plan.availableFields`) 295 261 296 - // Skip - offset (avoid if possible, use keyset pagination instead) 297 - function skip(source: RowStream, n: number): RowStream 262 + Output: 298 263 299 - // executor/joins.ts 264 + ```typescript 265 + interface StorePlan { 266 + source: string // primary store or denormalized index store name 267 + index: string | null // which index (null = primary key scan) 268 + range: KeyRange | null // pushed-down predicates 269 + direction: 'next' | 'prev' 270 + residualFilter: Filter | null 271 + availableFields: string[] // fields present in result rows 272 + estimatedRows: number 273 + } 274 + ``` 300 275 301 - // Hash join - build map from right, probe with left 302 - function hashJoin<L, R, Out>( 303 - left: RowStream, 304 - right: RowStream, 305 - leftKey: (row: L) => unknown, 306 - rightKey: (row: R) => unknown, 307 - project: (l: L, r: R) => Out, 308 - ): AsyncGenerator<Out> 276 + > **Status:** Implemented as `QueryPlan` interface. Also includes `selectivity`, `docIdField`, `intersection?: IntersectionPlan`, `inValues`/`inPrefix` for cursor-hopping `IN` queries. 309 277 310 - // Merge join - both sides sorted on join key 311 - function mergeJoin<L, R, Out>( 312 - left: RowStream, 313 - right: RowStream, 314 - leftKey: (row: L) => unknown, 315 - rightKey: (row: R) => unknown, 316 - project: (l: L, r: R) => Out, 317 - ): AsyncGenerator<Out> 278 + ### Cross-Store Planner 🟡 318 279 319 - // Nested loop with lookup - for small outer, indexed inner 320 - function nestedLoopJoin<L, R, Out>( 321 - outer: RowStream, 322 - lookup: (key: unknown) => Promise<R | undefined>, 323 - outerKey: (row: L) => unknown, 324 - project: (l: L, r: R) => Out, 325 - ): AsyncGenerator<Out> 280 + Handles joins and sorting that requires fields not in the chosen index: 326 281 327 - // executor/intersect.ts 282 + 1. **Join strategy** — picks hash join, merge join, nested loop, or lookup join based on cardinality estimates ❌ (joins are manual via `docstore-joins.ts`) 283 + 2. **Aggregate-join detection** — when template contains aggregates, plans grouped aggregation during join ❌ 284 + 3. **Lookup join insertion** — when `orderBy` field isn't in `availableFields`, wraps with lookup join to primary store ✅ (`docs()` handles this automatically) 285 + 4. **Sort operator insertion** — when index order doesn't match requested order ✅ (`plan.needsSort`, in-memory sort in `rowsSortedDirect()`) 328 286 329 - // Set intersection - collect IDs from multiple sources 330 - function intersect(sources: {stream: RowStream; docIdField: string}[]): AsyncGenerator<IDBValidKey> // yields doc IDs in intersection 287 + Output: 331 288 332 - // executor/execute.ts 289 + ```typescript 290 + interface ExecutionPlan { 291 + root: PlanNode 292 + finalSort?: {field: string; direction: 'asc' | 'desc'} 293 + limit?: number 294 + } 333 295 334 - // Main entry point - takes a plan tree, returns row stream 335 - function execute(plan: ExecutionPlan, catalog: Catalog): RowStream 296 + type PlanNode = 297 + | {type: 'scan'; plan: StorePlan} 298 + | {type: 'filter'; source: PlanNode; filter: Filter} 299 + | {type: 'sort'; source: PlanNode; field: string; direction: 'asc' | 'desc'} 300 + | {type: 'limit'; source: PlanNode; n: number} 301 + | {type: 'hashJoin'; left: PlanNode; right: PlanNode; leftKey: string; rightKey: string} 302 + | {type: 'lookupJoin'; source: PlanNode; targetStore: string; sourceKey: string; targetKey: string} 303 + | { 304 + type: 'aggregateJoin' 305 + left: PlanNode 306 + right: PlanNode 307 + leftKey: string 308 + rightKey: string 309 + aggregates: AggregateSpec[] 310 + } 311 + | {type: 'intersect'; sources: PlanNode[]; docIdField: string} 336 312 ``` 337 313 338 - ### Layer 4: Store-Level Planner 314 + > **Status:** The current implementation uses a flat `QueryPlan` object rather than a tree of `PlanNode`s. Operators are hardcoded in the executor functions. The intersection is embedded in `QueryPlan.intersection` as a linked list. 315 + > 316 + > **Design Note — Plan Node Tree:** 317 + > The flat `QueryPlan` structure works well for single-store queries. A tree structure becomes valuable when: 318 + > 319 + > 1. **Joins are planner-integrated**: `{type: 'hashJoin', left: PlanNode, right: PlanNode}` naturally represents the join structure 320 + > 2. **Plan visualization**: Tree structure maps directly to EXPLAIN output 321 + > 3. **Operator composition**: Each node type has a corresponding executor generator, composed via `yield*` 322 + > 323 + > Migration path: Keep `QueryPlan` for single-store, introduce `PlanNode` when adding join planning. The `intersection` linked list is already tree-like; could become `{type: 'intersect', sources: PlanNode[]}`. 324 + 325 + ### Lookup join for sorting 339 326 340 - **Goal:** Optimize single-store queries - index selection, intersection, predicate pushdown. 327 + When a query filters on a denormalized index but sorts on a non-replicated field: 341 328 342 329 ```typescript 343 - // planner/store-planner.ts 330 + where(entries, ['...tags', 'contains', {tag: 'audio'}]) 331 + // executed with: .rows({ orderBy: ['duration', 'desc'] }) 332 + ``` 333 + 334 + The planner: 335 + 336 + 1. Picks `entries_idx:tags` store with `by_tag` index for filtering 337 + 2. Sees `duration` not in `availableFields` 338 + 3. Wraps with lookup join to `entries` primary store 339 + 4. Adds sort operator on `duration` 340 + 341 + --- 342 + 343 + ## Executor ✅ 344 344 345 - interface StorePlan { 346 - source: 'primary' | string // which IDB store to read from 347 - index: string | null // which index (null = full scan or pk lookup) 348 - range: KeyRange | null // pushed-down predicates as key range 349 - direction: 'next' | 'prev' 350 - filter: Predicate[] // residual predicates 351 - projection: string[] | null // covering index fields, or null for full doc 352 - intersection?: IntersectionPlan 353 - estimatedRows: number 354 - selectivity: number 355 - availableFields: string[] // fields available in result rows (for sort validation) 356 - } 345 + Volcano-style streaming operators. All operators are `AsyncGenerator<Row>`. 357 346 358 - interface IntersectionPlan { 359 - source: string 360 - index: string 361 - range: KeyRange 362 - docIdField: string 363 - next?: IntersectionPlan // chain for N-way 364 - } 347 + > **Status:** Implemented. Uses batched `getAll()` with keyset pagination (`iterateRows()`) for ~1000x fewer IDB round-trips than cursor loops. 365 348 366 - class StorePlanner { 367 - constructor(schema: SchemaDefinition, indexes: IndexDefinition[], stats: StoreStats) 349 + ```typescript 350 + // Scan — entry point from storage 351 + function* scan(store: IndexStore, plan: StorePlan): AsyncGenerator<Row> 368 352 369 - plan(predicates: Predicate[], orderBy?: OrderBy): Promise<StorePlan> 370 - } 353 + // Filter — apply residual predicates 354 + function* filter(source: AsyncGenerator<Row>, predicate: Filter): AsyncGenerator<Row> 355 + 356 + // Sort — materialize and sort (when index order doesn't match) 357 + function* sort(source: AsyncGenerator<Row>, field: string, direction: 'asc' | 'desc'): AsyncGenerator<Row> 358 + 359 + // Limit — early termination 360 + function* limit(source: AsyncGenerator<Row>, n: number): AsyncGenerator<Row> 361 + 362 + // Hash join — build hash map from right, probe with left 363 + function* hashJoin(left: AsyncGenerator<Row>, right: AsyncGenerator<Row>, leftKey, rightKey): AsyncGenerator<Row> 364 + 365 + // Lookup join — for each left row, batch lookup from target store 366 + function* lookupJoin(source: AsyncGenerator<Row>, targetStore: IndexStore, sourceKey, targetKey): AsyncGenerator<Row> 367 + 368 + // Aggregate join — hash join with grouped aggregation on right side 369 + function* aggregateJoin(left, right, leftKey, rightKey, aggregates): AsyncGenerator<Row> 370 + 371 + // Intersect — set intersection on doc IDs from multiple index scans 372 + function* intersect(sources: AsyncGenerator<Row>[], docIdField: string): AsyncGenerator<IDBValidKey> 371 373 ``` 372 374 373 - **Important:** The Store-Level Planner reports `availableFields` in the plan. If the 374 - caller requests `orderBy` on a field not in `availableFields`, this signals to the 375 - Cross-Store Planner (Layer 5) that a **lookup join** back to the primary store is needed. 376 - The Store-Level Planner doesn't handle this itself — it just provides the information. 375 + > **Status:** 376 + > 377 + > - `scan` ✅ — `rowsDirect()` with batched `iterateRows()` 378 + > - `filter` ✅ — inline via `matchesFilters()` in scan loop 379 + > - `sort` ✅ — `rowsSortedDirect()` materializes and sorts 380 + > - `limit` ✅ — `take()` helper, or just break the generator loop 381 + > - `hashJoin` ✅ — `hashJoin()` in `docstore-joins.ts` (but on arrays, not generators) 382 + > - `lookupJoin` ✅ — `docs()` batches lookups to primary store 383 + > - `aggregateJoin` ❌ — not as a streaming operator 384 + > - `intersect` ✅ — `rowsIntersectDirect()` with chained `IntersectionPlan` 385 + > 386 + > **Design Note — Streaming Aggregate Join:** 387 + > To add as an `AsyncGenerator` operator: 388 + > 389 + > 1. Materialize right side into `Map<K, Acc[]>` (one accumulator array per group, one entry per aggregate function) 390 + > 2. Stream left side, for each row: lookup by key, call `finalize()` on accumulators, yield combined result 391 + > 3. For memory efficiency when right side is huge but ordered by join key: detect group boundaries during streaming, finalize and evict completed groups incrementally 377 392 378 - ### Layer 5: Cross-Store Planner 393 + Execution reads from storage using paginated cursors to reduce await overhead. 379 394 380 - **Goal:** Handle joins, pick join strategies, optimize across stores. 395 + --- 396 + 397 + ## Index Store 🟡 398 + 399 + Encapsulates a primary store and its denormalized index stores. 400 + 401 + > **Status:** No `IndexStore` class. Instead, `StoreDef<S>` holds schema/index definitions, and `docstore.install()`, `put()`, `get()`, etc. operate directly on IDB. 402 + > 403 + > **Design Note — IndexStore Class:** 404 + > The current function-based API (`put(db, storeDef, doc)`) is explicit but verbose. An `IndexStore` class could: 405 + > 406 + > - Bind `db` and `storeDef` once: `const entries = new IndexStore(db, entriesStoreDef)` 407 + > - Provide `entries.put(doc)`, `entries.get(key)`, `entries.query({where: ...})` 408 + > - Hold a reference to `StatsManager` for automatic stats updates 409 + > 410 + > Trade-off: Class adds convenience but hides the db/storeDef dependency. The current explicit style is more functional and testable. Consider a thin wrapper class that delegates to the existing functions rather than moving logic into the class. 381 411 382 412 ```typescript 383 - // planner/join-planner.ts 413 + class IndexStore<T> { 414 + readonly schema: SchemaDefinition<T> 415 + readonly indexes: IndexDefinition[] 384 416 385 - interface JoinPlan { 386 - type: 'hash' | 'merge' | 'nestedLoop' | 'lookup' 387 - left: StorePlan | JoinPlan // can be nested for multi-way 388 - right: StorePlan 389 - leftKey: string 390 - rightKey: string 391 - estimatedRows: number 392 - } 417 + // CRUD 418 + put(doc: T): Promise<void> 419 + putMany(docs: T[]): Promise<void> 420 + get(key: IDBValidKey): Promise<T | undefined> 421 + getMany(keys: IDBValidKey[]): Promise<Map<IDBValidKey, T>> 422 + delete(key: IDBValidKey): Promise<void> 393 423 394 - interface ExecutionPlan { 395 - root: StorePlan | JoinPlan 396 - finalProjection?: string[] 397 - finalSort?: OrderBy 398 - limit?: number 399 - } 424 + // Scanning (used by executor) 425 + scan(options: ScanOptions): AsyncGenerator<Row> 400 426 401 - class JoinPlanner { 402 - constructor(catalog: Catalog) // Catalog knows all stores and their stats 403 - 404 - plan(ast: QueryAST): Promise<ExecutionPlan> 427 + // Stats (used by planner) 428 + getStats(): StoreStats 405 429 } 406 430 ``` 407 431 408 - #### Cross-Store Self-Join (Lookup Join) 432 + > **Status:** CRUD operations exist as standalone functions: `put()`, `putMany()`, `get()`, `getMany()`, `del()`. Scanning via `rows()` generator. Stats via `StatsManager` class (separate from store). 409 433 410 - A critical pattern is the **cross-store self-join** — when a query uses a denormalized 411 - index store for filtering but needs fields from the primary store that aren't replicated. 434 + ### Denormalization ✅ 412 435 413 - **Example:** Query audio entries sorted by duration: 436 + Indexes with spread paths (e.g., `['...tags.tag', '...tags.value']`) create separate IDB object stores. On write, `put()` expands documents into the cartesian product of spread array elements: 414 437 415 438 ```typescript 416 - q.from('entries').whereMatch('tags', {tag: 'media-type', value: 'audio'}).orderBy('duration', 'desc') 439 + // Document 440 + { id: '1', title: 'Post', tags: [{ tag: 'a', value: '1' }, { tag: 'b', value: '2' }] } 441 + 442 + // Denormalized rows in entries_idx:tags store 443 + { _docId: '1', _tag: 'a', _value: '1', title: 'Post', ...replicated } 444 + { _docId: '1', _tag: 'b', _value: '2', title: 'Post', ...replicated } 417 445 ``` 418 446 419 - The denormalized index `entries_idx:tags` has the `by_tag` index for efficient filtering, 420 - but only replicates `[title, publishedAt, feedId]` — not `duration`. 447 + The index definition specifies which fields to replicate into denormalized rows. 421 448 422 - **Current problem:** The planner picks `entries_idx:tags` for filtering, sets `needsSort: true`, 423 - but the executor sorts on `undefined` values because `duration` isn't in the rows. 449 + > **Status:** Fully implemented via `ingest()` function. Index stores use compound primary key `[_docId, _rowId]`. Keys are translated via `toStorageKey()`/`fromStorageKey()` (e.g., `...tags.tag` → `_spread_tags_tag`). Cartesian product computed via `cartesianProduct()` when multiple spread roots exist. 450 + 451 + --- 452 + 453 + ## Schema & Index Definitions ✅ 424 454 425 - **Solution:** The Cross-Store Planner detects this pattern and produces a **lookup join**: 455 + Pure data definitions, no behavior: 426 456 427 457 ```typescript 428 - // Detected: orderBy.key ('duration') not in availableFields from entries_idx:tags 429 - // Solution: Self-join back to primary store 458 + interface SchemaDefinition<T> { 459 + name: string 460 + shape: ZodSchema<T> 461 + primaryKey: keyof T 462 + } 430 463 431 - { 432 - type: 'lookup', 433 - left: { 434 - // Filter side: use denormalized index 435 - source: 'entries_idx:tags', 436 - index: 'by_tag', 437 - range: { type: 'only', lower: ['media-type', 'audio'] }, 438 - // Only need docId for the join 439 - projection: ['_docId'] 440 - }, 441 - right: { 442 - // Lookup side: primary store has all fields 443 - source: 'primary', 444 - index: null, // lookup by primary key 445 - }, 446 - leftKey: '_docId', 447 - rightKey: 'id', 448 - // Now we have full docs, can sort properly 449 - finalSort: { key: 'duration', direction: 'desc' } 464 + interface IndexDefinition { 465 + name: string 466 + keys: string[] // supports '...spread.path' notation 467 + replicate: string[] // fields to copy into denormalized rows 468 + sparse: boolean 450 469 } 451 470 ``` 452 471 453 - **Execution pattern:** 472 + > **Status:** Implemented as `StoreDef<S>` and `IndexDef<S>`. Also tracks `spreadRoots: Set<string>`, groups indexes into `nativeIndexes` (on primary) and `indexStoreGroups` (denormalized stores). Supports `StorageRef` fields for external blob storage (schema-level, not index-related). 454 473 455 - 1. Scan `entries_idx:tags` with filter → collect `_docId` values 456 - 2. Batch lookup from `entries` primary store by IDs (use `getMany`) 457 - 3. Sort full documents by `duration` 458 - 4. Yield results 474 + Builder API: 475 + 476 + ```typescript 477 + const entries = defineStore('entries', { 478 + schema: z.object({ 479 + id: z.string(), 480 + feedId: z.string(), 481 + title: z.string(), 482 + publishedAt: z.date(), 483 + duration: z.number().optional(), 484 + readState: z.enum(['unread', 'read']), 485 + tags: z.array(z.object({tag: z.string(), value: z.string()})), 486 + }), 487 + primaryKey: 'id', 488 + indexes: [ 489 + index('by_feed_date').keys('feedId', 'publishedAt'), 490 + index('by_tags').keys('...tags.tag', '...tags.value').replicate('title', 'publishedAt', 'feedId'), 491 + ], 492 + }) 493 + ``` 459 494 460 - This is conceptually a join between two "stores" that happen to be the same logical 461 - entity (entries) — hence "cross-store self-join". The denormalized index store is 462 - treated as a separate store for planning purposes. 495 + > **Status:** Implemented via `docstore.define()` and `IndexBuilder` class. Use `.on()` for keys, `.replicate()` for replicated fields, `.sparse()` for sparse indexes. Type-safe `IndexKey<S>` constrains keys to indexable paths. 463 496 464 - **Detection heuristic in planner:** 497 + --- 498 + 499 + ## Storage Adapter ❌ 500 + 501 + Abstracts IndexedDB so the engine can be tested with in-memory storage or ported to other backends: 502 + 503 + > **Status:** Not implemented. Code directly uses `idb` library (`IDBPDatabase`). No `MemoryStorageAdapter` for testing - tests use real IndexedDB (via happy-dom or jsdom). 504 + > 505 + > **Design Note — Storage Adapter:** 506 + > Whether to add this abstraction depends on goals: 507 + > 508 + > - **For testing**: happy-dom's IDB works well, and testing against real IDB catches edge cases (key ordering, transaction semantics). An in-memory adapter risks hiding bugs. 509 + > - **For portability** (e.g., to SQLite/OPFS): Worth doing. The adapter interface should mirror IDB closely but allow different backends. 510 + > - **Minimal interface**: `Transaction`, `ObjectStore`, `Index` with `get/put/delete/getAll/getAllKeys/count`. The existing `KeyRangeSpec` type already abstracts `IDBKeyRange`. 511 + > 512 + > If added, inject adapter via `ExecutionContext` rather than changing all function signatures. The current direct `idb` usage is actually quite clean. 465 513 466 514 ```typescript 467 - // In StorePlanner or JoinPlanner 468 - if (plan.needsSort && orderBy) { 469 - const sortKeyAvailable = plan.availableFields.includes(orderBy.key) 470 - if (!sortKeyAvailable && plan.source !== 'primary') { 471 - // Need lookup join back to primary 472 - return wrapWithLookupJoin(plan, orderBy) 473 - } 515 + interface StorageAdapter { 516 + transaction(stores: string[], mode: 'readonly' | 'readwrite'): Transaction 517 + createStores(schemas: StoreSchema[]): Promise<void> 518 + } 519 + 520 + interface Transaction { 521 + store(name: string): ObjectStore 522 + done: Promise<void> 523 + } 524 + 525 + interface ObjectStore { 526 + get(key: IDBValidKey): Promise<unknown> 527 + put(value: unknown): Promise<void> 528 + delete(key: IDBValidKey): Promise<void> 529 + getAll(range?: KeyRange, count?: number): Promise<unknown[]> 530 + getAllKeys(range?: KeyRange, count?: number): Promise<IDBValidKey[]> 531 + count(range?: KeyRange): Promise<number> 532 + index(name: string): Index 474 533 } 534 + 535 + interface Index { 536 + getAll(range?: KeyRange, count?: number): Promise<unknown[]> 537 + getAllKeys(range?: KeyRange, count?: number): Promise<IDBValidKey[]> 538 + count(range?: KeyRange): Promise<number> 539 + } 540 + 541 + type KeyRange = 542 + | {type: 'only'; value: unknown} 543 + | {type: 'bound'; lower?: unknown; upper?: unknown; lowerOpen?: boolean; upperOpen?: boolean} 475 544 ``` 476 545 477 - ```` 546 + Implementations: `IDBStorageAdapter`, `MemoryStorageAdapter`. 547 + 548 + > **Status:** Neither implemented. Direct `idb` library usage throughout. 549 + 550 + --- 551 + 552 + ## Key Design Decisions 553 + 554 + - **Queries are combinators, not methods.** `where()`, `aggregate()`, `join()` are functions that compose stores and queries. Stores don't grow method surfaces. 478 555 479 - ### Layer 6: Query DSL 556 + > ❌ Not implemented. Current API is imperative (`QuerySpec` objects + `plan()` + `rows()`). 480 557 481 - **Goal:** Ergonomic, type-safe query builder that compiles to AST. 558 + - **No projection.** IDB reads whole documents regardless. Shaping output happens outside the query engine. `rows()` returns index rows, `docs()` returns full documents. 482 559 483 - ```typescript 484 - // dsl/query.ts 560 + > ✅ Implemented exactly as described. 485 561 486 - interface Query<S extends FieldShape, Selected = Infer<S>> { 487 - // Filtering 488 - where<K extends keyof S>( 489 - field: K, 490 - op: CompareOp, 491 - value: FieldValue<S[K]> 492 - ): Query<S, Selected> 562 + - **Spread match is an operator.** `['...tags', 'contains', { tag: 'x', value: 'y' }]` means "some array element matches all these fields." This is distinct from equality on a path. 493 563 494 - whereMatch(spread: string, match: Record<string, unknown>): Query<S, Selected> 564 + > ✅ Implemented via `SpreadMatch` interface in `QuerySpec.matches`. Uses explicit `{spread: 'tags', where: {tag: 'x', value: 'y'}}` syntax. 495 565 496 - // Joins 497 - join<T extends FieldShape>( 498 - store: SchemaDefinition<T>, 499 - leftKey: keyof S, 500 - rightKey: keyof T 501 - ): JoinedQuery<S, T, Selected> 566 + - **AsyncGenerator everywhere.** Native, composable, supports early termination. Observables are for live queries (a separate layer on top). 502 567 503 - leftJoin<T extends FieldShape>( 504 - store: SchemaDefinition<T>, 505 - leftKey: keyof S, 506 - rightKey: keyof T 507 - ): JoinedQuery<S, T, Selected> 568 + > ✅ `rows()`, `docs()`, `iterateRows()`, `iterateKeys()` are all `AsyncGenerator`s. No live query/observable layer yet. 508 569 509 - // Projection 510 - select<K extends keyof S>(...fields: K[]): Query<S, Pick<Infer<S>, K>> 570 + - **Denormalized indexes are separate stores.** The planner treats them as distinct data sources. A "lookup join" back to the primary store is just another join. 511 571 512 - // Ordering 513 - orderBy(field: keyof S, direction?: 'asc' | 'desc'): Query<S, Selected> 572 + > ✅ Index store groups (`IndexStoreGroup`) create separate IDB object stores named `{primary}_idx:{spreadRoots}`. `docs()` handles lookup join to primary. 514 573 515 - // Pagination 516 - limit(n: number): Query<S, Selected> 517 - skip(n: number): Query<S, Selected> 518 - cursor(c: PageCursor): Query<S, Selected> 574 + - **Sorting informs index selection.** The DSL captures `orderBy`, the planner tries to satisfy it with index order, falls back to sort operator + lookup join if needed. 519 575 520 - // Execution 521 - toAST(): QueryAST 522 - plan(): Promise<ExecutionPlan> 523 - execute(): AsyncGenerator<Selected> 524 - collect(): Promise<Selected[]> 525 - first(): Promise<Selected | undefined> 526 - count(): Promise<number> 527 - } 576 + > ✅ `QuerySpec.orderBy` informs planner. `candidate.coversOrderBy` scoring. `plan.needsSort` triggers in-memory sort. `plan.direction` set to 'prev' for desc when index covers it. 528 577 529 - // Entry point 530 - function from<S extends FieldShape>(store: IndexStore<S>): Query<S> 531 - ```` 578 + - **Stats drive planning.** HyperLogLog cardinality sketches and row counts inform index selection and join strategy. 579 + > ✅ `HyperLogLog` class for cardinality. `StatsManager` tracks per-store `IndexStatsRecord` with `rowCount`, `cardinalitySketch`, `bounds`. Used by `QueryPlanner` for selectivity estimation. Vacuum support for drift correction. 532 580 533 581 --- 534 582 535 - ## Key Design Decisions 583 + ## Build Order 536 584 537 - ### 1. AsyncGenerator Everywhere (not Observable) 585 + ### 1. Storage Adapter ❌ 586 + 587 + Start with the abstraction layer. Implement `MemoryStorageAdapter` first—it's simpler and lets you test everything without browser IDB. The interface mirrors IDB's API closely, so `IDBStorageAdapter` is mostly passthrough. 538 588 539 - The Observable layer adds complexity without clear benefit for query execution. AsyncGenerators: 589 + Key pieces: `Transaction`, `ObjectStore`, `Index`, `KeyRange` type and utilities for building/intersecting ranges. 540 590 541 - - Are native, no polyfill needed 542 - - Compose naturally with `for await` 543 - - Support early termination via `return()` 544 - - Work with existing `collect()`, `take()`, `skip()` utilities 591 + > **Status:** Skipped. Direct `idb` library usage. `KeyRangeSpec` exists for serializable key ranges, converted via `toIDBKeyRange()`. 545 592 546 - Keep Observable for **live queries** where you need push-based updates. 593 + ### 2. Schema & Index Definitions ✅ 547 594 548 - ### 2. Catalog as Central Registry 595 + Pure types and builders. `defineStore()` and `index()` produce `SchemaDefinition` and `IndexDefinition` objects. No runtime behavior yet—just data describing the shape of stores and their indexes. 549 596 550 - ```typescript 551 - interface Catalog { 552 - stores: Map<string, IndexStore> 597 + Include the path parsing logic for spread notation (`...tags.tag` -> `{ spread: 'tags', path: 'tag' }`). 553 598 554 - getStore(name: string): IndexStore 555 - getStats(name: string): StoreStats 599 + > **Status:** Complete. `docstore.define()`, `IndexBuilder`, `explodeKey()` for path parsing, `computeSpreadRoots()`, `groupIndexesBySpreadRoots()`. 556 600 557 - // For cross-store planning 558 - estimateJoinCardinality(leftStore: string, leftKey: string, rightStore: string, rightKey: string): number 559 - } 560 - ``` 601 + ### 3. Index Store ✅ 561 602 562 - The catalog is the bridge between stores and the planner. 603 + The core storage abstraction. Wraps a primary IDB store plus any denormalized index stores. 563 604 564 - ### 3. Predicate Representation 605 + Build in order: 565 606 566 - ```typescript 567 - type Predicate = 568 - | {type: 'eq'; field: string; value: unknown} 569 - | {type: 'neq'; field: string; value: unknown} 570 - | {type: 'lt' | 'lte' | 'gt' | 'gte'; field: string; value: unknown} 571 - | {type: 'in'; field: string; values: unknown[]} 572 - | {type: 'spreadMatch'; spread: string; match: Record<string, unknown>} 573 - | {type: 'and'; predicates: Predicate[]} 574 - | {type: 'or'; predicates: Predicate[]} 575 - ``` 607 + 1. Basic CRUD (`put`, `get`, `delete`) on primary store only ✅ 608 + 2. `scan()` with key ranges and direction ✅ (via `rows()`) 609 + 3. Denormalization logic—on `put()`, expand spread indexes into their stores ✅ (`ingest()` + index store writes) 610 + 4. `getMany()` for batch lookups ✅ 576 611 577 - This is the internal representation. The DSL compiles to this. 612 + Test with `MemoryStorageAdapter`. 578 613 579 - ### 4. SpreadMatch as First-Class Concept 614 + > **Status:** All implemented as standalone functions. No `IndexStore` class wrapper, but all functionality present. 580 615 581 - Your current `SpreadMatch` is great — it explicitly says "match these fields on ONE row of the spread array." Make it a core predicate type rather than a separate query path. 616 + ### 4. Stats ✅ 582 617 583 - ### 5. Separate Statistics Store 618 + `HyperLogLog` for cardinality estimation. `StatsStore` tracks per-store row counts and per-field cardinality sketches. Updated on writes, persisted to a `_stats` store. 584 619 585 - ```typescript 586 - // stats/stats-store.ts 587 - class StatsStore { 588 - // Per-store, per-field cardinality sketches 589 - private sketches: Map<string, Map<string, HyperLogLog>> 620 + This can be built independently and plugged into `IndexStore` later. 590 621 591 - // Row counts per store (primary + denormalized) 592 - private rowCounts: Map<string, number> 622 + > **Status:** Complete. `HyperLogLog` class (1024 registers, FNV-1a hash). `StatsManager` with `recordWrite()`, `recordWriteBatch()`, `recordDelete()`, `getCardinality()`, `getStats()`, `vacuum()`. Min/max bounds tracking. Idle-time vacuum scheduling via `requestIdleCallback`. 593 623 594 - // Bounds per field 595 - private bounds: Map<string, Map<string, {min: unknown; max: unknown}>> 624 + ### 5. Executor Operators ✅ 596 625 597 - // Async persist to IDB _stats store 598 - async flush(): Promise<void> 626 + Build the streaming operators bottom-up: 599 627 600 - // Background vacuum scheduling 601 - scheduleVacuum(storeId: string): void 602 - } 603 - ``` 628 + 1. `scan()` — wraps `IndexStore.scan()` ✅ (`rowsDirect()`) 629 + 2. `filter()` — predicate evaluation on rows ✅ (`matchesFilters()` inline) 630 + 3. `limit()` — early termination ✅ (`take()`) 631 + 4. `sort()` — materialize + sort (needed when index order doesn't match) ✅ (`rowsSortedDirect()`) 632 + 5. `intersect()` — collect doc IDs from multiple scans, yield intersection ✅ (`rowsIntersectDirect()`, `filterByIntersection()`) 633 + 6. `lookupJoin()` — batch lookup by key ✅ (`docs()` with batched fetches) 634 + 7. `hashJoin()` — build/probe pattern ✅ (`hashJoin()` in docstore-joins.ts) 635 + 8. `aggregateJoin()` — hash join with grouped aggregation ✅ (`hashJoinAggregate()` in docstore-joins.ts) 604 636 605 - --- 637 + Each operator is independently testable. 606 638 607 - ## File Structure 639 + > **Status:** All core operators implemented. Join operators are in separate module, work on arrays not generators. 608 640 609 - ``` 610 - src/lib/docstore/ 611 - ├── storage/ 612 - │ ├── adapter.ts # StorageAdapter interface 613 - │ ├── idb-adapter.ts # IDB implementation 614 - │ ├── memory-adapter.ts # In-memory for testing 615 - │ └── key-range.ts # KeyRange utilities 616 - 617 - ├── schema/ 618 - │ ├── types.ts # SchemaDefinition, IndexDefinition 619 - │ ├── builder.ts # defineStore(), defineIndex() 620 - │ ├── paths.ts # IndexablePath type magic 621 - │ ├── infer.ts # Type inference utilities 622 - │ └── storage-ref.ts # StorageRef for blobs 623 - 624 - ├── store/ 625 - │ ├── index-store.ts # IndexStore class 626 - │ ├── denormalize.ts # ingest() and cartesian product 627 - │ └── catalog.ts # Catalog registry 628 - 629 - ├── stats/ 630 - │ ├── hll.ts # HyperLogLog 631 - │ ├── stats-store.ts # StatsStore 632 - │ └── vacuum.ts # Background vacuum 633 - 634 - ├── executor/ 635 - │ ├── operators.ts # scan, filter, project, sort, limit 636 - │ ├── joins.ts # hashJoin, mergeJoin, nestedLoopJoin 637 - │ ├── intersect.ts # Set intersection for self-joins 638 - │ ├── execute.ts # Plan → RowStream 639 - │ └── page.ts # Keyset pagination 640 - 641 - ├── planner/ 642 - │ ├── predicates.ts # Predicate types and matching 643 - │ ├── store-planner.ts # Single-store optimization 644 - │ ├── join-planner.ts # Cross-store join planning 645 - │ └── cost.ts # Cost estimation utilities 646 - 647 - ├── dsl/ 648 - │ ├── query.ts # Query builder 649 - │ ├── ast.ts # QueryAST types 650 - │ └── compile.ts # AST → ExecutionPlan 651 - 652 - ├── live/ 653 - │ ├── live-query.ts # Observable-based live queries 654 - │ └── diff.ts # Result diffing for updates 655 - 656 - ├── analyzer.ts # Query tracing (keep as-is) 657 - └── index.ts # Public API exports 658 - ``` 641 + ### 6. Store-Level Planner ✅ 659 642 660 - --- 643 + Takes predicates + orderBy + index definitions + stats, produces `StorePlan`. 661 644 662 - ## Migration Path 645 + Build incrementally: 663 646 664 - 1. **Extract Schema** — Pull out `SchemaDefinition`, `IndexDefinition`, `IndexBuilder` into `schema/` 665 - 2. **Extract Storage Adapter** — Create `StorageAdapter` interface, wrap current IDB code 666 - 3. **Extract Stats** — Pull `HyperLogLog`, `StatsManager` into `stats/` 667 - 4. **Refactor IndexStore** — Encapsulate put/get/scan with adapter abstraction 668 - 5. **Extract Executor** — Pull operator functions into `executor/` 669 - 6. **Extract Planner** — Pull `QueryPlanner` into `planner/store-planner.ts` 670 - 7. **Add Join Planner** — New `planner/join-planner.ts` that uses `docstore-joins.ts` strategies 671 - 8. **Build DSL** — New `dsl/query.ts` that compiles to AST and executes 647 + 1. Simple index selection (first index that matches any predicate) ✅ 648 + 2. Predicate pushdown to key ranges ✅ (`#buildKeyRange()`, `#buildCompoundKeyRange()`) 649 + 3. Compound index prefix matching ✅ (`eqPrefixLength`) 650 + 4. Cost-based selection using stats ✅ (`#scoreCandidate()`, selectivity from HLL) 651 + 5. Intersection planning for multiple spread predicates ✅ (`#evaluateIntersection()`, `#evaluateSelfJoin()`, `#evaluateSpreadMatches()`) 652 + 6. `availableFields` tracking ✅ (`#getAvailableFields()`) 672 653 673 - Each step is independently testable. The current tests should keep passing as you refactor. 654 + > **Status:** Complete in `QueryPlanner` class. Also handles `IN` queries via cursor hopping (`inValues`/`inPrefix`). 674 655 675 - --- 656 + ### 7. Cross-Store Planner 🟡 676 657 677 - ## Wild Ideas (Bonus) 658 + Takes a `Query` (possibly a join), produces `ExecutionPlan`. 678 659 679 - ### Materialized Views 660 + Build incrementally: 680 661 681 - ```typescript 682 - const unreadByFeed = defineView('unread_by_feed', { 683 - source: entries, 684 - groupBy: 'feedId', 685 - aggregate: { 686 - count: count(), 687 - latest: max('publishedAt'), 688 - }, 689 - where: [{field: 'readState', op: '=', value: 'unread'}], 690 - }) 691 - ``` 662 + 1. Single-store queries (delegate to store planner, add sort/limit nodes) ✅ 663 + 2. Simple joins (hash join strategy) ❌ (manual via `docstore-joins.ts`) 664 + 3. Lookup join insertion when sort field unavailable ✅ (`docs()` handles) 665 + 4. Aggregate-join detection and planning ❌ 666 + 5. Join strategy selection based on cardinality ❌ 692 667 693 - Maintained incrementally on write. Query it like a store. 668 + > **Status:** Single-store planning complete. Cross-store joins are manual—user composes queries and join functions explicitly. 669 + > 670 + > **Design Note — Cross-Store Planner:** 671 + > To integrate joins into the planner: 672 + > 673 + > 1. **Join strategy selection**: Compare cardinalities from `StatsManager`: 674 + > - Left << Right → nested loop with batched lookup (like `docs()`) 675 + > - Left ≈ Right → hash join (materialize smaller side) 676 + > - Both ordered by join key → merge join (stream both) 677 + > 2. **Aggregate-join detection**: When the DSL template contains `aggregate()` calls, extract them during planning. The planner groups aggregates by their source/key and batches them into a single `AggregateJoinNode`. 678 + > 3. **Cost model**: `scanCost = rowCount * (indexMatch ? 0.01 : 1.0)`, `joinCost = leftRows + rightRows * hashFactor`. Pick plan with lowest total cost. 694 679 695 - ### Declarative Indexes from Queries 680 + ### 8. Query DSL ❌ 696 681 697 - ```typescript 698 - // Analyze query patterns, suggest indexes 699 - const suggestions = catalog.analyzeQueryPatterns(queryLog) 700 - // => "Consider adding index on [feedId, readState, publishedAt] - would improve 47% of queries" 701 - ``` 682 + The user-facing API. `where()`, `aggregate()`, `join()` build `Query<T>` objects that capture the query structure. `rows()` and `docs()` compile to a plan and execute. 702 683 703 - ### Query Plan Caching 684 + This is mostly glue—the hard work is in the planner and executor. 704 685 705 - ```typescript 706 - const prepared = query.prepare() // Plans once 707 - const results1 = await prepared.execute({feedId: 'f1'}) 708 - const results2 = await prepared.execute({feedId: 'f2'}) // Reuses plan 709 - ``` 686 + > **Status:** Not implemented. Current API is imperative: 687 + > 688 + > ```typescript 689 + > const plan = await planner.plan({store: 'entries', where: [...], orderBy: {...}}) 690 + > for await (const row of rows(ctx, plan)) { ... } 691 + > ``` 692 + > 693 + > DSL layer would wrap this in a fluent/combinator style. 694 + > 695 + > **Design Note — DSL Implementation:** 696 + > The DSL is a thin layer that builds `QuerySpec` objects and calls the existing planner/executor: 697 + > 698 + > ```typescript 699 + > class Query<T> { 700 + > #spec: QuerySpec 701 + > #planner: QueryPlanner 702 + > #ctx: ExecutionContext 703 + > 704 + > where(filter: Filter<T>): Query<T> { 705 + > // Clone spec, append to where array, return new Query 706 + > } 707 + > 708 + > async *rows(opts?: RowsOptions): AsyncGenerator<T> { 709 + > const plan = await this.#planner.plan(this.#spec) 710 + > yield* rows(this.#ctx, plan, opts) 711 + > } 712 + > } 713 + > 714 + > // Factory function 715 + > function from<S>(store: StoreDef<S>, ctx: ExecutionContext): Query<Infer<S>> 716 + > ``` 717 + > 718 + > Key: `Query<T>` is immutable—each combinator returns a new instance. This enables safe composition and potential plan caching (same spec → same plan). 710 719 711 720 --- 712 721 713 - What aspects of this would you like to dig into? I'm happy to: 722 + ## Future Considerations 723 + 724 + - **Live queries.** Observable layer on top of the generator-based execution. Cold observables that re-execute on relevant writes. 725 + 726 + > ❌ Not implemented. 727 + > 728 + > **Design Note:** Two approaches: 729 + > 730 + > 1. **Re-execution**: Track which stores/indexes a query touches. On write to those stores, invalidate and re-run. Simple but potentially expensive. 731 + > 2. **Differential**: For simple queries, compute delta from the write. E.g., for `where(entries, ['feedId', '=', 'f1'])`, a new entry with `feedId='f1'` can be appended without re-querying. 732 + > 733 + > Start with (1), optimize hot paths with (2). Use `WriteBatch` events from `put()`/`del()` to trigger invalidation. Return `Observable<T[]>` or `Observable<Delta<T>>` depending on use case. 734 + 735 + - **Plan caching.** Queries with parameters could cache the plan structure and just bind new values. Requires adding parameter placeholders to the DSL. 736 + 737 + > ❌ Not implemented. 738 + > 739 + > **Design Note:** The planner is already fast (<1ms for typical queries), so caching is mainly useful for: 740 + > 741 + > 1. **Parameterized queries**: `where(entries, ['feedId', '=', $param])` caches the plan structure, binds `$param` at execution time. Store plans in `WeakMap<QuerySpec, QueryPlan>` keyed by spec identity. 742 + > 2. **Prepared statements**: For hot paths, pre-plan and reuse. The plan's `keyRange` would need a "template" mode where values are bound later. 743 + > 744 + > Low priority—measure planning overhead first. If it becomes a bottleneck, the current planner outputs are already serializable (no closures). 714 745 715 - - Flesh out any layer in more detail 716 - - Prototype a specific piece 717 - - Discuss tradeoffs in the design 718 - - Think through the migration strategy 746 + - **Materialized views.** Pre-computed aggregates maintained incrementally on write. 719 747 720 - --- 748 + > ❌ Not implemented. 749 + > 750 + > **Design Note:** Materialized views are essentially "derived stores" that update on write: 751 + > 752 + > 1. **Definition**: `defineView('unreadCounts', aggregate(entries, ['feedId'], count()), {where: ['readState', '=', 'unread']})` 753 + > 2. **Storage**: Separate IDB store holding aggregated values keyed by group key 754 + > 3. **Incremental update**: On `put(entry)`: 755 + > - If entry was updated: decrement old group's count, increment new group's count 756 + > - If entry is new: increment group's count 757 + > - Requires tracking previous state or computing delta 758 + > 4. **Rebuild**: Periodic full recomputation to fix drift (similar to `StatsManager.vacuum()`) 759 + > 760 + > The existing `collectIndexValues()` is a non-materialized version of this. Materialization trades write cost for read speed. 761 + 762 + - **Query analysis.** Log query patterns, suggest missing indexes. 763 + > 🟡 `Analyzer` class exists for structured timing/tracing (open/close spans, print). No automatic index suggestion. 764 + > 765 + > **Design Note — Index Suggestion:** 766 + > The `Analyzer` already captures query execution details. To suggest indexes: 767 + > 768 + > 1. **Track slow queries**: Log queries where `plan.estimatedRows` >> actual rows returned (poor selectivity) or `plan.needsSort = true` (missing index for orderBy) 769 + > 2. **Identify patterns**: Collect `{store, whereKeys[], orderByKey}` tuples from executed queries. Group by pattern, count frequency. 770 + > 3. **Suggest indexes**: For frequent patterns with high scan counts: 771 + > - If `whereKeys` not covered by any index prefix → suggest `index.on(...whereKeys)` 772 + > - If `orderByKey` not covered → suggest adding to existing index or new index 773 + > - If spread key in where clause → suggest spread index with replication of filter/sort fields 774 + > 4. **Output**: `analyzer.suggestIndexes()` returns `IndexDef[]` that could improve observed query patterns 775 + > 776 + > Could integrate with `StatsManager` to estimate cardinalities for suggested indexes and rank by expected impact.
+463
src/lib/idbase/builder-scratch.ts
··· 1 + // todo: 2 + 3 + export type MigrationTx< 4 + ReadSchema extends SchemaConstraint, 5 + WriteSchema extends SchemaConstraint 6 + > = any 7 + 8 + ///// 9 + 10 + // todo: should eventually be "stuff that's serializable to idb" 11 + export type RowConstraint = Record<string | number | symbol, unknown> 12 + 13 + // store name -> document type 14 + export type SchemaConstraint = Record<string, RowConstraint> 15 + 16 + // generated index name -> keypaths 17 + // eg { idx_feedid_readat: readonly ['feedId', 'readAt'], ... } 18 + export type IndexesConstraint = Record<string, readonly string[]> 19 + 20 + /// 21 + 22 + const autogen = Symbol.for('autogenerate') 23 + 24 + type PrimaryKey<Row> = typeof autogen | keyof Row & string | (keyof Row & string)[] 25 + type ReplicableKey<Row> = IndexPath<Row> | keyof Row & string 26 + 27 + /// 28 + 29 + export interface DatabaseDef<Name extends string, Version extends number, Schema extends SchemaConstraint> { 30 + name: Name 31 + version: Version 32 + versions: VersionDef<any>[] 33 + 34 + // phantom types 35 + __schema: Schema 36 + } 37 + 38 + export interface VersionDef<Schema extends SchemaConstraint> { 39 + stores: Map<string, StoreDef<string, any, any>> 40 + migrations: MigrationDef[] 41 + 42 + // phantom types 43 + __schema: Schema 44 + } 45 + 46 + export type MigrationDef = 47 + | MigrationFn<any, any> 48 + | ['create', string, StoreDef<string, any>] 49 + | ['update', string, StoreDef<string, any>] 50 + | ['drop', string] 51 + 52 + export type MigrationFn< 53 + OldSchema extends SchemaConstraint, 54 + NewSchema extends SchemaConstraint 55 + > = ((tx: MigrationTx<OldSchema, NewSchema>) => Promise<void>) 56 + 57 + interface StoreDef< 58 + Name extends string, 59 + Row extends RowConstraint, 60 + Doc = Row, 61 + Indexes extends IndexesConstraint = IndexesConstraint 62 + > { 63 + name: Name 64 + pkey: PrimaryKey<Row> 65 + indexes: Map<keyof Indexes, IndexDef<Row>> 66 + codec?: CodecDef<Row, Doc> 67 + 68 + __row: Row 69 + __doc: Doc 70 + __indexes: Indexes 71 + } 72 + 73 + interface CodecDef<Row, Doc> { 74 + encode: (doc: Doc) => Row 75 + decode: (row: Row) => Doc 76 + } 77 + 78 + interface IndexDef< 79 + Row extends RowConstraint, 80 + Name extends string = string, 81 + Keys extends readonly string [] = IndexPath<Row>[] 82 + > { 83 + name: Name 84 + keys: Keys 85 + replicate: ReplicableKey<Row>[] 86 + roots: Set<keyof Row> 87 + } 88 + 89 + export type ExtractIndexName<T> = T extends IndexDef<any, infer N, any> ? N : never 90 + export type ExtractIndexKeys<T> = T extends IndexDef<any, any, infer K> ? K : never 91 + 92 + // builders 93 + 94 + export class DatabaseBuilder< 95 + Name extends string, 96 + Version extends number = 0, 97 + Schema extends SchemaConstraint = {} 98 + > { 99 + #name: Name 100 + #versions: VersionDef<any>[] 101 + 102 + constructor(name: Name) { 103 + this.#name = name 104 + this.#versions = [] 105 + } 106 + 107 + version<NewSchema extends SchemaConstraint>( 108 + cb: (builder: VersionBuilder<Schema>) => VersionDef<NewSchema> 109 + ): DatabaseBuilder<Name, Increment<Version>, NewSchema> { 110 + this.#versions.push(cb(new VersionBuilder<Schema>())) 111 + return this as unknown as DatabaseBuilder<Name, Increment<Version>, NewSchema> 112 + } 113 + 114 + build(): DatabaseDef<Name, Version, Schema> { 115 + return { 116 + name: this.#name, 117 + version: this.#versions.length as Version, 118 + versions: this.#versions, 119 + 120 + __schema: undefined as unknown as Schema 121 + } 122 + } 123 + } 124 + 125 + class VersionBuilder<Schema extends SchemaConstraint> { 126 + #stores: Map<string, StoreDef<string, any>> 127 + #migrations: MigrationDef[] 128 + 129 + constructor() { 130 + this.#stores = new Map() 131 + this.#migrations = [] 132 + } 133 + 134 + store<Row extends RowConstraint, Name extends string>( 135 + name: UnusedKey<Name, Schema>, 136 + cb: (builder: StoreBuilder<Name, Row>) => StoreDef<Name, Row> 137 + ): VersionBuilder<Schema & {[K in Name]: Row}> 138 + 139 + store<Row extends RowConstraint, Name extends string>( 140 + name: ExtantKey<Name, Schema>, 141 + cb: (builder: StoreUpdater<Name, Schema[Name], any, any>) => StoreDef<Name, Row> 142 + ): VersionBuilder<Schema & {[K in Name]: Row}> 143 + 144 + store<Name extends string>( 145 + name: ExtantKey<Name, Schema>, 146 + cb: 'drop' 147 + ): VersionBuilder<Omit<Schema, Name>> 148 + 149 + store<Row extends RowConstraint, Name extends string>( 150 + name: Name, 151 + cb: 'drop' | ((builder: any) => StoreDef<Name, Row>) 152 + ): VersionBuilder<any> { 153 + if (cb === 'drop') { 154 + return this.#dropStore(name) 155 + } 156 + else if (this.#stores.has(name)) { 157 + return this.#updateStore(name, cb) 158 + } 159 + else { 160 + return this.#createStore(name, cb) 161 + } 162 + } 163 + 164 + migrate<NewSchema extends SchemaConstraint>( 165 + cb: MigrationFn<Schema, NewSchema> 166 + ): VersionBuilder<NewSchema> { 167 + this.#migrations.push(cb) 168 + return this as unknown as VersionBuilder<NewSchema> 169 + } 170 + 171 + build(): VersionDef<Schema> { 172 + return { 173 + stores: this.#stores, 174 + migrations: this.#migrations, 175 + 176 + __schema: undefined as unknown as Schema 177 + } 178 + } 179 + 180 + #createStore<Row extends RowConstraint, Name extends string>( 181 + name: Name, 182 + cb: (builder: StoreBuilder<Name, Row>) => StoreDef<Name, Row> 183 + ): VersionBuilder<Schema & {[K in Name]: Row}> { 184 + if (this.#stores.has(name)) 185 + throw new Error(`cannot create store ${name} - already exists`) 186 + 187 + const created = cb(new StoreBuilder<Name, Row>(name)) 188 + this.#stores.set(name, created) 189 + this.#migrations.push(['create', name, created]) 190 + 191 + return this as unknown as VersionBuilder<Schema & {[K in Name]: Row}> 192 + } 193 + 194 + #updateStore<NewRow extends RowConstraint, Name extends string>( 195 + name: Name, 196 + cb: (u: StoreUpdater<Name, Schema[Name], any, any>) => StoreDef<Name, NewRow> 197 + ): VersionBuilder<Schema & {[K in Name]: NewRow}> { 198 + const extant = this.#stores.get(name as Name) 199 + if (extant == null) throw new Error(`cannot update store ${name} - does not exist`) 200 + 201 + const updated = cb(new StoreUpdater<Name, Schema[Name], typeof extant.__doc, typeof extant.__indexes>(extant as any)) 202 + this.#stores.set(name, updated) 203 + this.#migrations.push(['update', name, updated]) 204 + 205 + return this as unknown as VersionBuilder<Schema & {[K in Name]: NewRow}> 206 + } 207 + 208 + #dropStore<Name extends string>(name: Name): VersionBuilder<Omit<Schema, Name>> { 209 + this.#stores.delete(name) 210 + this.#migrations.push(['drop', name]) 211 + 212 + return this as unknown as VersionBuilder<Omit<Schema, Name>> 213 + } 214 + } 215 + 216 + class StoreBuilder< 217 + Name extends string, 218 + Row extends RowConstraint, 219 + Doc = Row, 220 + Indexes extends IndexesConstraint = {} 221 + > { 222 + #name: Name 223 + #pkey: PrimaryKey<Row> = autogen 224 + #indexes: Map<keyof Indexes, IndexDef<Row>> = new Map() 225 + #codec?: CodecDef<Row, Doc> 226 + 227 + constructor(name: Name) { 228 + this.#name = name 229 + } 230 + 231 + pkey(key: PrimaryKey<Row>): this { 232 + this.#pkey = key 233 + return this 234 + } 235 + 236 + codec<D>(codec: CodecDef<Row, D>): StoreBuilder<Name, Row, D, Indexes> { 237 + this.#codec = codec as any 238 + return this as unknown as StoreBuilder<Name, Row, D, Indexes> 239 + } 240 + 241 + index< 242 + IDef extends IndexDef<Row>, 243 + IName extends string = '<unnamed>', 244 + IndexName = AutoIndexName<IName, ExtractIndexKeys<IDef>>, 245 + > ( 246 + name: typeof autogen | UnusedKey<IName, Indexes>, 247 + cb: ((builder: IndexBuilder<Row, IName>) => IDef) 248 + ): StoreBuilder<Name, Row, Doc, Indexes & {[K in IndexName & string]: ExtractIndexKeys<IDef>}> { 249 + this.#indexes.set(name as any, cb(new IndexBuilder<Row, IName>(name))) 250 + return this as unknown as StoreBuilder<Name, Row, Doc, Indexes & {[K in IndexName & string]: ExtractIndexKeys<IDef>}> 251 + } 252 + 253 + build(): StoreDef<Name, Row, Doc, Indexes> { 254 + return { 255 + name: this.#name, 256 + pkey: this.#pkey, 257 + indexes: this.#indexes, 258 + codec: this.#codec, 259 + 260 + __row: undefined as unknown as Row, 261 + __doc: undefined as unknown as Doc, 262 + __indexes: undefined as unknown as Indexes 263 + } 264 + } 265 + } 266 + 267 + class StoreUpdater< 268 + Name extends string, 269 + Row extends RowConstraint, 270 + Doc, 271 + Indexes extends IndexesConstraint 272 + > { 273 + #extant: StoreDef<Name, Row, Doc, Indexes> 274 + #indexes: Map<keyof Indexes, IndexDef<Row>> 275 + #codec?: CodecDef<Row, Doc> 276 + 277 + constructor(extant: StoreDef<Name, Row, Doc, Indexes>) { 278 + this.#extant = extant 279 + this.#indexes = extant.indexes 280 + this.#codec = extant.codec 281 + } 282 + 283 + codec<D>(codec: CodecDef<Row, D>): StoreUpdater<Name, Row, D, Indexes> { 284 + this.#codec = codec as any 285 + return this as unknown as StoreUpdater<Name, Row, D, Indexes> 286 + } 287 + 288 + index< 289 + IDef extends IndexDef<Row>, 290 + IName extends string = '<unnamed>', 291 + IndexName = AutoIndexName<IName, ExtractIndexKeys<IDef>>, 292 + > ( 293 + name: typeof autogen | UnusedKey<IName, Indexes>, 294 + cb: ((builder: IndexBuilder<Row, IName>) => IDef) 295 + ): StoreUpdater<Name, Row, Doc, Indexes & {[K in IndexName & string]: ExtractIndexKeys<IDef>}> { 296 + this.#indexes.set(name as any, cb(new IndexBuilder<Row, IName>(name))) 297 + return this as unknown as StoreUpdater<Name, Row, Doc, Indexes & {[K in IndexName & string]: ExtractIndexKeys<IDef>}> 298 + } 299 + 300 + dropIndex<IName extends string>(name: ExtantKey<IName, Indexes>): StoreUpdater<Name, Row, Doc, Omit<Indexes, IName>> { 301 + this.#indexes.delete(name) 302 + return this as unknown as StoreUpdater<Name, Row, Doc, Omit<Indexes, IName>> 303 + } 304 + 305 + build(): StoreDef<Name, Row, Doc, Indexes> { 306 + return { 307 + ...this.#extant, 308 + indexes: this.#indexes, 309 + codec: this.#codec, 310 + 311 + __row: undefined as unknown as Row, 312 + __doc: undefined as unknown as Doc, 313 + __indexes: undefined as unknown as Indexes 314 + } 315 + } 316 + } 317 + 318 + class IndexBuilder< 319 + Row extends RowConstraint, 320 + Name extends typeof autogen | string, 321 + Keys extends readonly IndexPath<Row>[] = [] 322 + > { 323 + #name: typeof autogen | string 324 + #keys: Keys 325 + #replicate: ReplicableKey<Row>[] 326 + 327 + constructor(name: typeof autogen | string = autogen) { 328 + this.#name = name 329 + this.#keys = [] as unknown as Keys 330 + this.#replicate = [] 331 + } 332 + 333 + on<K extends readonly IndexPath<Row>[]>(...keys: K): IndexBuilder<Row, Name, K> { 334 + const self = this as unknown as IndexBuilder<Row, Name, K> 335 + 336 + self.#keys = keys 337 + return self 338 + } 339 + 340 + replicate(...keys: ReplicableKey<Row>[]): this { 341 + this.#replicate = keys 342 + return this 343 + } 344 + 345 + build(): IndexDef<Row, AutoIndexName<Name, Keys> & string, Keys> { 346 + if (this.#keys.length === 0) 347 + throw new Error('no keys specified!') 348 + 349 + const roots = computeIndexRoots(this.#keys) 350 + const name = this.#name === autogen 351 + ? computeIndexName(this.#keys) 352 + : this.#name 353 + 354 + return { 355 + name: name as AutoIndexName<Name, Keys> & string, 356 + keys: this.#keys, 357 + replicate: this.#replicate, 358 + roots 359 + } 360 + } 361 + } 362 + 363 + type ReplaceDotsWithUnderscores<S extends string> = 364 + S extends `${infer Head}.${infer Tail}` 365 + ? `${Head}_${ReplaceDotsWithUnderscores<Tail>}` 366 + : S 367 + 368 + type KeyToNameSegment<K extends string> = 369 + K extends `...${infer Rest}` 370 + ? `multi_${ReplaceDotsWithUnderscores<Rest>}` 371 + : ReplaceDotsWithUnderscores<K> 372 + 373 + type JoinWithUnderscore<T extends readonly string[]> = 374 + T extends readonly [infer First extends string] 375 + ? KeyToNameSegment<First> 376 + : T extends readonly [infer First extends string, ...infer Rest extends readonly string[]] 377 + ? `${KeyToNameSegment<First>}_${JoinWithUnderscore<Rest>}` 378 + : '' 379 + 380 + type ComputedIndexName<Keys extends readonly string[]> = 381 + `idx_${JoinWithUnderscore<Keys>}` 382 + 383 + type AutoIndexName<Name extends typeof autogen | string, Keys extends readonly string[]> = 384 + Name extends typeof autogen 385 + ? ComputedIndexName<Keys> 386 + : Name 387 + 388 + function explodeIndexKey(key: string) { 389 + const spread = key.startsWith('...') 390 + const path = key.slice(spread ? 3 : 0).split('.') 391 + return { 392 + spread, root: path[0], path 393 + } 394 + } 395 + 396 + function computeIndexRoots(keys: readonly string[]): Set<string> { 397 + const roots = new Set<string>() 398 + for (const key of keys) { 399 + const {spread, root} = explodeIndexKey(key) 400 + if (spread && root) { 401 + roots.add(root) 402 + } 403 + } 404 + 405 + return roots 406 + } 407 + 408 + function computeIndexName<Keys extends readonly string[]>(keys: Keys): ComputedIndexName<Keys> { 409 + const segments = keys.map((k: string) => k 410 + .replaceAll('...', 'multi_') 411 + .replaceAll('.', '_') 412 + ) 413 + 414 + return `idx_${segments.join('_')}` as ComputedIndexName<Keys> 415 + } 416 + 417 + /////// 418 + // fun types 419 + 420 + type Increment<N extends number> = [ 421 + 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 422 + 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 423 + 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 424 + 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 425 + 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 426 + 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 427 + 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 428 + 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 429 + 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 430 + 91, 92, 93, 94, 95, 96, 97, 98, 99, 100 431 + ][N] 432 + 433 + type UnusedKey<K extends string, T> = 434 + K & (K extends keyof T ? never : K) 435 + 436 + type ExtantKey<K extends string, T> = 437 + K & (K extends keyof T ? K : never) 438 + 439 + /** returns a path extended with a dot */ 440 + type ExtendPath<Path extends string, Next extends string> = Path extends '' ? Next : `${Path}.${Next}` 441 + 442 + /** primitives that idb can index */ 443 + export type Indexable = string | number | Date | boolean 444 + 445 + /** recursive path into an object. */ 446 + export type IndexPath<T, Path extends string = '', Spread extends boolean = false> = 447 + string & T extends Indexable 448 + ? Spread extends true ? `...${Path}` : Path 449 + : T extends Array<infer O> 450 + ? IndexPath<O, Path, true> 451 + : T extends Map<infer K, infer V> 452 + ? | IndexPath<K, ExtendPath<Path, 'key'>, true> 453 + | IndexPath<V, ExtendPath<Path, 'value'>, true> 454 + : T extends object 455 + ? { 456 + [P in keyof T]: 457 + IndexPath< 458 + NonNullable<T[P]>, // strip null/undefined, as it's not part of the path 459 + ExtendPath<Path, P & string>, 460 + Spread 461 + > 462 + }[keyof T] 463 + : never
+494
src/lib/idbase/builder.ts
··· 1 + /** 2 + * Schema Builder DSL - Type Sketch 3 + * 4 + * This file sketches out the full DSL for defining versioned IndexedDB schemas 5 + * with type-safe migrations. No implementation yet, just types and signatures. 6 + */ 7 + 8 + import type {IndexPath, ReplicablePath} from './schema' 9 + 10 + // ============================================================================= 11 + // Core Type Definitions 12 + // ============================================================================= 13 + 14 + /** Map of store names to their document types */ 15 + type SchemaMap = Record<string, unknown> 16 + 17 + /** IDB-compatible primary key types */ 18 + type ValidKey = string | number | Date | ArrayBufferView | ArrayBuffer | IDBValidKey[] 19 + 20 + // ============================================================================= 21 + // Index Builder (reusing concepts from schema-index.ts) 22 + // ============================================================================= 23 + 24 + type ReplaceDotsWithUnderscores<S extends string> = 25 + S extends `${infer Head}.${infer Tail}` 26 + ? `${Head}_${ReplaceDotsWithUnderscores<Tail>}` 27 + : S 28 + 29 + type KeyToNameSegment<K extends string> = 30 + K extends `...${infer Rest}` 31 + ? `multi_${ReplaceDotsWithUnderscores<Rest>}` 32 + : ReplaceDotsWithUnderscores<K> 33 + 34 + type JoinWithUnderscore<T extends readonly string[]> = 35 + T extends readonly [infer First extends string] 36 + ? KeyToNameSegment<First> 37 + : T extends readonly [infer First extends string, ...infer Rest extends readonly string[]] 38 + ? `${KeyToNameSegment<First>}_${JoinWithUnderscore<Rest>}` 39 + : '' 40 + 41 + type ComputedIndexName<Keys extends readonly string[]> = 42 + `idx_${JoinWithUnderscore<Keys>}` 43 + 44 + type AutoIndexName<Name extends string, Keys extends readonly string[]> = 45 + Name extends '<unnamed>' 46 + ? ComputedIndexName<Keys> 47 + : Name 48 + 49 + function computeIndexName<Keys extends readonly string[]>(keys: Keys): ComputedIndexName<Keys> { 50 + const segments = keys.map((k: string) => k 51 + .replaceAll('...', 'multi_') 52 + .replaceAll('.', '_') 53 + ) 54 + 55 + return `idx_${segments.join('_')}` as ComputedIndexName<Keys> 56 + } 57 + 58 + function explodeKey(key: string) { 59 + const spread = key.startsWith('...') 60 + const path = key.slice(spread ? 3 : 0).split('.') 61 + return { 62 + spread, root: path[0], path 63 + } 64 + } 65 + 66 + function computeRoots(keys: readonly string[]): Set<string> { 67 + const roots = new Set<string>() 68 + for (const key of keys) { 69 + const {spread, root} = explodeKey(key) 70 + if (spread && root) { 71 + roots.add(root) 72 + } 73 + } 74 + 75 + return roots 76 + } 77 + 78 + // 79 + 80 + export interface IndexDef< 81 + Row, 82 + Name extends string = string, 83 + Keys extends readonly string[] = readonly IndexPath<Row>[] 84 + > { 85 + name: Name 86 + keys: Keys 87 + replicate: ReplicablePath<Row>[] 88 + roots: Set<string> 89 + } 90 + 91 + export class IndexBuilder< 92 + Row, 93 + Name extends string = '<unnamed>', 94 + Keys extends readonly IndexPath<Row>[] = [] 95 + > { 96 + declare __name: Name 97 + declare __keys: Keys 98 + 99 + #name?: Name 100 + #keys?: Keys 101 + #replicate?: ReplicablePath<Row>[] 102 + 103 + constructor(name?: Name) { 104 + this.#name = name 105 + } 106 + 107 + on<K extends readonly IndexPath<Row>[]>(...keys: K): IndexBuilder<Row, Name, K> { 108 + this.#keys = keys as unknown as Keys 109 + return this as unknown as IndexBuilder<Row, Name, K> 110 + } 111 + 112 + replicate(...keys: ReplicablePath<Row>[]): this { 113 + this.#replicate = keys 114 + return this 115 + } 116 + 117 + build(): IndexDef<Row, AutoIndexName<Name, Keys>, Keys> { 118 + if (this.#keys === undefined) 119 + throw new Error('no keys specified!') 120 + 121 + const name = (this.#name ?? computeIndexName<Keys>(this.#keys)) as AutoIndexName<Name, Keys> 122 + const roots = computeRoots(this.#keys) 123 + 124 + return { 125 + name, 126 + keys: this.#keys, 127 + replicate: this.#replicate ?? [], 128 + roots 129 + } 130 + } 131 + } 132 + 133 + // ============================================================================= 134 + // Store Builder (for new stores in addStore) 135 + // ============================================================================= 136 + 137 + /** 138 + * Indexes phantom type: a record from index name to its key paths tuple 139 + * e.g. { idx_feedId_readAt: readonly ['feedId', 'readAt'], idx_multi_tags_tag: readonly ['...tags.tag'] } 140 + */ 141 + type StoreIndexesConstraint = Record<string, readonly string[]> 142 + 143 + /** Extract name from an IndexDef */ 144 + type ExtractIndexName<T> = T extends IndexDef<any, infer N, any> ? N : never 145 + 146 + /** Extract keys from an IndexDef */ 147 + type ExtractIndexKeys<T> = T extends IndexDef<any, any, infer K> ? K : never 148 + 149 + /** Build a record from a single index def */ 150 + type IndexToRecord<T> = { 151 + [K in ExtractIndexName<T>]: ExtractIndexKeys<T> 152 + } 153 + 154 + /** Recursively merge index records from a tuple */ 155 + type MergeIndexRecords<T extends readonly unknown[]> = T extends readonly [infer First, ...infer Rest] 156 + ? IndexToRecord<First> & MergeIndexRecords<Rest> 157 + : {} 158 + 159 + /** Convert an array of index builders to a record */ 160 + type IndexArrayToRecord<T extends readonly unknown[]> = 161 + MergeIndexRecords<T> extends infer R ? {[K in keyof R]: R[K]} : never 162 + 163 + type CodecFns<Row, Doc> = { 164 + encode: (value: Doc) => Row 165 + decode: (value: Row) => Doc 166 + } 167 + 168 + export interface StoreDef< 169 + Name extends string, 170 + Doc, 171 + Row = Doc, 172 + Indexes extends StoreIndexesConstraint = StoreIndexesConstraint, 173 + > { 174 + name: Name 175 + pkey: IndexPath<Row> | IndexPath<Row>[] 176 + indexes: IndexDef<Row, string, readonly string[]>[] 177 + encode?: (value: Doc) => Row 178 + decode?: (row: Row) => Doc 179 + 180 + // phantom types for extraction 181 + __row: Row 182 + __doc: Doc 183 + __indexes: Indexes 184 + } 185 + 186 + export class StoreBuilder< 187 + Name extends string, 188 + Doc, 189 + Row = Doc, 190 + Indexes extends StoreIndexesConstraint = {} 191 + > { 192 + declare __doc: Doc 193 + declare __row: Row 194 + declare __indexes: Indexes 195 + 196 + #name: Name 197 + #pkey: IndexPath<Row> | IndexPath<Row>[] 198 + #indexes?: IndexDef<Row>[] 199 + #encoder?: (value: Doc) => Row 200 + #decoder?: (value: Row) => Doc 201 + 202 + constructor(name: Name, pkey: IndexPath<Row> | IndexPath<Row>[]) { 203 + this.#name = name 204 + this.#pkey = pkey 205 + } 206 + 207 + codec<R>(fns: CodecFns<R, Doc>) { 208 + this.#encoder = fns.encode as any 209 + this.#decoder = fns.decode as any 210 + 211 + return this as unknown as StoreBuilder<Name, Doc, R, Indexes> 212 + } 213 + 214 + indexes<const T extends readonly IndexDef<Row>[]>( 215 + fn: (maker: <N extends string = string>(name?: N) => IndexBuilder<Row, N, readonly []>) => T 216 + ) { 217 + const maker = <N extends string = string> (name?: N) => new IndexBuilder<Row, N>(name) 218 + this.#indexes = fn(maker) as unknown as IndexDef<Row>[] 219 + 220 + return this as unknown as StoreBuilder<Name, Doc, Row, IndexArrayToRecord<T>> 221 + } 222 + 223 + build(): StoreDef<Name, Doc, Row, Indexes> { 224 + return { 225 + name: this.#name, 226 + pkey: this.#pkey, 227 + indexes: this.#indexes ?? [], 228 + encode: this.#encoder, 229 + decode: this.#decoder, 230 + 231 + __doc: undefined as unknown as Doc, 232 + __row: undefined as unknown as Row, 233 + __indexes: undefined as unknown as Indexes 234 + } 235 + } 236 + } 237 + 238 + // ============================================================================= 239 + // Store Amender (for existing stores in mutateStore) 240 + // ============================================================================= 241 + 242 + declare class StoreAmender<T, Row = T> { 243 + /** Add an index to the store */ 244 + addIndex<N extends string, K extends readonly IndexPath<Row>[]>( 245 + fn: (idx: IndexBuilder<Row, N, readonly []>) => IndexBuilder<Row, N, K>, 246 + ): this 247 + 248 + /** Drop an index by name */ 249 + dropIndex(name: string): this 250 + } 251 + 252 + // ============================================================================= 253 + // Migration Transaction & Store 254 + // ============================================================================= 255 + 256 + // TODO: replace with a specialization of the IDB ObjectStore 257 + 258 + /** A store handle during migration - reads return Old type, writes expect New type */ 259 + interface MigrationStore<Old, New> { 260 + get(key: ValidKey): Promise<Old | undefined> 261 + getAll(): Promise<Old[]> 262 + getAllKeys(): Promise<ValidKey[]> 263 + 264 + put(value: New, key?: ValidKey): Promise<ValidKey> 265 + add(value: New, key?: ValidKey): Promise<ValidKey> 266 + delete(key: ValidKey): Promise<void> 267 + clear(): Promise<void> 268 + 269 + // Cursor iteration - yields old type, put expects new type 270 + iterate(): AsyncIterable<{ 271 + key: ValidKey 272 + value: Old 273 + update(value: New): Promise<void> 274 + delete(): Promise<void> 275 + }> 276 + } 277 + 278 + /** 279 + * Migration transaction type. 280 + * 281 + * - Schema: the current schema (types as they exist before this migration) 282 + * - WriteOverrides: stores being migrated, mapping to their NEW type 283 + * 284 + * objectStore() returns MigrationStore<OldType, NewType> where: 285 + * - OldType comes from Schema 286 + * - NewType comes from WriteOverrides (or falls back to Schema if not overridden) 287 + */ 288 + type MigrationTx<Schema extends SchemaMap, WriteOverrides extends Partial<Schema> = {}> = { 289 + objectStore<K extends keyof Schema & string>( 290 + name: K, 291 + ): MigrationStore<Schema[K], K extends keyof WriteOverrides ? WriteOverrides[K] : Schema[K]> 292 + } 293 + 294 + // ============================================================================= 295 + // Version Builder 296 + // ============================================================================= 297 + 298 + declare class VersionBuilder<Schema extends SchemaMap> { 299 + /** 300 + * Add a new store to the database. 301 + * Only valid if the store doesn't already exist in the schema. 302 + */ 303 + addStore<T, Name extends string>( 304 + name: Name & (Name extends keyof Schema ? never : Name), 305 + configure: (builder: StoreBuilder<T>) => StoreBuilder<T, any>, 306 + ): VersionBuilder<Schema & {[K in Name]: T}> 307 + 308 + /** 309 + * Mutate an existing store (add/drop indexes). 310 + * Only valid if the store already exists in the schema. 311 + */ 312 + mutateStore<Name extends keyof Schema & string>( 313 + name: Name, 314 + configure: (amender: StoreAmender<Schema[Name]>) => StoreAmender<Schema[Name], any>, 315 + ): VersionBuilder<Schema> 316 + 317 + /** 318 + * Drop a store from the database. 319 + * Removes it from the schema, allowing re-creation with addStore. 320 + */ 321 + dropStore<Name extends keyof Schema & string>(name: Name): VersionBuilder<Omit<Schema, Name>> 322 + 323 + /** 324 + * Run a data migration. 325 + * 326 + * WriteOverrides specifies which stores are being migrated and their NEW types. 327 + * After this migration, those stores will have the new types in the schema. 328 + * 329 + * The migration function receives a transaction where: 330 + * - Reads return the OLD type (from current schema) 331 + * - Writes expect the NEW type (from WriteOverrides) 332 + */ 333 + migrate<WriteOverrides extends Partial<Schema>>( 334 + fn: (tx: MigrationTx<Schema, WriteOverrides>) => void | Promise<void>, 335 + ): VersionBuilder<Omit<Schema, keyof WriteOverrides> & WriteOverrides> 336 + } 337 + 338 + // ============================================================================= 339 + // Database Builder 340 + // ============================================================================= 341 + 342 + declare class DatabaseBuilder<Schema extends SchemaMap = {}> { 343 + /** 344 + * Define a version of the database. 345 + * 346 + * Each version receives the schema from the previous version and can: 347 + * - Add new stores 348 + * - Mutate existing stores (add/drop indexes) 349 + * - Drop stores 350 + * - Run data migrations 351 + * 352 + * The returned schema becomes the input for the next version. 353 + */ 354 + version<NewSchema extends SchemaMap>( 355 + v: number, 356 + fn: (vb: VersionBuilder<Schema>) => VersionBuilder<NewSchema>, 357 + ): DatabaseBuilder<NewSchema> 358 + } 359 + 360 + // ============================================================================= 361 + // Database Definition (output of the builder) 362 + // ============================================================================= 363 + 364 + interface DatabaseDef<Schema extends SchemaMap> { 365 + name: string 366 + version: number 367 + schema: Schema 368 + // ... runtime stuff for opening, migrations, etc. 369 + } 370 + 371 + // ============================================================================= 372 + // Public API 373 + // ============================================================================= 374 + 375 + declare function db<Schema extends SchemaMap>( 376 + name: string, 377 + fn: (builder: DatabaseBuilder<{}>) => DatabaseBuilder<Schema>, 378 + ): DatabaseDef<Schema> 379 + 380 + // ============================================================================= 381 + // Example Usage 382 + // ============================================================================= 383 + 384 + // --- Version 1 types --- 385 + type EntityV1 = { 386 + id: string 387 + name: string 388 + createdAt: Date 389 + } 390 + 391 + type Feed = { 392 + id: string 393 + url: string 394 + title: string 395 + } 396 + 397 + // --- Version 2 types --- 398 + type EntityV2 = { 399 + id: string 400 + name: string 401 + createdAt: Date 402 + updatedAt: Date // new field 403 + tags: string[] // new field 404 + } 405 + 406 + // --- Version 3 types --- 407 + type EntityV3 = { 408 + id: string 409 + name: string 410 + createdAt: number // changed from Date to timestamp 411 + updatedAt: number // changed from Date to timestamp 412 + tags: string[] 413 + } 414 + 415 + // --- Schema Definition --- 416 + const myDb = db('myapp', (builder) => 417 + builder 418 + // Version 1: Initial schema 419 + .version(1, (v) => 420 + v 421 + .addStore<EntityV1>('entities', (s) => 422 + s.pkey('id').indexes((idx) => [idx().on('createdAt'), idx().on('name')]), 423 + ) 424 + .addStore<Feed>('feeds', (s) => s.pkey('id').indexes((idx) => [idx().on('url').unique()])), 425 + ) 426 + 427 + // Version 2: Add fields to Entity, add index 428 + .version(2, (v) => 429 + v 430 + .mutateStore('entities', (s) => s.addIndex((idx) => idx().on('...tags'))) 431 + .migrate<{entities: EntityV2}>(async (tx) => { 432 + const store = tx.objectStore('entities') 433 + // store.get() returns EntityV1 434 + // store.put() expects EntityV2 435 + for await (const cursor of store.iterate()) { 436 + cursor.update({ 437 + ...cursor.value, 438 + updatedAt: cursor.value.createdAt, 439 + tags: [], 440 + }) 441 + } 442 + }), 443 + ) 444 + 445 + // Version 3: Change Date to timestamp 446 + .version(3, (v) => 447 + v.migrate<{entities: EntityV3}>(async (tx) => { 448 + const store = tx.objectStore('entities') 449 + // store.get() returns EntityV2 450 + // store.put() expects EntityV3 451 + for await (const cursor of store.iterate()) { 452 + cursor.update({ 453 + ...cursor.value, 454 + createdAt: cursor.value.createdAt.getTime(), 455 + updatedAt: cursor.value.updatedAt.getTime(), 456 + }) 457 + } 458 + }), 459 + ) 460 + 461 + // Version 4: Drop a store, recreate with different pkey 462 + .version(4, (v) => 463 + v.dropStore('feeds').addStore<Feed>( 464 + 'feeds', 465 + (s) => s.pkey('url'), // different primary key now 466 + ), 467 + ), 468 + ) 469 + 470 + // ============================================================================= 471 + // Type Tests (these will error until implementation exists) 472 + // ============================================================================= 473 + 474 + // Once implemented, these should work: 475 + // 476 + // type _FinalSchema = typeof myDb.schema 477 + // type _Entities = _FinalSchema['entities'] // should be EntityV3 478 + // type _Feeds = _FinalSchema['feeds'] // should be Feed 479 + // 480 + // const _testEntity: _Entities = { 481 + // id: '1', 482 + // name: 'test', 483 + // createdAt: 123456, // number, not Date 484 + // updatedAt: 123456, 485 + // tags: ['a', 'b'], 486 + // } 487 + // 488 + // const _testFeed: _Feeds = { 489 + // id: '1', 490 + // url: 'https://example.com', 491 + // title: 'Example', 492 + // } 493 + 494 + void myDb
+1
src/lib/idbase/index.ts
··· 1 + // hello
src/lib/idbase/schema-blobref.ts

This is a binary file and will not be displayed.

+545
src/lib/idbase/schema-builder.ts
··· 1 + import { 2 + openDB, 3 + type DBSchema, 4 + type IDBPDatabase, 5 + type IDBPTransaction, 6 + type IndexNames, 7 + type StoreNames, 8 + } from 'idb' 9 + import {IndexBuilder, type IndexDef} from './schema-index' 10 + 11 + type StringKeys<S> = keyof S & string 12 + 13 + // ───────────────────────────────────────────────────────────────────────────── 14 + // Store Definition & Builder 15 + // ───────────────────────────────────────────────────────────────────────────── 16 + 17 + /** 18 + * Indexes phantom type: a record from index name to its key paths tuple 19 + * e.g. { idx_feedId_readAt: readonly ['feedId', 'readAt'], idx_multi_tags_tag: readonly ['...tags.tag'] } 20 + */ 21 + type IndexesConstraint = Record<string, readonly string[]> 22 + 23 + export interface StoreDef< 24 + Name extends string, 25 + Doc, 26 + Row = Doc, 27 + Indexes extends IndexesConstraint = IndexesConstraint, 28 + > { 29 + name: Name 30 + pkey: StringKeys<Row> | StringKeys<Row>[] 31 + indexes: IndexDef<Row, string, readonly string[]>[] 32 + encode?: (value: Doc) => Row 33 + decode?: (row: Row) => Doc 34 + 35 + // phantom types for extraction 36 + __row: Row 37 + __doc: Doc 38 + __indexes: Indexes 39 + } 40 + 41 + /** Extract name from an IndexBuilder or IndexDef */ 42 + type ExtractIndexName<T> = 43 + T extends IndexBuilder<any, infer N, any> ? N : T extends IndexDef<any, infer N, any> ? N : never 44 + 45 + /** Extract keys from an IndexBuilder or IndexDef */ 46 + type ExtractIndexKeys<T> = 47 + T extends IndexBuilder<any, any, infer K> ? K : T extends IndexDef<any, any, infer K> ? K : never 48 + 49 + /** Build a record from a single index builder/def */ 50 + type IndexToRecord<T> = { 51 + [K in ExtractIndexName<T>]: ExtractIndexKeys<T> 52 + } 53 + 54 + /** Recursively merge index records from a tuple */ 55 + type MergeIndexRecords<T extends readonly unknown[]> = T extends readonly [infer First, ...infer Rest] 56 + ? IndexToRecord<First> & MergeIndexRecords<Rest> 57 + : {} 58 + 59 + /** Convert an array of index builders to a record */ 60 + type IndexArrayToRecord<T extends readonly unknown[]> = 61 + MergeIndexRecords<T> extends infer R ? {[K in keyof R]: R[K]} : never 62 + 63 + /** Base type for IndexBuilder/IndexDef that allows any keys (used in constraints) */ 64 + type AnyIndexBuilder<Row> = IndexBuilder<Row, string, readonly string[]> 65 + type AnyIndexDef<Row> = IndexDef<Row, string, readonly string[]> 66 + 67 + export class StoreBuilder<Name extends string, Doc, Row = Doc, Indexes extends IndexesConstraint = {}> { 68 + readonly name: Name 69 + 70 + declare __doc: Doc 71 + declare __row: Row 72 + declare __indexes: Indexes 73 + 74 + #pkey: string | string[] 75 + #indexes?: AnyIndexDef<Row>[] 76 + #encoder?: (value: Doc) => Row 77 + #decoder?: (row: Row) => Doc 78 + 79 + constructor(name: Name, pkey: string | string[]) { 80 + this.name = name 81 + this.#pkey = pkey 82 + } 83 + 84 + codec<R>(fns: {encode: (value: Doc) => R; decode: (row: R) => Doc}): StoreBuilder<Name, Doc, R, {}> { 85 + this.#encoder = fns.encode as any 86 + this.#decoder = fns.decode as any 87 + return this as unknown as StoreBuilder<Name, Doc, R, {}> 88 + } 89 + 90 + indexes<const T extends readonly (AnyIndexDef<Row> | AnyIndexBuilder<Row>)[]>( 91 + cb: (maker: <N extends string = string>(name?: N) => IndexBuilder<Row, N, readonly []>) => T, 92 + ): StoreBuilder<Name, Doc, Row, IndexArrayToRecord<T>> { 93 + const defs = cb(<N extends string = string>(name?: N) => new IndexBuilder<Row, N, readonly []>(name)).map( 94 + (v) => (v instanceof IndexBuilder ? v.build() : v) as AnyIndexDef<Row>, 95 + ) 96 + this.#indexes = defs 97 + return this as unknown as StoreBuilder<Name, Doc, Row, IndexArrayToRecord<T>> 98 + } 99 + 100 + build(): StoreDef<Name, Doc, Row, Indexes> { 101 + return { 102 + name: this.name, 103 + pkey: this.#pkey as StringKeys<Row> | StringKeys<Row>[], 104 + indexes: this.#indexes ?? [], 105 + encode: this.#encoder, 106 + decode: this.#decoder, 107 + __doc: undefined as unknown as Doc, 108 + __row: undefined as unknown as Row, 109 + __indexes: undefined as unknown as Indexes, 110 + } 111 + } 112 + } 113 + 114 + // ───────────────────────────────────────────────────────────────────────────── 115 + // Database Definition 116 + // ───────────────────────────────────────────────────────────────────────────── 117 + 118 + type StoresConstraint = Record<string, StoreDef<string, any, any>> 119 + 120 + type SeedFn<Stores extends StoresConstraint> = ( 121 + tx: IDBPTransaction<ToDBSchema<Stores>, StoreNames<ToDBSchema<Stores>>[], 'versionchange'>, 122 + ) => void 123 + 124 + type MigrationFn<Stores extends StoresConstraint> = ( 125 + tx: IDBPTransaction<ToDBSchema<Stores>, StoreNames<ToDBSchema<Stores>>[], 'versionchange'>, 126 + oldVersion: number, 127 + newVersion: number | null, 128 + ) => void 129 + 130 + export interface DatabaseDef<Stores extends StoresConstraint> { 131 + name: string 132 + stores: Stores 133 + seedFn?: SeedFn<Stores> 134 + migrations: MigrationFn<Stores>[] 135 + } 136 + 137 + // ───────────────────────────────────────────────────────────────────────────── 138 + // Path Resolution - convert index paths to their value types 139 + // ───────────────────────────────────────────────────────────────────────────── 140 + 141 + /** Resolve a dot-path like "foo.bar" into its value type */ 142 + type ResolvePath<T, Path extends string> = Path extends `${infer First}.${infer Rest}` 143 + ? First extends keyof T 144 + ? ResolvePath<NonNullable<T[First]>, Rest> 145 + : never 146 + : Path extends keyof T 147 + ? T[Path] 148 + : never 149 + 150 + /** 151 + * Resolve a spread path like "...tags.tag" where tags is Tag[] and we want Tag['tag'] 152 + * Path format: "...arrayProp.nestedPath" 153 + */ 154 + type ResolveSpreadPath<Row, Path extends string> = Path extends `...${infer Rest}` 155 + ? Rest extends `${infer ArrayProp}.${infer NestedPath}` 156 + ? ArrayProp extends keyof Row 157 + ? NonNullable<Row[ArrayProp]> extends Array<infer El> 158 + ? ResolvePath<El, NestedPath> 159 + : never 160 + : never 161 + : Rest extends keyof Row 162 + ? NonNullable<Row[Rest]> extends Array<infer El> 163 + ? El 164 + : never 165 + : never 166 + : ResolvePath<Row, Path> 167 + 168 + /** Convert a single index key path to its runtime type (excludes undefined since it's not a valid IDBValidKey) */ 169 + type PathToKeyType<Row, Path extends string> = NonNullable< 170 + Path extends `...${string}` ? ResolveSpreadPath<Row, Path> : ResolvePath<Row, Path> 171 + > 172 + 173 + /** Convert a tuple of paths to a tuple of key types (preserves tuple structure) */ 174 + type PathsToKeyTypes<Row, Paths extends readonly string[]> = Paths extends readonly [ 175 + infer First extends string, 176 + ...infer Rest extends readonly string[], 177 + ] 178 + ? [PathToKeyType<Row, First>, ...PathsToKeyTypes<Row, Rest>] 179 + : [] 180 + 181 + /** Convert our __indexes phantom type to idb's indexes format */ 182 + type IndexesToDBIndexes<Row, Indexes extends IndexesConstraint> = { 183 + [Name in keyof Indexes]: Indexes[Name] extends readonly string[] 184 + ? PathsToKeyTypes<Row, Indexes[Name]> 185 + : never 186 + } 187 + 188 + // Convert our StoreDef types to idb's DBSchema format 189 + type ToDBSchema<Stores extends StoresConstraint> = { 190 + [K in keyof Stores & string]: { 191 + key: string 192 + value: Stores[K]['__row'] 193 + indexes: IndexesToDBIndexes<Stores[K]['__row'], Stores[K]['__indexes']> 194 + } 195 + } 196 + 197 + // The live database handle returned by open() 198 + export interface Database<Stores extends StoresConstraint> { 199 + name: string 200 + stores: Stores 201 + idb: IDBPDatabase<ToDBSchema<Stores>> 202 + close(): void 203 + } 204 + 205 + // ───────────────────────────────────────────────────────────────────────────── 206 + // Database Builder 207 + // ───────────────────────────────────────────────────────────────────────────── 208 + 209 + export class DatabaseBuilder<Stores extends StoresConstraint = {}> { 210 + readonly #name: string 211 + readonly #storesList: StoreDef<string, any, any>[] 212 + #seedFn?: SeedFn<Stores> 213 + #migrations: MigrationFn<Stores>[] = [] 214 + 215 + constructor(name: string, stores: StoreDef<string, any, any>[] = []) { 216 + this.#name = name 217 + this.#storesList = stores 218 + } 219 + 220 + /** 221 + * Add a store to the database. 222 + * Usage: db.store<DocType>()('storeName', 'primaryKey', s => s.codec(...).indexes(...)) 223 + */ 224 + store<Doc>(): <Name extends string, Row = Doc, Indexes extends IndexesConstraint = {}>( 225 + name: Name, 226 + pkey: StringKeys<Doc> | StringKeys<Doc>[], 227 + configure?: (builder: StoreBuilder<Name, Doc, Doc, {}>) => StoreBuilder<Name, Doc, Row, Indexes>, 228 + ) => DatabaseBuilder<Stores & {[K in Name]: StoreDef<Name, Doc, Row, Indexes>}> { 229 + return <Name extends string, Row = Doc, Indexes extends IndexesConstraint = {}>( 230 + name: Name, 231 + pkey: StringKeys<Doc> | StringKeys<Doc>[], 232 + configure?: (builder: StoreBuilder<Name, Doc, Doc, {}>) => StoreBuilder<Name, Doc, Row, Indexes>, 233 + ) => { 234 + const builder = new StoreBuilder<Name, Doc, Doc, {}>(name, pkey) 235 + const configured = configure ? configure(builder) : builder 236 + const storeDef = configured.build() 237 + 238 + this.#storesList.push(storeDef) 239 + return this as unknown as DatabaseBuilder<Stores & {[K in Name]: StoreDef<Name, Doc, Row, Indexes>}> 240 + } 241 + } 242 + 243 + seed(fn: SeedFn<Stores>): this { 244 + this.#seedFn = fn 245 + return this 246 + } 247 + 248 + migrate(fn: MigrationFn<Stores>): this { 249 + this.#migrations.push(fn) 250 + return this 251 + } 252 + 253 + build(): DatabaseDef<Stores> { 254 + const stores: Record<string, StoreDef<string, any, any>> = {} 255 + for (const def of this.#storesList) { 256 + stores[def.name] = def 257 + } 258 + return { 259 + name: this.#name, 260 + stores: stores as Stores, 261 + seedFn: this.#seedFn, 262 + migrations: this.#migrations, 263 + } 264 + } 265 + } 266 + 267 + // ───────────────────────────────────────────────────────────────────────────── 268 + // Public API 269 + // ───────────────────────────────────────────────────────────────────────────── 270 + 271 + /** 272 + * Define a database schema. 273 + * Returns a DatabaseDef that can be passed to open(). 274 + */ 275 + export function define<Stores extends StoresConstraint>( 276 + name: string, 277 + cb: (db: DatabaseBuilder<{}>) => DatabaseBuilder<Stores>, 278 + ): DatabaseDef<Stores> { 279 + const builder = new DatabaseBuilder<{}>(name) 280 + const configured = cb(builder) 281 + return configured.build() 282 + } 283 + 284 + export interface MetaSchema extends DBSchema { 285 + status: { 286 + key: 'status' 287 + value: { 288 + version: number 289 + applied: Date 290 + schema: unknown 291 + } 292 + } 293 + } 294 + 295 + /** 296 + * Open a database from a definition. 297 + */ 298 + export async function open<Stores extends StoresConstraint>( 299 + def: DatabaseDef<Stores>, 300 + ): Promise<Database<Stores>> { 301 + const metadb = await openDB<MetaSchema>(`${def.name}-meta`, 1, { 302 + upgrade(db, ov) { 303 + console.log('upgrading metadb') 304 + if (ov < 1) { 305 + const store = db.createObjectStore('status') 306 + store.put({version: 0, schema: undefined, applied: new Date()}, 'status') 307 + } 308 + }, 309 + }) 310 + 311 + console.log('upgrade metadb complete') 312 + const current = await metadb.get('status', 'status') 313 + if (!current) { 314 + throw new Error('metadb upgrade never ran?') 315 + } 316 + 317 + let version = current.version 318 + const schema = def.stores 319 + if (version < 1 || current.schema != schema) { 320 + version = version + 1 321 + } 322 + 323 + console.log('opening db', def, version) 324 + const storedb = await openDB<ToDBSchema<Stores>>(def.name, version, { 325 + upgrade(db, ov, nv, tx) { 326 + try { 327 + console.log('upgrading db', ov, nv) 328 + 329 + // Create object stores if this is a fresh database 330 + if (ov < 1) { 331 + for (const key in def.stores) { 332 + const store = def.stores[key] 333 + db.createObjectStore(store.name as StoreNames<ToDBSchema<Stores>>, {keyPath: store.pkey}) 334 + } 335 + } 336 + 337 + // Create indexes (checking if they already exist) 338 + for (const key in def.stores) { 339 + const store = def.stores[key] 340 + const table = tx.objectStore(store.name as StoreNames<ToDBSchema<Stores>>) 341 + for (const idx of store.indexes) { 342 + const idxname = idx.name as IndexNames<ToDBSchema<Stores>, StoreNames<ToDBSchema<Stores>>> 343 + if (table.indexNames.contains(idxname)) { 344 + continue // TODO: diff and rebuild 345 + } 346 + 347 + // TODO: this is wrong, should be getting index keys from row? 348 + // strip "..." prefix from spread keys for IndexedDB 349 + const keyPaths = idx.keys.map((k) => k.replace('...', 'multi_').replace('.', '_')) 350 + console.log('creating index', idx.name, 'on', store.name, 'with keyPath', keyPaths) 351 + table.createIndex(idxname, keyPaths) 352 + } 353 + } 354 + 355 + if (ov < 1) { 356 + console.log('running seed') 357 + def.seedFn?.(tx) 358 + console.log('seed complete') 359 + } 360 + 361 + console.log('running migrations from', ov, 'count:', def.migrations.slice(ov).length) 362 + def.migrations.slice(ov).forEach((mig, i) => { 363 + console.log('running migration', i + ov) 364 + mig(tx, ov, nv) 365 + console.log('migration', i + ov, 'complete') 366 + }) 367 + console.log('upgrade complete') 368 + } catch (e) { 369 + console.error('error in upgrade', e) 370 + throw e 371 + } 372 + }, 373 + }) 374 + 375 + console.log('we have database version:', storedb) 376 + 377 + // TODO: implement the open logic 378 + // 1. Probe current state 379 + // 2. Compare schemas 380 + // 3. Open with version bump if needed 381 + // 4. Apply schema diff in onupgradeneeded 382 + // 5. Run seed (if fresh) or migrations (if upgrade) 383 + // 6. Save schema to _meta 384 + 385 + return { 386 + name: def.name, 387 + stores: def.stores, 388 + idb: null as any, // TODO: actual idb handle 389 + close: () => {}, 390 + } 391 + } 392 + 393 + // ───────────────────────────────────────────────────────────────────────────── 394 + // Type Tests 395 + // ───────────────────────────────────────────────────────────────────────────── 396 + 397 + type Feed = { 398 + id: string 399 + title: string 400 + tags: {tag: string; value: string}[] 401 + } 402 + 403 + type Entry = { 404 + id: string 405 + readAt?: Date 406 + feedId: string 407 + tags: {tag: string; value: string}[] 408 + } 409 + 410 + const myDb = define('mydb', (db) => 411 + db 412 + .store<Entry>()('entries', ['id', 'feedId'], (s) => 413 + s 414 + .codec({ 415 + encode: (e) => ({...e, readAt: e.readAt?.getTime()}), 416 + decode: (r) => ({...r, readAt: r.readAt ? new Date(r.readAt) : undefined}), 417 + }) 418 + .indexes((idx) => [idx().on('feedId', 'readAt'), idx().on('...tags.tag')]), 419 + ) 420 + .store<Feed>()('feeds', 'id', (s) => s.indexes((idx) => [idx().on('...tags.tag')])) 421 + .seed((_tx) => {}) 422 + .migrate((_tx) => {}), 423 + ) 424 + 425 + // Test type extraction 426 + type _TestStores = typeof myDb.stores 427 + type _TestEntries = _TestStores['entries'] 428 + type _TestFeeds = _TestStores['feeds'] 429 + type _TestEntryDoc = _TestEntries['__row'] 430 + type _TestFeedDoc = _TestFeeds['__doc'] 431 + 432 + // Should be EntryRow (with number) - will error if wrong 433 + const _e: _TestEntryDoc = {id: '1', feedId: 'f1', readAt: undefined, tags: []} 434 + void _e 435 + // Should be Feed - will error if wrong 436 + const _f: _TestFeedDoc = {id: '1', title: 'hi', tags: []} 437 + void _f 438 + 439 + // Negative test - should error if types are correct 440 + // @ts-expect-error - Entry doesn't have 'title' 441 + const _e2: _TestEntryDoc = {id: '1', title: 'wrong', tags: []} 442 + // @ts-expect-error - Feed doesn't have 'feedId' 443 + const _f2: _TestFeedDoc = {id: '1', feedId: 'wrong', tags: []} 444 + 445 + // Test open returns correct type 446 + void async function _testOpen() { 447 + const db = await open(myDb) 448 + // db.stores should have entries and feeds 449 + const _entries: typeof db.stores.entries = db.stores.entries 450 + void _entries 451 + const _feeds: typeof db.stores.feeds = db.stores.feeds 452 + void _feeds 453 + } 454 + 455 + // Test __indexes phantom type extraction 456 + type _TestEntryIndexes = _TestEntries['__indexes'] 457 + type _TestFeedIndexes = _TestFeeds['__indexes'] 458 + 459 + // Verify the index types are captured correctly 460 + // Entry should have: { idx_feedId_readAt: readonly ['feedId', 'readAt'], idx_multi_tags_tag: readonly ['...tags.tag'] } 461 + const _assertEntryIndex1: _TestEntryIndexes['idx_feedId_readAt'] = ['feedId', 'readAt'] as const 462 + const _assertEntryIndex2: _TestEntryIndexes['idx_multi_tags_tag'] = ['...tags.tag'] as const 463 + 464 + // Feed should have: { idx_multi_tags_tag: readonly ['...tags.tag'] } 465 + const _assertFeedIndex: _TestFeedIndexes['idx_multi_tags_tag'] = ['...tags.tag'] as const 466 + void [_assertEntryIndex1, _assertEntryIndex2, _assertFeedIndex] 467 + 468 + // Negative tests - these should error if types are correct 469 + // @ts-expect-error - wrong key order 470 + const _wrongOrder: _TestEntryIndexes['idx_feedId_readAt'] = ['readAt', 'feedId'] as const 471 + // @ts-expect-error - wrong key name 472 + const _wrongKey: _TestEntryIndexes['idx_feedId_readAt'] = ['feedId', 'wrongKey'] as const 473 + 474 + // Test DBSchema derivation - verify index key types are resolved 475 + type _TestDBSchema = ToDBSchema<typeof myDb.stores> 476 + type _TestEntriesDBSchema = _TestDBSchema['entries'] 477 + type _TestEntriesIndexes = _TestEntriesDBSchema['indexes'] 478 + 479 + // Direct test of PathsToKeyTypes with literal tuple 480 + type _DebugRow = _TestEntries['__row'] 481 + type _DirectTest = PathsToKeyTypes<_DebugRow, readonly ['feedId', 'readAt']> 482 + 483 + // idx_feedId_readAt should resolve to [string, number | undefined] 484 + // idx_multi_tags_tag should resolve to [string] 485 + // But the actual types depend on whether the tuple literal is preserved through IndexesToDBIndexes 486 + 487 + // Use the direct test which should work - verifies path resolution logic is correct 488 + const _dbIndexKey1: _DirectTest = ['f1', 123] 489 + void _dbIndexKey1 490 + 491 + // Test that we can at least get the index names from the derived schema 492 + type _IndexNames = keyof _TestEntriesIndexes // should be 'idx_feedId_readAt' | 'idx_multi_tags_tag' 493 + const _indexName1: _IndexNames = 'idx_feedId_readAt' 494 + const _indexName2: _IndexNames = 'idx_multi_tags_tag' 495 + void [_indexName1, _indexName2] 496 + 497 + // Now test the full resolved key types from the DBSchema 498 + type _ResolvedIdx1 = _TestEntriesIndexes['idx_feedId_readAt'] // should be [string, number | undefined] 499 + type _ResolvedIdx2 = _TestEntriesIndexes['idx_multi_tags_tag'] // should be [string] 500 + 501 + // These should work if the types are correctly resolved 502 + const _resolvedKey1: _ResolvedIdx1 = ['feed123', 12345] 503 + const _resolvedKey2: _ResolvedIdx2 = ['mytag'] 504 + void [_resolvedKey1, _resolvedKey2] 505 + 506 + // @ts-expect-error - wrong types should fail 507 + const _badResolvedKey1: _ResolvedIdx1 = [123, 'wrong'] 508 + // @ts-expect-error - wrong tuple length 509 + const _badResolvedKey2: _ResolvedIdx2 = ['tag', 'extra'] 510 + 511 + // Verify resolved types are tuples not unions 512 + // _ResolvedIdx1 should be [string, number | undefined], so [0] is string, [1] is number | undefined 513 + const _resolvedIdx1_0: _ResolvedIdx1[0] = 'test' // should be string 514 + const _resolvedIdx1_1: _ResolvedIdx1[1] = 123 // should be number | undefined 515 + // @ts-expect-error - [0] is string, not number 516 + const _badResolved1_0: _ResolvedIdx1[0] = 123 517 + // @ts-expect-error - [1] is number | undefined, not string 518 + const _badResolved1_1: _ResolvedIdx1[1] = 'wrong' 519 + void [_resolvedIdx1_0, _resolvedIdx1_1, _badResolved1_0, _badResolved1_1] 520 + 521 + // Debug: check actual structure of _TestEntryIndexes 522 + // If keys are tuples, these should work; if union, they'll be different 523 + const _debugIdx1: _TestEntryIndexes['idx_feedId_readAt'] = ['feedId', 'readAt'] 524 + const _debugIdx1_0: _TestEntryIndexes['idx_feedId_readAt'][0] = 'feedId' 525 + const _debugIdx1_1: _TestEntryIndexes['idx_feedId_readAt'][1] = 'readAt' 526 + void [_debugIdx1, _debugIdx1_0, _debugIdx1_1] 527 + 528 + // This should fail if it's a proper tuple (not a union array) 529 + // @ts-expect-error - if tuple, index 0 should only be 'feedId', not 'readAt' 530 + const _debugBad: _TestEntryIndexes['idx_feedId_readAt'][0] = 'readAt' 531 + void _debugBad 532 + 533 + // Debug: what are the actual index names? 534 + type _DebugIndexNames = keyof _TestEntryIndexes 535 + const _debugName1: _DebugIndexNames = 'idx_feedId_readAt' 536 + // @ts-expect-error - should fail if names are literal 537 + const _debugBadName: _DebugIndexNames = 'wrong_name' 538 + void [_debugName1, _debugBadName] 539 + 540 + // Debug: check DBSchema index names 541 + type _DebugDBIndexNames = keyof _TestEntriesIndexes 542 + const _debugDBName1: _DebugDBIndexNames = 'idx_feedId_readAt' 543 + // @ts-expect-error - should fail if names are literal 544 + const _debugBadDBName: _DebugDBIndexNames = 'wrong_db_name' 545 + void [_debugDBName1, _debugBadDBName]
+111
src/lib/idbase/schema-index.ts
··· 1 + import type {IndexPath, ReplicablePath} from './schema' 2 + 3 + /** Replace all dots with underscores in a string */ 4 + type ReplaceDotsWithUnderscores<S extends string> = S extends `${infer Head}.${infer Tail}` 5 + ? `${Head}_${ReplaceDotsWithUnderscores<Tail>}` 6 + : S 7 + 8 + /** Convert a key path to its index name segment (replace ... with multi_, and . with _) */ 9 + type KeyToNameSegment<K extends string> = K extends `...${infer Rest}` 10 + ? `multi_${ReplaceDotsWithUnderscores<Rest>}` 11 + : ReplaceDotsWithUnderscores<K> 12 + 13 + /** Join segments with underscore */ 14 + type JoinWithUnderscore<T extends readonly string[]> = T extends readonly [infer First extends string] 15 + ? KeyToNameSegment<First> 16 + : T extends readonly [infer First extends string, ...infer Rest extends readonly string[]] 17 + ? `${KeyToNameSegment<First>}_${JoinWithUnderscore<Rest>}` 18 + : '' 19 + 20 + /** Generate auto index name from keys */ 21 + type AutoIndexName<Keys extends readonly string[]> = `idx_${JoinWithUnderscore<Keys>}` 22 + 23 + export interface IndexDef< 24 + Row, 25 + Name extends string = string, 26 + Keys extends readonly string[] = readonly IndexPath<Row>[], 27 + > { 28 + name: Name 29 + keys: Keys 30 + replicate: ReplicablePath<Row>[] 31 + roots: Set<string> 32 + } 33 + 34 + // 35 + 36 + export class IndexBuilder<Proto, Name extends string = string, Keys extends readonly string[] = readonly []> { 37 + #name?: Name 38 + #keys?: Keys 39 + #replicate?: ReplicablePath<Proto>[] 40 + 41 + // Phantom types to extract Name and Keys 42 + declare __name: Name 43 + declare __keys: Keys 44 + 45 + constructor(name?: Name) { 46 + this.#name = name 47 + } 48 + 49 + /** 50 + * Set the index keys. If no name was provided, the Name type will be derived from the keys. 51 + */ 52 + on<K extends readonly IndexPath<Proto>[]>( 53 + ...keys: K 54 + ): IndexBuilder<Proto, Name extends string ? (string extends Name ? AutoIndexName<K> : Name) : Name, K> { 55 + this.#keys = keys as unknown as Keys 56 + return this as unknown as IndexBuilder< 57 + Proto, 58 + Name extends string ? (string extends Name ? AutoIndexName<K> : Name) : Name, 59 + K 60 + > 61 + } 62 + 63 + replicate(...keys: ReplicablePath<Proto>[]): this { 64 + this.#replicate = keys 65 + return this 66 + } 67 + 68 + #explodeRoot(key: string) { 69 + const spread = key.startsWith('...') 70 + if (spread) { 71 + const path = key.slice(3).split('.') 72 + return { 73 + spread: true, 74 + root: path[0], 75 + path, 76 + } 77 + } else { 78 + return { 79 + spread: false, 80 + root: null, 81 + path: key.split('.'), 82 + } 83 + } 84 + } 85 + 86 + #computeRoots(keys: readonly string[]): Set<string> { 87 + const roots = new Set<string>() 88 + for (const key of keys) { 89 + const {spread, root} = this.#explodeRoot(key) 90 + if (spread && root) { 91 + roots.add(root) 92 + } 93 + } 94 + 95 + return roots 96 + } 97 + 98 + build(): IndexDef<Proto, Name, Keys> { 99 + if (this.#keys === undefined) throw new Error('no keys specified') 100 + 101 + const keys = this.#keys 102 + const name = (this.#name ?? `idx_${keys.map((k) => k.replace('...', 'multi_')).join('_')}`) as Name 103 + 104 + return { 105 + name, 106 + keys, 107 + replicate: this.#replicate ?? [], 108 + roots: this.#computeRoots(keys), 109 + } 110 + } 111 + }
+11
src/lib/idbase/schema-mimetype.ts
··· 1 + import {z} from 'zod/mini' 2 + 3 + export type MimeType = z.infer<typeof mimeTypeSchema> 4 + export const mimeTypeSchema = z.templateLiteral([z.string(), '/', z.string()]) 5 + 6 + export type MimeTypeOutput<F extends MimeType> = 7 + F extends `application/json` 8 + ? unknown 9 + : F extends `text/${string}` 10 + ? string 11 + : Uint8Array
+3
src/lib/idbase/schema-store.ts
··· 1 + // Re-export from schema-builder for backwards compatibility 2 + export type {StoreDef} from './schema-builder' 3 + export {StoreBuilder} from './schema-builder'
+48
src/lib/idbase/schema.test.ts
··· 1 + import {describe, it, expectTypeOf} from 'vitest' 2 + import type {DatabaseDefinition, IndexPath} from './schema' 3 + 4 + type Proto = { 5 + id: string 6 + title: string 7 + optional?: string 8 + meta: { 9 + duration: number 10 + authors: string[] 11 + } 12 + tags: {tag: string; value: string}[] 13 + categories: Map<string, {category_id: string; title: string}> 14 + } 15 + 16 + type Indices = IndexPath<Proto> 17 + 18 + describe('IndexablePath', () => { 19 + it('accepts valid scalar paths', () => { 20 + expectTypeOf<'id'>().toExtend<Indices>() 21 + expectTypeOf<'title'>().toExtend<Indices>() 22 + expectTypeOf<'optional'>().toExtend<Indices>() 23 + }) 24 + 25 + it('accepts valid nested paths', () => { 26 + expectTypeOf<'meta.duration'>().toExtend<Indices>() 27 + }) 28 + 29 + it('accepts spread paths for arrays', () => { 30 + expectTypeOf<'...meta.authors'>().toExtend<Indices>() 31 + expectTypeOf<'...tags.tag'>().toExtend<Indices>() 32 + expectTypeOf<'...tags.value'>().toExtend<Indices>() 33 + }) 34 + 35 + it('accepts spread paths for Map keys and values', () => { 36 + expectTypeOf<'id'>().toExtend<Indices>() 37 + expectTypeOf<'...categories.key'>().toExtend<Indices>() 38 + expectTypeOf<'...categories.value.category_id'>().toExtend<Indices>() 39 + expectTypeOf<'...categories.value.title'>().toExtend<Indices>() 40 + }) 41 + 42 + it('rejects invalid paths', () => expectTypeOf<'hello'>().not.toExtend<Indices>()) 43 + it('rejects direct object access', () => expectTypeOf<'meta'>().not.toExtend<Indices>()) 44 + it('rejects direct index access', () => expectTypeOf<'tags'>().not.toExtend<Indices>()) 45 + it('rejects direct map access', () => expectTypeOf<'categories'>().not.toExtend<Indices>()) 46 + it('rejects direct map access', () => expectTypeOf<'categories.value'>().not.toExtend<Indices>()) 47 + it('rejects direct array access', () => expectTypeOf<'meta.authors'>().not.toExtend<Indices>()) 48 + })
+28
src/lib/idbase/schema.ts
··· 1 + /** returns a path extended with a dot */ 2 + type ExtendPath<Path extends string, Next extends string> = Path extends '' ? Next : `${Path}.${Next}` 3 + 4 + /** primitives that idb can index */ 5 + export type Indexable = string | number | Date | boolean 6 + 7 + /** recursive path into an object. */ 8 + export type IndexPath<T, Path extends string = '', Spread extends boolean = false> = 9 + string & T extends Indexable 10 + ? Spread extends true ? `...${Path}` : Path 11 + : T extends Array<infer O> 12 + ? IndexPath<O, Path, true> 13 + : T extends Map<infer K, infer V> 14 + ? | IndexPath<K, ExtendPath<Path, 'key'>, true> 15 + | IndexPath<V, ExtendPath<Path, 'value'>, true> 16 + : T extends object 17 + ? { 18 + [P in keyof T]: 19 + IndexPath< 20 + NonNullable<T[P]>, // strip null/undefined, as it's not part of the path 21 + ExtendPath<Path, P & string>, 22 + Spread 23 + > 24 + }[keyof T] 25 + : never 26 + 27 + /** anything (including maybe recursing) */ 28 + export type ReplicablePath<T> = IndexPath<T> | keyof T & string
+69
src/lib/idbase/test-open.ts
··· 1 + import 'fake-indexeddb/auto' 2 + import {define, open} from './schema-builder' 3 + 4 + type Feed = { 5 + id: string 6 + title: string 7 + tags: {tag: string; value: string}[] 8 + } 9 + 10 + type Entry = { 11 + id: string 12 + readAt?: Date 13 + feedId: string 14 + tags: {tag: string; value: string}[] 15 + } 16 + 17 + const myDb = define('mydb', (db) => 18 + db 19 + .store<Entry>()('entries', 'id', (s) => 20 + s 21 + .codec({ 22 + encode: (e) => ({...e, readAt: e.readAt?.getTime()}), 23 + decode: (r) => ({...r, readAt: r.readAt ? new Date(r.readAt) : undefined}), 24 + }) 25 + .indexes((idx) => [idx().on('feedId', 'readAt'), idx().on('...tags.tag')]), 26 + ) 27 + .store<Feed>()('feeds', 'id', (s) => s.indexes((idx) => [idx().on('...tags.tag')])) 28 + .seed((tx) => { 29 + const entriesStore = tx.objectStore('entries') 30 + const feedsStore = tx.objectStore('feeds') 31 + 32 + feedsStore.add({id: 'feed1', title: 'My Feed', tags: [{tag: 'tech', value: 'yes'}]}) 33 + entriesStore.add({id: 'entry1', feedId: 'feed1', readAt: Date.now(), tags: []}) 34 + 35 + console.log('Seeded database') 36 + }) 37 + .migrate((tx, ov, nv) => { 38 + const entriesStore = tx.objectStore('entries') 39 + console.log('Running migration', ov, nv, entriesStore.name, entriesStore.indexNames, entriesStore.indexNames) 40 + }) 41 + .migrate((tx, ov, nv) => { 42 + const entriesStore = tx.objectStore('feeds') 43 + console.log('Running a second migration', ov, nv, entriesStore.name, entriesStore.indexNames) 44 + }), 45 + ) 46 + 47 + async function main() { 48 + console.log('Database definition:', myDb) 49 + console.log('Stores:', Object.keys(myDb.stores)) 50 + 51 + for (const [name, store] of Object.entries(myDb.stores)) { 52 + console.log(`Store "${name}":`, { 53 + pkey: store.pkey, 54 + indexes: store.indexes.map((i) => ({name: i.name, keys: i.keys})), 55 + }) 56 + } 57 + 58 + const db = await open(myDb) 59 + console.log('Opened database:', db.name) 60 + 61 + // Test transactions 62 + // const tx = db.idb.transaction(['entries', 'feeds'], 'readonly') 63 + // const entries = await tx.objectStore('entries').getAll() 64 + // console.log('Entries:', entries) 65 + 66 + db.close() 67 + } 68 + 69 + main().catch(console.error)
+16
src/lib/idbase/type-utils.ts
··· 1 + /** Replace all dots with underscores in a string */ 2 + export type ReplaceDotsWithUnderscores<S extends string> = S extends `${infer Head}.${infer Tail}` 3 + ? `${Head}_${ReplaceDotsWithUnderscores<Tail>}` 4 + : S 5 + 6 + /** Convert a key path to its index name segment (replace ... with multi_, and . with _) */ 7 + export type KeyToNameSegment<K extends string> = K extends `...${infer Rest}` 8 + ? `multi_${ReplaceDotsWithUnderscores<Rest>}` 9 + : ReplaceDotsWithUnderscores<K> 10 + 11 + /** Join segments with underscore */ 12 + export type JoinWithUnderscore<T extends readonly string[]> = T extends readonly [infer First extends string] 13 + ? KeyToNameSegment<First> 14 + : T extends readonly [infer First extends string, ...infer Rest extends readonly string[]] 15 + ? `${KeyToNameSegment<First>}_${JoinWithUnderscore<Rest>}` 16 + : ''