WIP! A BB-style forum, on the ATmosphere! We're still working... we'll be back soon when we have something to show off!
node typescript hono htmx atproto

docs: add SQLite support implementation plan

+1612
+1612
docs/plans/2026-02-24-sqlite-support-plan.md
··· 1 + # SQLite Support Implementation Plan 2 + 3 + > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. 4 + 5 + **Goal:** Add SQLite as a fully supported database backend alongside PostgreSQL, replacing the `permissions text[]` column with a normalized `role_permissions` join table that works on both dialects. 6 + 7 + **Architecture:** `createDb(url)` in `packages/db/src/index.ts` inspects the URL prefix to select postgres.js or @libsql/client. Two Drizzle schema files mirror each other with dialect-appropriate column types. The Postgres schema is the TypeScript source of truth; the SQLite instance is cast at the boundary. The Postgres migration is split into two steps with a data migration script between them to avoid data loss. 8 + 9 + **Tech Stack:** `@libsql/client`, `drizzle-orm/libsql`, `drizzle-orm/postgres-js`, Drizzle Kit 0.31 10 + 11 + --- 12 + 13 + ## Important notes before starting 14 + 15 + - **`seed-roles.ts` needs no changes.** It creates AT Proto records with `permissions: string[]` — that's the lexicon format, not the DB schema. The indexer handles DB writes. 16 + - **`rolePermissions` cascade deletes.** The `onDelete: "cascade"` FK means deleting a role automatically cleans up its `role_permissions` rows. Test cleanup code in `test-context.ts` does not need a separate `rolePermissions` delete. 17 + - **Two-stage Postgres schema.** You MUST generate migration 0011 (add table) BEFORE removing the `permissions` column from schema.ts. Removing it first would generate a single migration that drops data. 18 + - **Run commands inside `devenv shell`.** All `pnpm` commands must run inside the Nix devenv shell, or use the absolute path `.devenv/profile/bin/pnpm`. 19 + - **Test environment.** `DATABASE_URL` must be set for all test runs. For SQLite: `DATABASE_URL=file::memory:`. 20 + 21 + --- 22 + 23 + ## Task 1: Add @libsql/client to packages/db 24 + 25 + **Files:** 26 + - Modify: `packages/db/package.json` 27 + 28 + **Step 1: Add the dependency** 29 + 30 + In `packages/db/package.json`, add `@libsql/client` to `dependencies`: 31 + 32 + ```json 33 + "dependencies": { 34 + "@libsql/client": "^0.14.0", 35 + "drizzle-orm": "^0.45.1", 36 + "postgres": "^3.4.8" 37 + } 38 + ``` 39 + 40 + **Step 2: Install** 41 + 42 + ```sh 43 + pnpm install 44 + ``` 45 + 46 + Expected: no errors, `@libsql/client` appears in `pnpm-lock.yaml`. 47 + 48 + **Step 3: Commit** 49 + 50 + ```sh 51 + git add packages/db/package.json pnpm-lock.yaml 52 + git commit -m "feat(db): add @libsql/client dependency for SQLite support" 53 + ``` 54 + 55 + --- 56 + 57 + ## Task 2: Create packages/db/src/schema.sqlite.ts 58 + 59 + **Files:** 60 + - Create: `packages/db/src/schema.sqlite.ts` 61 + 62 + **Step 1: Write the full SQLite schema** 63 + 64 + The SQLite schema is identical to the Postgres schema in table names and column names but uses `sqlite-core` helpers. Key differences: `integer({ mode: "bigint" }).primaryKey({ autoIncrement: true })` replaces `bigserial`, `integer({ mode: "timestamp" })` replaces `timestamp({ withTimezone: true })`, and `integer({ mode: "boolean" })` replaces `boolean`. There is no `permissions` column — `role_permissions` table is included from the start. 65 + 66 + Create `packages/db/src/schema.sqlite.ts`: 67 + 68 + ```typescript 69 + import { 70 + sqliteTable, 71 + text, 72 + integer, 73 + uniqueIndex, 74 + index, 75 + primaryKey, 76 + } from "drizzle-orm/sqlite-core"; 77 + 78 + // ── forums ────────────────────────────────────────────── 79 + export const forums = sqliteTable( 80 + "forums", 81 + { 82 + id: integer("id", { mode: "bigint" }).primaryKey({ autoIncrement: true }), 83 + did: text("did").notNull(), 84 + rkey: text("rkey").notNull(), 85 + cid: text("cid").notNull(), 86 + name: text("name").notNull(), 87 + description: text("description"), 88 + indexedAt: integer("indexed_at", { mode: "timestamp" }).notNull(), 89 + }, 90 + (table) => [uniqueIndex("forums_did_rkey_idx").on(table.did, table.rkey)] 91 + ); 92 + 93 + // ── categories ────────────────────────────────────────── 94 + export const categories = sqliteTable( 95 + "categories", 96 + { 97 + id: integer("id", { mode: "bigint" }).primaryKey({ autoIncrement: true }), 98 + did: text("did").notNull(), 99 + rkey: text("rkey").notNull(), 100 + cid: text("cid").notNull(), 101 + name: text("name").notNull(), 102 + description: text("description"), 103 + slug: text("slug"), 104 + sortOrder: integer("sort_order"), 105 + forumId: integer("forum_id", { mode: "bigint" }).references(() => forums.id), 106 + createdAt: integer("created_at", { mode: "timestamp" }).notNull(), 107 + indexedAt: integer("indexed_at", { mode: "timestamp" }).notNull(), 108 + }, 109 + (table) => [ 110 + uniqueIndex("categories_did_rkey_idx").on(table.did, table.rkey), 111 + ] 112 + ); 113 + 114 + // ── boards ────────────────────────────────────────────── 115 + export const boards = sqliteTable( 116 + "boards", 117 + { 118 + id: integer("id", { mode: "bigint" }).primaryKey({ autoIncrement: true }), 119 + did: text("did").notNull(), 120 + rkey: text("rkey").notNull(), 121 + cid: text("cid").notNull(), 122 + name: text("name").notNull(), 123 + description: text("description"), 124 + slug: text("slug"), 125 + sortOrder: integer("sort_order"), 126 + categoryId: integer("category_id", { mode: "bigint" }).references(() => categories.id), 127 + categoryUri: text("category_uri").notNull(), 128 + createdAt: integer("created_at", { mode: "timestamp" }).notNull(), 129 + indexedAt: integer("indexed_at", { mode: "timestamp" }).notNull(), 130 + }, 131 + (table) => [ 132 + uniqueIndex("boards_did_rkey_idx").on(table.did, table.rkey), 133 + index("boards_category_id_idx").on(table.categoryId), 134 + ] 135 + ); 136 + 137 + // ── users ─────────────────────────────────────────────── 138 + export const users = sqliteTable("users", { 139 + did: text("did").primaryKey(), 140 + handle: text("handle"), 141 + indexedAt: integer("indexed_at", { mode: "timestamp" }).notNull(), 142 + }); 143 + 144 + // ── memberships ───────────────────────────────────────── 145 + export const memberships = sqliteTable( 146 + "memberships", 147 + { 148 + id: integer("id", { mode: "bigint" }).primaryKey({ autoIncrement: true }), 149 + did: text("did").notNull().references(() => users.did), 150 + rkey: text("rkey").notNull(), 151 + cid: text("cid").notNull(), 152 + forumId: integer("forum_id", { mode: "bigint" }).references(() => forums.id), 153 + forumUri: text("forum_uri").notNull(), 154 + role: text("role"), 155 + roleUri: text("role_uri"), 156 + joinedAt: integer("joined_at", { mode: "timestamp" }), 157 + createdAt: integer("created_at", { mode: "timestamp" }).notNull(), 158 + indexedAt: integer("indexed_at", { mode: "timestamp" }).notNull(), 159 + }, 160 + (table) => [ 161 + uniqueIndex("memberships_did_rkey_idx").on(table.did, table.rkey), 162 + index("memberships_did_idx").on(table.did), 163 + ] 164 + ); 165 + 166 + // ── posts ─────────────────────────────────────────────── 167 + export const posts = sqliteTable( 168 + "posts", 169 + { 170 + id: integer("id", { mode: "bigint" }).primaryKey({ autoIncrement: true }), 171 + did: text("did").notNull().references(() => users.did), 172 + rkey: text("rkey").notNull(), 173 + cid: text("cid").notNull(), 174 + title: text("title"), 175 + text: text("text").notNull(), 176 + forumUri: text("forum_uri"), 177 + boardUri: text("board_uri"), 178 + boardId: integer("board_id", { mode: "bigint" }).references(() => boards.id), 179 + rootPostId: integer("root_post_id", { mode: "bigint" }).references((): any => posts.id), 180 + parentPostId: integer("parent_post_id", { mode: "bigint" }).references((): any => posts.id), 181 + rootUri: text("root_uri"), 182 + parentUri: text("parent_uri"), 183 + createdAt: integer("created_at", { mode: "timestamp" }).notNull(), 184 + indexedAt: integer("indexed_at", { mode: "timestamp" }).notNull(), 185 + bannedByMod: integer("banned_by_mod", { mode: "boolean" }).notNull().default(false), 186 + deletedByUser: integer("deleted_by_user", { mode: "boolean" }).notNull().default(false), 187 + }, 188 + (table) => [ 189 + uniqueIndex("posts_did_rkey_idx").on(table.did, table.rkey), 190 + index("posts_forum_uri_idx").on(table.forumUri), 191 + index("posts_board_id_idx").on(table.boardId), 192 + index("posts_board_uri_idx").on(table.boardUri), 193 + index("posts_root_post_id_idx").on(table.rootPostId), 194 + ] 195 + ); 196 + 197 + // ── mod_actions ───────────────────────────────────────── 198 + export const modActions = sqliteTable( 199 + "mod_actions", 200 + { 201 + id: integer("id", { mode: "bigint" }).primaryKey({ autoIncrement: true }), 202 + did: text("did").notNull(), 203 + rkey: text("rkey").notNull(), 204 + cid: text("cid").notNull(), 205 + action: text("action").notNull(), 206 + subjectDid: text("subject_did"), 207 + subjectPostUri: text("subject_post_uri"), 208 + forumId: integer("forum_id", { mode: "bigint" }).references(() => forums.id), 209 + reason: text("reason"), 210 + createdBy: text("created_by").notNull(), 211 + expiresAt: integer("expires_at", { mode: "timestamp" }), 212 + createdAt: integer("created_at", { mode: "timestamp" }).notNull(), 213 + indexedAt: integer("indexed_at", { mode: "timestamp" }).notNull(), 214 + }, 215 + (table) => [ 216 + uniqueIndex("mod_actions_did_rkey_idx").on(table.did, table.rkey), 217 + index("mod_actions_subject_did_idx").on(table.subjectDid), 218 + index("mod_actions_subject_post_uri_idx").on(table.subjectPostUri), 219 + ] 220 + ); 221 + 222 + // ── firehose_cursor ───────────────────────────────────── 223 + export const firehoseCursor = sqliteTable("firehose_cursor", { 224 + service: text("service").primaryKey().default("jetstream"), 225 + cursor: integer("cursor", { mode: "bigint" }).notNull(), 226 + updatedAt: integer("updated_at", { mode: "timestamp" }).notNull(), 227 + }); 228 + 229 + // ── roles ─────────────────────────────────────────────── 230 + // No `permissions` column — permissions are stored in role_permissions. 231 + export const roles = sqliteTable( 232 + "roles", 233 + { 234 + id: integer("id", { mode: "bigint" }).primaryKey({ autoIncrement: true }), 235 + did: text("did").notNull(), 236 + rkey: text("rkey").notNull(), 237 + cid: text("cid").notNull(), 238 + name: text("name").notNull(), 239 + description: text("description"), 240 + priority: integer("priority").notNull(), 241 + createdAt: integer("created_at", { mode: "timestamp" }).notNull(), 242 + indexedAt: integer("indexed_at", { mode: "timestamp" }).notNull(), 243 + }, 244 + (table) => [ 245 + uniqueIndex("roles_did_rkey_idx").on(table.did, table.rkey), 246 + index("roles_did_idx").on(table.did), 247 + index("roles_did_name_idx").on(table.did, table.name), 248 + ] 249 + ); 250 + 251 + // ── role_permissions ───────────────────────────────────── 252 + // Normalized join table replacing the permissions text[] column. 253 + // Cascade delete ensures permissions are cleaned up when a role is deleted. 254 + export const rolePermissions = sqliteTable( 255 + "role_permissions", 256 + { 257 + roleId: integer("role_id", { mode: "bigint" }) 258 + .notNull() 259 + .references(() => roles.id, { onDelete: "cascade" }), 260 + permission: text("permission").notNull(), 261 + }, 262 + (t) => [primaryKey({ columns: [t.roleId, t.permission] })] 263 + ); 264 + 265 + // ── backfill_progress ─────────────────────────────────── 266 + export const backfillProgress = sqliteTable("backfill_progress", { 267 + id: integer("id", { mode: "bigint" }).primaryKey({ autoIncrement: true }), 268 + status: text("status").notNull(), 269 + backfillType: text("backfill_type").notNull(), 270 + lastProcessedDid: text("last_processed_did"), 271 + didsTotal: integer("dids_total").notNull().default(0), 272 + didsProcessed: integer("dids_processed").notNull().default(0), 273 + recordsIndexed: integer("records_indexed").notNull().default(0), 274 + startedAt: integer("started_at", { mode: "timestamp" }).notNull(), 275 + completedAt: integer("completed_at", { mode: "timestamp" }), 276 + errorMessage: text("error_message"), 277 + }); 278 + 279 + // ── backfill_errors ───────────────────────────────────── 280 + export const backfillErrors = sqliteTable( 281 + "backfill_errors", 282 + { 283 + id: integer("id", { mode: "bigint" }).primaryKey({ autoIncrement: true }), 284 + backfillId: integer("backfill_id", { mode: "bigint" }) 285 + .notNull() 286 + .references(() => backfillProgress.id), 287 + did: text("did").notNull(), 288 + collection: text("collection").notNull(), 289 + errorMessage: text("error_message").notNull(), 290 + createdAt: integer("created_at", { mode: "timestamp" }).notNull(), 291 + }, 292 + (table) => [index("backfill_errors_backfill_id_idx").on(table.backfillId)] 293 + ); 294 + ``` 295 + 296 + **Step 2: Verify it compiles** 297 + 298 + ```sh 299 + pnpm --filter @atbb/db build 300 + ``` 301 + 302 + Expected: `dist/` updated with no TypeScript errors. 303 + 304 + **Step 3: Commit** 305 + 306 + ```sh 307 + git add packages/db/src/schema.sqlite.ts 308 + git commit -m "feat(db): add SQLite schema file" 309 + ``` 310 + 311 + --- 312 + 313 + ## Task 3: Stage 1 — Add rolePermissions to Postgres schema (keep permissions column) 314 + 315 + > **Critical:** Do NOT remove the `permissions` column yet. That happens in Task 8 after the data migration runs. Removing it here would collapse both migrations into one, losing existing data. 316 + 317 + **Files:** 318 + - Modify: `packages/db/src/schema.ts` 319 + 320 + **Step 1: Add `primaryKey` to imports and add `rolePermissions` table** 321 + 322 + In `packages/db/src/schema.ts`: 323 + 324 + 1. Add `primaryKey` to the import from `drizzle-orm/pg-core` (line 1): 325 + 326 + ```typescript 327 + import { 328 + pgTable, 329 + bigserial, 330 + text, 331 + timestamp, 332 + integer, 333 + boolean, 334 + bigint, 335 + uniqueIndex, 336 + index, 337 + primaryKey, 338 + } from "drizzle-orm/pg-core"; 339 + ``` 340 + 341 + 2. After the `roles` table definition (after line 215), add the `rolePermissions` table: 342 + 343 + ```typescript 344 + // ── role_permissions ───────────────────────────────────── 345 + // Normalized join table replacing the permissions text[] column. 346 + // Cascade delete ensures permissions are cleaned up when a role is deleted. 347 + export const rolePermissions = pgTable( 348 + "role_permissions", 349 + { 350 + roleId: bigint("role_id", { mode: "bigint" }) 351 + .notNull() 352 + .references(() => roles.id, { onDelete: "cascade" }), 353 + permission: text("permission").notNull(), 354 + }, 355 + (t) => [primaryKey({ columns: [t.roleId, t.permission] })] 356 + ); 357 + ``` 358 + 359 + Do NOT touch the `permissions` column on `roles` yet. 360 + 361 + **Step 2: Export `rolePermissions` from packages/db/src/index.ts** 362 + 363 + `packages/db/src/index.ts` line 40 currently does `export * from "./schema.js"` — `rolePermissions` is automatically included. No change needed. 364 + 365 + **Step 3: Build to verify** 366 + 367 + ```sh 368 + pnpm --filter @atbb/db build 369 + ``` 370 + 371 + Expected: clean build. 372 + 373 + **Step 4: Commit** 374 + 375 + ```sh 376 + git add packages/db/src/schema.ts 377 + git commit -m "feat(db): add role_permissions table to Postgres schema (permissions column still present)" 378 + ``` 379 + 380 + --- 381 + 382 + ## Task 4: Update createDb factory for URL-based dialect detection 383 + 384 + **Files:** 385 + - Modify: `packages/db/src/index.ts` 386 + 387 + **Step 1: Rewrite index.ts** 388 + 389 + Replace the entire file with: 390 + 391 + ```typescript 392 + import { drizzle as drizzlePg } from "drizzle-orm/postgres-js"; 393 + import { drizzle as drizzleSqlite } from "drizzle-orm/libsql"; 394 + import { createClient } from "@libsql/client"; 395 + import postgres from "postgres"; 396 + import * as pgSchema from "./schema.js"; 397 + import * as sqliteSchema from "./schema.sqlite.js"; 398 + 399 + /** 400 + * Create a Drizzle database instance from a connection URL. 401 + * 402 + * URL prefix determines the driver: 403 + * postgres:// or postgresql:// → postgres.js (PostgreSQL) 404 + * file: → @libsql/client (SQLite file) 405 + * file::memory: → @libsql/client (SQLite in-memory, tests) 406 + * libsql:// → @libsql/client (Turso cloud) 407 + */ 408 + export function createDb(databaseUrl: string): Database { 409 + if (databaseUrl.startsWith("postgres")) { 410 + return drizzlePg(postgres(databaseUrl), { schema: pgSchema }) as Database; 411 + } 412 + return drizzleSqlite( 413 + createClient({ url: databaseUrl }), 414 + { schema: sqliteSchema } 415 + ) as unknown as Database; 416 + } 417 + 418 + // Database type uses the Postgres schema as the TypeScript source of truth. 419 + // Both dialects produce identical column names and compatible TypeScript types 420 + // (bigint for IDs, Date for timestamps), so the cast is safe at the app layer. 421 + export type Database = ReturnType<typeof drizzlePg<typeof pgSchema>>; 422 + 423 + export type Transaction = Parameters<Parameters<Database["transaction"]>[0]>[0]; 424 + 425 + export type DbOrTransaction = Database | Transaction; 426 + 427 + export * from "./schema.js"; 428 + ``` 429 + 430 + **Step 2: Build and verify** 431 + 432 + ```sh 433 + pnpm --filter @atbb/db build 434 + ``` 435 + 436 + Expected: clean build. The `as unknown as Database` cast is intentional — it bridges the Drizzle dialect types at the one boundary where we cross dialects. 437 + 438 + **Step 3: Commit** 439 + 440 + ```sh 441 + git add packages/db/src/index.ts 442 + git commit -m "feat(db): URL-based driver detection in createDb (postgres vs SQLite)" 443 + ``` 444 + 445 + --- 446 + 447 + ## Task 5: Set up Drizzle config files and update package.json scripts 448 + 449 + **Files:** 450 + - Create: `apps/appview/drizzle.postgres.config.ts` 451 + - Create: `apps/appview/drizzle.sqlite.config.ts` 452 + - Modify: `apps/appview/package.json` 453 + - The old `apps/appview/drizzle.config.ts` will be deleted 454 + 455 + **Step 1: Create `drizzle.postgres.config.ts`** 456 + 457 + ```typescript 458 + import { defineConfig } from "drizzle-kit"; 459 + 460 + export default defineConfig({ 461 + schema: "../../packages/db/src/schema.ts", 462 + out: "./drizzle", 463 + dialect: "postgresql", 464 + dbCredentials: { 465 + url: process.env.DATABASE_URL!, 466 + }, 467 + }); 468 + ``` 469 + 470 + **Step 2: Create `drizzle.sqlite.config.ts`** 471 + 472 + ```typescript 473 + import { defineConfig } from "drizzle-kit"; 474 + 475 + export default defineConfig({ 476 + schema: "../../packages/db/src/schema.sqlite.ts", 477 + out: "./drizzle-sqlite", 478 + dialect: "sqlite", 479 + dbCredentials: { 480 + url: process.env.DATABASE_URL!, 481 + }, 482 + }); 483 + ``` 484 + 485 + **Step 3: Update `apps/appview/package.json` scripts** 486 + 487 + Replace the two `db:*` scripts with four: 488 + 489 + ```json 490 + "db:generate": "drizzle-kit generate --config=drizzle.postgres.config.ts", 491 + "db:migrate": "drizzle-kit migrate --config=drizzle.postgres.config.ts", 492 + "db:generate:sqlite": "drizzle-kit generate --config=drizzle.sqlite.config.ts", 493 + "db:migrate:sqlite": "drizzle-kit migrate --config=drizzle.sqlite.config.ts" 494 + ``` 495 + 496 + **Step 4: Delete the old config** 497 + 498 + ```sh 499 + rm apps/appview/drizzle.config.ts 500 + ``` 501 + 502 + **Step 5: Verify both configs parse correctly** 503 + 504 + ```sh 505 + pnpm --filter @atbb/appview db:generate -- --dry-run 2>&1 | head -5 506 + ``` 507 + 508 + Expected: drizzle-kit starts without config errors (may error on DB connection which is fine for dry-run). 509 + 510 + **Step 6: Commit** 511 + 512 + ```sh 513 + git add apps/appview/drizzle.postgres.config.ts apps/appview/drizzle.sqlite.config.ts apps/appview/package.json 514 + git rm apps/appview/drizzle.config.ts 515 + git commit -m "feat(appview): add dialect-specific Drizzle configs and update db scripts" 516 + ``` 517 + 518 + --- 519 + 520 + ## Task 6: Generate and apply Postgres migration 0011 (add role_permissions table) 521 + 522 + **Step 1: Generate the migration** 523 + 524 + ```sh 525 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview db:generate 526 + ``` 527 + 528 + Expected: a new file `apps/appview/drizzle/0011_*.sql` is created. Verify its contents — it should `CREATE TABLE role_permissions` and nothing else. If it also tries to `DROP COLUMN permissions`, STOP — the schema.ts has the permissions column removed prematurely. Fix by re-adding permissions to schema.ts before continuing. 529 + 530 + **Step 2: Apply the migration** 531 + 532 + ```sh 533 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview db:migrate 534 + ``` 535 + 536 + Expected: `role_permissions` table now exists in the database. 537 + 538 + **Step 3: Verify** 539 + 540 + Connect to the database and confirm the table exists: 541 + 542 + ```sh 543 + psql postgres://atbb:atbb@localhost:5432/atbb -c "\d role_permissions" 544 + ``` 545 + 546 + Expected: table with `role_id` (bigint), `permission` (text), primary key on both. 547 + 548 + **Step 4: Commit** 549 + 550 + ```sh 551 + git add apps/appview/drizzle/ 552 + git commit -m "feat(db): migration 0011 — add role_permissions table" 553 + ``` 554 + 555 + --- 556 + 557 + ## Task 7: Write the data migration script 558 + 559 + **Files:** 560 + - Create: `apps/appview/scripts/migrate-permissions.ts` 561 + 562 + **Step 1: Write the script** 563 + 564 + ```typescript 565 + /** 566 + * One-time data migration: copies permissions from roles.permissions[] 567 + * into the role_permissions join table. 568 + * 569 + * Safe to re-run (ON CONFLICT DO NOTHING). 570 + * Must be run AFTER migration 0011 (role_permissions table exists) 571 + * and BEFORE migration 0012 (permissions column is dropped). 572 + */ 573 + import postgres from "postgres"; 574 + import { drizzle } from "drizzle-orm/postgres-js"; 575 + import * as schema from "../../packages/db/src/schema.js"; 576 + import { sql } from "drizzle-orm"; 577 + 578 + const databaseUrl = process.env.DATABASE_URL; 579 + if (!databaseUrl) { 580 + console.error("DATABASE_URL is required"); 581 + process.exit(1); 582 + } 583 + 584 + const client = postgres(databaseUrl); 585 + const db = drizzle(client, { schema }); 586 + 587 + async function run() { 588 + // Read roles that still have permissions in the array column. 589 + // We use raw SQL here because the Drizzle schema will have the 590 + // permissions column removed by the time this script ships. 591 + const roles = await db.execute( 592 + sql`SELECT id, permissions FROM roles WHERE array_length(permissions, 1) > 0` 593 + ); 594 + 595 + if (roles.length === 0) { 596 + console.log("No roles with permissions to migrate."); 597 + await client.end(); 598 + return; 599 + } 600 + 601 + let totalPermissions = 0; 602 + 603 + for (const role of roles) { 604 + const roleId = role.id as bigint; 605 + const permissions = role.permissions as string[]; 606 + 607 + if (!permissions || permissions.length === 0) continue; 608 + 609 + // Insert each permission as a row, skip duplicates (idempotent) 610 + await db.execute( 611 + sql`INSERT INTO role_permissions (role_id, permission) 612 + SELECT ${roleId}, unnest(${sql.raw(`ARRAY[${permissions.map(p => `'${p.replace(/'/g, "''")}'`).join(",")}]`)}::text[]) 613 + ON CONFLICT DO NOTHING` 614 + ); 615 + 616 + totalPermissions += permissions.length; 617 + console.log(` Role ${roleId}: migrated ${permissions.length} permissions`); 618 + } 619 + 620 + console.log( 621 + `\nMigrated ${totalPermissions} permissions across ${roles.length} roles.` 622 + ); 623 + console.log("Safe to proceed with migration 0012 (drop permissions column)."); 624 + 625 + await client.end(); 626 + } 627 + 628 + run().catch((err) => { 629 + console.error("Migration failed:", err); 630 + process.exit(1); 631 + }); 632 + ``` 633 + 634 + **Step 2: Add a package.json script to run it** 635 + 636 + In `apps/appview/package.json`, add: 637 + 638 + ```json 639 + "migrate-permissions": "tsx --env-file=../../.env scripts/migrate-permissions.ts" 640 + ``` 641 + 642 + **Step 3: Run it against your development database** 643 + 644 + ```sh 645 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview migrate-permissions 646 + ``` 647 + 648 + Expected output: "Migrated N permissions across M roles." or "No roles with permissions to migrate." (both are valid depending on whether you had seeded roles). 649 + 650 + **Step 4: Verify (optional)** 651 + 652 + ```sh 653 + psql postgres://atbb:atbb@localhost:5432/atbb -c "SELECT r.name, rp.permission FROM roles r JOIN role_permissions rp ON r.id = rp.role_id ORDER BY r.name, rp.permission" 654 + ``` 655 + 656 + **Step 5: Commit** 657 + 658 + ```sh 659 + git add apps/appview/scripts/migrate-permissions.ts apps/appview/package.json 660 + git commit -m "feat(appview): add migrate-permissions data migration script" 661 + ``` 662 + 663 + --- 664 + 665 + ## Task 8: Stage 2 — Remove permissions column, generate and apply migration 0012 666 + 667 + **Files:** 668 + - Modify: `packages/db/src/schema.ts` 669 + 670 + **Step 1: Remove the permissions column from roles in schema.ts** 671 + 672 + In `packages/db/src/schema.ts`, remove the `permissions` line (currently line 205) from the `roles` table: 673 + 674 + ```typescript 675 + // REMOVE this line: 676 + permissions: text("permissions").array().notNull().default(sql`'{}'::text[]`), 677 + ``` 678 + 679 + Also remove the `sql` import from `drizzle-orm` if it's no longer used anywhere else in the file. Check by searching for other `sql\`` usages in schema.ts — if none, remove: 680 + 681 + ```typescript 682 + import { sql } from "drizzle-orm"; 683 + ``` 684 + 685 + **Step 2: Build to verify the schema compiles** 686 + 687 + ```sh 688 + pnpm --filter @atbb/db build 689 + ``` 690 + 691 + Expected: clean build. 692 + 693 + **Step 3: Generate migration 0012** 694 + 695 + ```sh 696 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview db:generate 697 + ``` 698 + 699 + Expected: a new `apps/appview/drizzle/0012_*.sql` containing only `ALTER TABLE roles DROP COLUMN permissions`. 700 + 701 + **Step 4: Apply migration 0012** 702 + 703 + ```sh 704 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview db:migrate 705 + ``` 706 + 707 + **Step 5: Verify** 708 + 709 + ```sh 710 + psql postgres://atbb:atbb@localhost:5432/atbb -c "\d roles" 711 + ``` 712 + 713 + Expected: no `permissions` column in the output. 714 + 715 + **Step 6: Commit** 716 + 717 + ```sh 718 + git add packages/db/src/schema.ts apps/appview/drizzle/ 719 + git commit -m "feat(db): migration 0012 — drop permissions column from roles (data already in role_permissions)" 720 + ``` 721 + 722 + --- 723 + 724 + ## Task 9: Generate SQLite migrations 725 + 726 + SQLite deployments always start clean, so a single migration contains the full schema without the `permissions` column. 727 + 728 + **Step 1: Generate SQLite migrations** 729 + 730 + Use an in-memory SQLite URL (no actual database needed, drizzle-kit generates from schema): 731 + 732 + ```sh 733 + DATABASE_URL=file::memory: pnpm --filter @atbb/appview db:generate:sqlite 734 + ``` 735 + 736 + Expected: `apps/appview/drizzle-sqlite/` directory created with `0000_*.sql` containing the full schema including `role_permissions`, no `permissions` column. 737 + 738 + **Step 2: Inspect the generated migration** 739 + 740 + ```sh 741 + cat apps/appview/drizzle-sqlite/0000_*.sql 742 + ``` 743 + 744 + Verify: all 11 tables present, `role_permissions` table present, no `permissions` column in `roles`. 745 + 746 + **Step 3: Commit** 747 + 748 + ```sh 749 + git add apps/appview/drizzle-sqlite/ 750 + git commit -m "feat(db): add SQLite migrations (single clean initial migration)" 751 + ``` 752 + 753 + --- 754 + 755 + ## Task 10: Update checkPermission and getUserRole (TDD) 756 + 757 + **Files:** 758 + - Modify: `apps/appview/src/middleware/__tests__/permissions.test.ts` 759 + - Modify: `apps/appview/src/middleware/permissions.ts` 760 + 761 + The current `checkPermission` at lines 57 and 62 uses `role.permissions.includes(...)`. After this change, it queries `rolePermissions` directly. The `getUserRole` function at line 91 returns `permissions: string[]` — its return type and select must change. 762 + 763 + **Step 1: Update the existing failing tests** 764 + 765 + In `permissions.test.ts`, find all `db.insert(roles).values({...})` calls that include a `permissions` field. For each, you must: 766 + 1. Remove `permissions` from the roles insert 767 + 2. Add a separate `db.insert(rolePermissions).values(...)` for each permission 768 + 769 + Find all occurrences: 770 + 771 + ```sh 772 + grep -n "permissions:" apps/appview/src/middleware/__tests__/permissions.test.ts 773 + ``` 774 + 775 + For each role insertion like: 776 + ```typescript 777 + // OLD: single insert with permissions array 778 + await ctx.db.insert(roles).values({ 779 + did: ctx.config.forumDid, 780 + rkey: "member", 781 + cid: "bafy...", 782 + name: "Member", 783 + permissions: ["space.atbb.permission.createPosts"], 784 + priority: 30, 785 + createdAt: new Date(), 786 + indexedAt: new Date(), 787 + }); 788 + ``` 789 + 790 + Replace with: 791 + ```typescript 792 + // NEW: role insert + separate rolePermissions inserts 793 + const [memberRole] = await ctx.db.insert(roles).values({ 794 + did: ctx.config.forumDid, 795 + rkey: "member", 796 + cid: "bafy...", 797 + name: "Member", 798 + priority: 30, 799 + createdAt: new Date(), 800 + indexedAt: new Date(), 801 + }).returning({ id: roles.id }); 802 + 803 + await ctx.db.insert(rolePermissions).values([ 804 + { roleId: memberRole.id, permission: "space.atbb.permission.createPosts" }, 805 + ]); 806 + ``` 807 + 808 + For the wildcard Owner role test: 809 + ```typescript 810 + await ctx.db.insert(rolePermissions).values([ 811 + { roleId: ownerRole.id, permission: "*" }, 812 + ]); 813 + ``` 814 + 815 + Also add `import { rolePermissions } from "@atbb/db";` to the test file imports. 816 + 817 + **Step 2: Run the tests to confirm they fail (permissions column no longer exists)** 818 + 819 + ```sh 820 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview test -- --reporter=verbose middleware/__tests__/permissions 821 + ``` 822 + 823 + Expected: compilation errors or test failures because `roles.permissions` no longer exists in the schema. 824 + 825 + **Step 3: Update permissions.ts** 826 + 827 + In `apps/appview/src/middleware/permissions.ts`: 828 + 829 + 1. Add `rolePermissions` to the import (line 4): 830 + ```typescript 831 + import { memberships, roles, rolePermissions } from "@atbb/db"; 832 + ``` 833 + 834 + 2. Add `or` to the drizzle-orm import (line 5): 835 + ```typescript 836 + import { eq, and, or } from "drizzle-orm"; 837 + ``` 838 + 839 + 3. Replace steps 4 and 5 in `checkPermission` (lines 56-62) with a `rolePermissions` query: 840 + 841 + ```typescript 842 + // 4. Check if user has the permission (wildcard or specific) 843 + const [match] = await ctx.db 844 + .select() 845 + .from(rolePermissions) 846 + .where( 847 + and( 848 + eq(rolePermissions.roleId, role.id), 849 + or( 850 + eq(rolePermissions.permission, permission), 851 + eq(rolePermissions.permission, "*") 852 + ) 853 + ) 854 + ) 855 + .limit(1); 856 + 857 + return !!match; // fail-closed: undefined → false 858 + ``` 859 + 860 + 4. Update `getUserRole` return type (line 91) — remove `permissions` from the return type: 861 + 862 + ```typescript 863 + async function getUserRole( 864 + ctx: AppContext, 865 + did: string 866 + ): Promise<{ id: bigint; name: string; priority: number } | null> { 867 + ``` 868 + 869 + 5. Update the `select` inside `getUserRole` (lines 109-114) — remove `permissions` from the select: 870 + 871 + ```typescript 872 + const [role] = await ctx.db 873 + .select({ 874 + id: roles.id, 875 + name: roles.name, 876 + priority: roles.priority, 877 + }) 878 + .from(roles) 879 + .where( 880 + and( 881 + eq(roles.did, ctx.config.forumDid), 882 + eq(roles.rkey, roleRkey) 883 + ) 884 + ) 885 + .limit(1); 886 + ``` 887 + 888 + **Step 4: Run the tests again** 889 + 890 + ```sh 891 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview test -- --reporter=verbose middleware/__tests__/permissions 892 + ``` 893 + 894 + Expected: all permissions middleware tests pass. 895 + 896 + **Step 5: Commit** 897 + 898 + ```sh 899 + git add apps/appview/src/middleware/permissions.ts apps/appview/src/middleware/__tests__/permissions.test.ts 900 + git commit -m "feat(appview): update checkPermission and getUserRole to use role_permissions table" 901 + ``` 902 + 903 + --- 904 + 905 + ## Task 11: Update indexer for role records (TDD) 906 + 907 + **Files:** 908 + - Modify: `apps/appview/src/lib/__tests__/indexer-roles.test.ts` 909 + - Modify: `apps/appview/src/lib/indexer.ts` 910 + 911 + The indexer receives role records from the AT Proto firehose. Currently it stores `permissions: record.permissions` in the `roles` table. After this change, it must: 912 + 1. Upsert the role row (no permissions field) 913 + 2. Delete existing `role_permissions` rows for that role (handles permission updates) 914 + 3. Insert new `role_permissions` rows 915 + 916 + **Step 1: Update the indexer tests** 917 + 918 + In `indexer-roles.test.ts`: 919 + 920 + 1. Find all assertions like `expect(role.permissions).toEqual([...])` — these will now need to query `rolePermissions`: 921 + 922 + ```typescript 923 + // OLD 924 + const [role] = await ctx.db.select().from(roles).where(...); 925 + expect(role.permissions).toEqual(["space.atbb.permission.createPosts"]); 926 + 927 + // NEW 928 + const [role] = await ctx.db.select().from(roles).where(...); 929 + const perms = await ctx.db 930 + .select({ permission: rolePermissions.permission }) 931 + .from(rolePermissions) 932 + .where(eq(rolePermissions.roleId, role.id)); 933 + expect(perms.map(p => p.permission)).toEqual(["space.atbb.permission.createPosts"]); 934 + ``` 935 + 936 + 2. Add `import { rolePermissions } from "@atbb/db";` to the test file imports. 937 + 938 + **Step 2: Run tests to confirm they fail** 939 + 940 + ```sh 941 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview test -- --reporter=verbose lib/__tests__/indexer-roles 942 + ``` 943 + 944 + Expected: test failures (role.permissions no longer exists as a column). 945 + 946 + **Step 3: Find the role indexing code in indexer.ts** 947 + 948 + Search for where the indexer inserts into the `roles` table: 949 + 950 + ```sh 951 + grep -n "roles\|permissions" apps/appview/src/lib/indexer.ts | head -40 952 + ``` 953 + 954 + Find the block that inserts/upserts a role. It likely looks like: 955 + 956 + ```typescript 957 + // OLD pattern in indexer 958 + await ctx.db 959 + .insert(roles) 960 + .values({ 961 + did: ..., 962 + rkey: ..., 963 + cid: ..., 964 + name: record.name, 965 + description: record.description, 966 + permissions: record.permissions, // ← remove this 967 + priority: record.priority, 968 + createdAt: new Date(record.createdAt), 969 + indexedAt: new Date(), 970 + }) 971 + .onConflictDoUpdate({ target: [roles.did, roles.rkey], set: { ... } }); 972 + ``` 973 + 974 + **Step 4: Update the role indexing code** 975 + 976 + Replace the role upsert block with a three-step operation: 977 + 978 + ```typescript 979 + // 1. Upsert the role row (no permissions field) 980 + const [upsertedRole] = await ctx.db 981 + .insert(roles) 982 + .values({ 983 + did: record_did, // use whatever variable holds the record author's DID 984 + rkey: record_rkey, // use whatever variable holds the record key 985 + cid: record_cid, 986 + name: record.name, 987 + description: record.description ?? null, 988 + priority: record.priority, 989 + createdAt: new Date(record.createdAt), 990 + indexedAt: new Date(), 991 + }) 992 + .onConflictDoUpdate({ 993 + target: [roles.did, roles.rkey], 994 + set: { 995 + name: record.name, 996 + description: record.description ?? null, 997 + priority: record.priority, 998 + cid: record_cid, 999 + indexedAt: new Date(), 1000 + }, 1001 + }) 1002 + .returning({ id: roles.id }); 1003 + 1004 + // 2. Replace all permissions for this role (handles updates) 1005 + await ctx.db 1006 + .delete(rolePermissions) 1007 + .where(eq(rolePermissions.roleId, upsertedRole.id)); 1008 + 1009 + // 3. Insert new permissions (skip if empty) 1010 + if (record.permissions && record.permissions.length > 0) { 1011 + await ctx.db.insert(rolePermissions).values( 1012 + record.permissions.map((permission: string) => ({ 1013 + roleId: upsertedRole.id, 1014 + permission, 1015 + })) 1016 + ); 1017 + } 1018 + ``` 1019 + 1020 + Also add `rolePermissions` to the indexer's imports from `@atbb/db`. 1021 + 1022 + **Step 5: Run indexer tests** 1023 + 1024 + ```sh 1025 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview test -- --reporter=verbose lib/__tests__/indexer-roles 1026 + ``` 1027 + 1028 + Expected: all indexer role tests pass. 1029 + 1030 + **Step 6: Commit** 1031 + 1032 + ```sh 1033 + git add apps/appview/src/lib/indexer.ts apps/appview/src/lib/__tests__/indexer-roles.test.ts 1034 + git commit -m "feat(appview): update indexer to store role permissions in role_permissions table" 1035 + ``` 1036 + 1037 + --- 1038 + 1039 + ## Task 12: Update admin routes to return permissions via join (TDD) 1040 + 1041 + **Files:** 1042 + - Modify: `apps/appview/src/routes/__tests__/admin.test.ts` 1043 + - Modify: `apps/appview/src/routes/admin.ts` 1044 + 1045 + The admin routes return role objects with a `permissions` array in the response. Since `getUserRole` no longer returns permissions, any code reading `role.permissions` from that function needs to query `rolePermissions` directly. 1046 + 1047 + **Step 1: Find all admin.ts usages of permissions** 1048 + 1049 + ```sh 1050 + grep -n "permissions" apps/appview/src/routes/admin.ts 1051 + ``` 1052 + 1053 + Look for: 1054 + - Responses that include `role.permissions` (the GET /api/admin/roles endpoint) 1055 + - The GET /api/admin/members/me endpoint that returns the caller's permissions 1056 + 1057 + **Step 2: Update the test — ensure role inserts include rolePermissions** 1058 + 1059 + In `admin.test.ts`, find all `db.insert(roles).values({...})` calls with `permissions` arrays. Apply the same pattern as Task 10 — split into role insert + rolePermissions insert. 1060 + 1061 + Run the failing tests first: 1062 + 1063 + ```sh 1064 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview test -- --reporter=verbose routes/__tests__/admin 1065 + ``` 1066 + 1067 + **Step 3: Add a helper to fetch a role's permissions** 1068 + 1069 + In `admin.ts`, wherever a role's permissions are needed in the response, add a helper query: 1070 + 1071 + ```typescript 1072 + // Fetch permissions for a role by ID 1073 + async function getRolePermissions( 1074 + ctx: AppContext, 1075 + roleId: bigint 1076 + ): Promise<string[]> { 1077 + const perms = await ctx.db 1078 + .select({ permission: rolePermissions.permission }) 1079 + .from(rolePermissions) 1080 + .where(eq(rolePermissions.roleId, roleId)); 1081 + return perms.map((p) => p.permission); 1082 + } 1083 + ``` 1084 + 1085 + Use this helper when building role response objects: 1086 + 1087 + ```typescript 1088 + // Instead of: permissions: role.permissions 1089 + permissions: await getRolePermissions(ctx, role.id), 1090 + ``` 1091 + 1092 + For the GET /api/admin/members/me endpoint returning the caller's permissions: 1093 + 1094 + ```typescript 1095 + const userRole = await getUserRole(ctx, user.did); 1096 + const permissions = userRole 1097 + ? await getRolePermissions(ctx, userRole.id) 1098 + : []; 1099 + ``` 1100 + 1101 + Add `rolePermissions` to admin.ts imports from `@atbb/db`. 1102 + 1103 + **Step 4: Run admin tests** 1104 + 1105 + ```sh 1106 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview test -- --reporter=verbose routes/__tests__/admin 1107 + ``` 1108 + 1109 + Expected: all admin tests pass. 1110 + 1111 + **Step 5: Run the full test suite to catch any remaining permissions references** 1112 + 1113 + ```sh 1114 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview test 1115 + ``` 1116 + 1117 + Expected: all tests pass. If any still fail with `role.permissions` errors, find them with: 1118 + 1119 + ```sh 1120 + grep -rn "\.permissions" apps/appview/src/routes/ apps/appview/src/middleware/ apps/appview/src/lib/ 1121 + ``` 1122 + 1123 + Fix any remaining cases using the same `rolePermissions` query pattern. 1124 + 1125 + **Step 6: Commit** 1126 + 1127 + ```sh 1128 + git add apps/appview/src/routes/admin.ts apps/appview/src/routes/__tests__/admin.test.ts 1129 + git commit -m "feat(appview): update admin routes to fetch permissions from role_permissions table" 1130 + ``` 1131 + 1132 + --- 1133 + 1134 + ## Task 13: Update test context for SQLite support 1135 + 1136 + **Files:** 1137 + - Modify: `apps/appview/src/lib/__tests__/test-context.ts` 1138 + 1139 + The test context currently hardcodes postgres.js. It needs to detect the URL and use `createDb` for both dialects. For SQLite, it must run migrations programmatically before tests, and skip the manual `sql.end()` call (libsql clients close automatically). 1140 + 1141 + **Step 1: Update test-context.ts** 1142 + 1143 + Replace the file with this updated version (preserving all existing cleanup logic): 1144 + 1145 + ```typescript 1146 + import { eq, or, like } from "drizzle-orm"; 1147 + import { createDb } from "@atbb/db"; 1148 + import { forums, posts, users, categories, memberships, boards, roles, modActions, backfillProgress, backfillErrors } from "@atbb/db"; 1149 + import * as schema from "@atbb/db"; 1150 + import { createLogger } from "@atbb/logger"; 1151 + import path from "path"; 1152 + import { fileURLToPath } from "url"; 1153 + import type { AppConfig } from "../config.js"; 1154 + import type { AppContext } from "../app-context.js"; 1155 + 1156 + const __dirname = fileURLToPath(new URL(".", import.meta.url)); 1157 + 1158 + export interface TestContext extends AppContext { 1159 + cleanup: () => Promise<void>; 1160 + cleanDatabase: () => Promise<void>; 1161 + } 1162 + 1163 + export interface TestContextOptions { 1164 + emptyDb?: boolean; 1165 + } 1166 + 1167 + export async function createTestContext( 1168 + options: TestContextOptions = {} 1169 + ): Promise<TestContext> { 1170 + const config: AppConfig = { 1171 + port: 3000, 1172 + forumDid: "did:plc:test-forum", 1173 + pdsUrl: "https://test.pds", 1174 + databaseUrl: process.env.DATABASE_URL ?? "", 1175 + jetstreamUrl: "wss://test.jetstream", 1176 + logLevel: "warn", 1177 + oauthPublicUrl: "http://localhost:3000", 1178 + sessionSecret: "test-secret-at-least-32-characters-long", 1179 + sessionTtlDays: 7, 1180 + backfillRateLimit: 10, 1181 + backfillConcurrency: 10, 1182 + backfillCursorMaxAgeHours: 48, 1183 + }; 1184 + 1185 + const db = createDb(config.databaseUrl); 1186 + 1187 + const isSqlite = !config.databaseUrl.startsWith("postgres"); 1188 + 1189 + // For SQLite: run migrations programmatically before any tests 1190 + if (isSqlite) { 1191 + const { migrate } = await import("drizzle-orm/libsql/migrator"); 1192 + const migrationsFolder = path.resolve(__dirname, "../../../../../apps/appview/drizzle-sqlite"); 1193 + await migrate(db as any, { migrationsFolder }); 1194 + } 1195 + 1196 + const stubFirehose = { 1197 + start: () => Promise.resolve(), 1198 + stop: () => Promise.resolve(), 1199 + } as any; 1200 + 1201 + const stubOAuthClient = {} as any; 1202 + const stubOAuthStateStore = { destroy: () => {} } as any; 1203 + const stubOAuthSessionStore = { destroy: () => {} } as any; 1204 + const stubCookieSessionStore = { destroy: () => {} } as any; 1205 + const stubForumAgent = null; 1206 + 1207 + const cleanDatabase = async () => { 1208 + if (isSqlite) { 1209 + // SQLite in-memory: delete in FK order (role_permissions cascade from roles) 1210 + await db.delete(posts).catch(() => {}); 1211 + await db.delete(memberships).catch(() => {}); 1212 + await db.delete(users).catch(() => {}); 1213 + await db.delete(boards).catch(() => {}); 1214 + await db.delete(categories).catch(() => {}); 1215 + await db.delete(roles).catch(() => {}); // cascades to role_permissions 1216 + await db.delete(modActions).catch(() => {}); 1217 + await db.delete(backfillErrors).catch(() => {}); 1218 + await db.delete(backfillProgress).catch(() => {}); 1219 + await db.delete(forums).catch(() => {}); 1220 + return; 1221 + } 1222 + 1223 + // Postgres: delete by test DID patterns 1224 + await db.delete(posts).where(eq(posts.did, config.forumDid)).catch(() => {}); 1225 + await db.delete(posts).where(like(posts.did, "did:plc:test-%")).catch(() => {}); 1226 + await db.delete(memberships).where(like(memberships.did, "did:plc:test-%")).catch(() => {}); 1227 + await db.delete(users).where(like(users.did, "did:plc:test-%")).catch(() => {}); 1228 + await db.delete(boards).where(eq(boards.did, config.forumDid)).catch(() => {}); 1229 + await db.delete(categories).where(eq(categories.did, config.forumDid)).catch(() => {}); 1230 + await db.delete(roles).where(eq(roles.did, config.forumDid)).catch(() => {}); // cascades to role_permissions 1231 + await db.delete(modActions).where(eq(modActions.did, config.forumDid)).catch(() => {}); 1232 + await db.delete(backfillErrors).catch(() => {}); 1233 + await db.delete(backfillProgress).catch(() => {}); 1234 + await db.delete(forums).where(eq(forums.did, config.forumDid)).catch(() => {}); 1235 + }; 1236 + 1237 + await cleanDatabase(); 1238 + 1239 + if (!options.emptyDb) { 1240 + await db.insert(forums).values({ 1241 + did: config.forumDid, 1242 + rkey: "self", 1243 + cid: "bafytest", 1244 + name: "Test Forum", 1245 + description: "A test forum", 1246 + indexedAt: new Date(), 1247 + }); 1248 + } 1249 + 1250 + const logger = createLogger({ 1251 + service: "atbb-appview-test", 1252 + level: "warn", 1253 + }); 1254 + 1255 + return { 1256 + db, 1257 + config, 1258 + logger, 1259 + firehose: stubFirehose, 1260 + oauthClient: stubOAuthClient, 1261 + oauthStateStore: stubOAuthStateStore, 1262 + oauthSessionStore: stubOAuthSessionStore, 1263 + cookieSessionStore: stubCookieSessionStore, 1264 + forumAgent: stubForumAgent, 1265 + backfillManager: null, 1266 + cleanDatabase, 1267 + cleanup: async () => { 1268 + const testDidPattern = or( 1269 + eq(posts.did, "did:plc:test-user"), 1270 + eq(posts.did, "did:plc:topicsuser"), 1271 + like(posts.did, "did:plc:test-%"), 1272 + like(posts.did, "did:plc:duptest-%"), 1273 + like(posts.did, "did:plc:create-%"), 1274 + like(posts.did, "did:plc:pds-fail-%") 1275 + ); 1276 + await db.delete(posts).where(testDidPattern); 1277 + 1278 + const testMembershipPattern = or( 1279 + eq(memberships.did, "did:plc:test-user"), 1280 + eq(memberships.did, "did:plc:topicsuser"), 1281 + like(memberships.did, "did:plc:test-%"), 1282 + like(memberships.did, "did:plc:duptest-%"), 1283 + like(memberships.did, "did:plc:create-%"), 1284 + like(memberships.did, "did:plc:pds-fail-%") 1285 + ); 1286 + await db.delete(memberships).where(testMembershipPattern); 1287 + 1288 + const testUserPattern = or( 1289 + eq(users.did, "did:plc:test-user"), 1290 + eq(users.did, "did:plc:topicsuser"), 1291 + like(users.did, "did:plc:test-%"), 1292 + like(users.did, "did:plc:duptest-%"), 1293 + like(users.did, "did:plc:create-%"), 1294 + like(users.did, "did:plc:pds-fail-%") 1295 + ); 1296 + await db.delete(users).where(testUserPattern); 1297 + 1298 + await db.delete(boards).where(eq(boards.did, config.forumDid)); 1299 + await db.delete(categories).where(eq(categories.did, config.forumDid)); 1300 + await db.delete(roles).where(eq(roles.did, config.forumDid)); // cascades to role_permissions 1301 + await db.delete(modActions).where(eq(modActions.did, config.forumDid)); 1302 + await db.delete(backfillErrors).catch(() => {}); 1303 + await db.delete(backfillProgress).catch(() => {}); 1304 + await db.delete(forums).where(eq(forums.did, config.forumDid)); 1305 + 1306 + // Close connection — only needed for postgres.js 1307 + if (!isSqlite) { 1308 + // Access the underlying postgres client via the db instance internals 1309 + // The createDb function owns the client lifecycle; for tests we accept 1310 + // that postgres connections close when the process exits. 1311 + // If connection leak warnings appear, add explicit client tracking to createDb. 1312 + } 1313 + }, 1314 + } as TestContext; 1315 + } 1316 + ``` 1317 + 1318 + > **Note on postgres connection cleanup:** The old test context held a `sql` reference to call `sql.end()`. Since `createDb` now owns the client, tests rely on process exit to close Postgres connections. If this causes "too many connections" errors in CI, refactor `createDb` to return `{ db, close: () => Promise<void> }` and thread `close()` through to the test context cleanup. 1319 + 1320 + **Step 2: Run the full test suite with Postgres** 1321 + 1322 + ```sh 1323 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm --filter @atbb/appview test 1324 + ``` 1325 + 1326 + Expected: all tests pass. 1327 + 1328 + **Step 3: Run the full test suite with SQLite** 1329 + 1330 + ```sh 1331 + DATABASE_URL=file::memory: pnpm --filter @atbb/appview test 1332 + ``` 1333 + 1334 + Expected: all tests pass. If any tests fail due to SQLite behavior differences (e.g., boolean handling, BigInt serialization), fix them individually — the errors will be self-descriptive. 1335 + 1336 + **Step 4: Commit** 1337 + 1338 + ```sh 1339 + git add apps/appview/src/lib/__tests__/test-context.ts 1340 + git commit -m "feat(appview): update test context to support SQLite in-memory testing" 1341 + ``` 1342 + 1343 + --- 1344 + 1345 + ## Task 14: Create docker-compose.sqlite.yml 1346 + 1347 + **Files:** 1348 + - Create: `docker-compose.sqlite.yml` 1349 + 1350 + **Step 1: Write the compose file** 1351 + 1352 + ```yaml 1353 + # SQLite deployment — no external database required. 1354 + # Data is persisted via a named volume at /data/atbb.db inside the container. 1355 + # 1356 + # Usage: 1357 + # docker compose -f docker-compose.sqlite.yml up 1358 + # 1359 + # The existing docker-compose.yml (PostgreSQL) is unchanged. 1360 + services: 1361 + appview: 1362 + build: . 1363 + environment: 1364 + DATABASE_URL: file:/data/atbb.db 1365 + NODE_ENV: production 1366 + env_file: 1367 + - .env 1368 + volumes: 1369 + - atbb_data:/data 1370 + ports: 1371 + - "80:80" 1372 + restart: unless-stopped 1373 + 1374 + volumes: 1375 + atbb_data: 1376 + driver: local 1377 + ``` 1378 + 1379 + **Step 2: Commit** 1380 + 1381 + ```sh 1382 + git add docker-compose.sqlite.yml 1383 + git commit -m "feat: add docker-compose.sqlite.yml for SQLite deployments" 1384 + ``` 1385 + 1386 + --- 1387 + 1388 + ## Task 15: Update nix/module.nix 1389 + 1390 + **Files:** 1391 + - Modify: `nix/module.nix` 1392 + 1393 + **Step 1: Read the current module.nix** 1394 + 1395 + Read `nix/module.nix` in full to find the exact location of: 1396 + - The `database.enable` option definition 1397 + - The `database.name` option definition 1398 + - The `atbb-migrate` service script 1399 + - The `atbb-appview` environment block 1400 + - The `services.postgresql` block 1401 + 1402 + **Step 2: Add `database.type` and `database.path` options** 1403 + 1404 + Find the `database` attribute set in the options block and add two new options alongside the existing ones: 1405 + 1406 + ```nix 1407 + type = lib.mkOption { 1408 + type = lib.types.enum [ "postgresql" "sqlite" ]; 1409 + default = "postgresql"; 1410 + description = "Database backend. Use 'sqlite' for embedded single-file storage without a separate PostgreSQL service."; 1411 + }; 1412 + 1413 + path = lib.mkOption { 1414 + type = lib.types.path; 1415 + default = "/var/lib/atbb/atbb.db"; 1416 + description = "Path to the SQLite database file. Only used when database.type = \"sqlite\"."; 1417 + }; 1418 + ``` 1419 + 1420 + **Step 3: Make `database.enable` default to true only for PostgreSQL** 1421 + 1422 + Find the `database.enable` option and update its default: 1423 + 1424 + ```nix 1425 + enable = lib.mkOption { 1426 + type = lib.types.bool; 1427 + default = cfg.database.type == "postgresql"; 1428 + description = "Enable local PostgreSQL 17 service. Ignored when database.type = \"sqlite\"."; 1429 + }; 1430 + ``` 1431 + 1432 + **Step 4: Make the DATABASE_URL conditional in atbb-appview** 1433 + 1434 + Find the `environment` block in the `atbb-appview` systemd service and update the `DATABASE_URL` line (currently hardcoded to postgres): 1435 + 1436 + ```nix 1437 + DATABASE_URL = if cfg.database.type == "sqlite" 1438 + then "file:${cfg.database.path}" 1439 + else "postgres:///atbb?host=/run/postgresql"; 1440 + ``` 1441 + 1442 + **Step 5: Add StateDirectory for SQLite file persistence** 1443 + 1444 + In the `serviceConfig` block of `atbb-appview`, add: 1445 + 1446 + ```nix 1447 + StateDirectory = lib.mkIf (cfg.database.type == "sqlite") "atbb"; 1448 + ``` 1449 + 1450 + This creates `/var/lib/atbb/` and makes it writable by the service user. 1451 + 1452 + **Step 6: Make the atbb-migrate script dialect-aware** 1453 + 1454 + Find the `atbb-migrate` oneshot service and update its `script`: 1455 + 1456 + ```nix 1457 + script = if cfg.database.type == "sqlite" 1458 + then "${atbb}/bin/atbb db:migrate:sqlite" 1459 + else "${atbb}/bin/atbb db:migrate"; 1460 + ``` 1461 + 1462 + **Step 7: Make PostgreSQL conditional** 1463 + 1464 + Find the `services.postgresql` block and wrap it: 1465 + 1466 + ```nix 1467 + services.postgresql = lib.mkIf (cfg.database.type == "postgresql" && cfg.database.enable) { 1468 + # ... existing content unchanged 1469 + }; 1470 + ``` 1471 + 1472 + **Step 8: Commit** 1473 + 1474 + ```sh 1475 + git add nix/module.nix 1476 + git commit -m "feat(nix): add database.type option to NixOS module (postgresql | sqlite)" 1477 + ``` 1478 + 1479 + --- 1480 + 1481 + ## Task 16: Update nix/package.nix 1482 + 1483 + **Files:** 1484 + - Modify: `nix/package.nix` 1485 + 1486 + **Step 1: Find the drizzle migrations copy in installPhase** 1487 + 1488 + Read `nix/package.nix` and find the line that copies the drizzle directory (search for `drizzle`). 1489 + 1490 + **Step 2: Add drizzle-sqlite alongside the existing copy** 1491 + 1492 + Find the line (approximately): 1493 + ```nix 1494 + cp -r apps/appview/drizzle $out/apps/appview/drizzle 1495 + ``` 1496 + 1497 + Add immediately after: 1498 + ```nix 1499 + cp -r apps/appview/drizzle-sqlite $out/apps/appview/drizzle-sqlite 1500 + ``` 1501 + 1502 + **Step 3: Commit** 1503 + 1504 + ```sh 1505 + git add nix/package.nix 1506 + git commit -m "feat(nix): include SQLite migrations in Nix package output" 1507 + ``` 1508 + 1509 + --- 1510 + 1511 + ## Task 17: Update devenv.nix for optional PostgreSQL 1512 + 1513 + **Files:** 1514 + - Modify: `devenv.nix` 1515 + 1516 + **Step 1: Wrap the postgres service in mkDefault** 1517 + 1518 + Read `devenv.nix` and find the `services.postgres` block. Change `enable = true` (or the implicit enablement) to use `lib.mkDefault`: 1519 + 1520 + ```nix 1521 + services.postgres = { 1522 + enable = lib.mkDefault true; 1523 + # ... rest of existing config unchanged 1524 + }; 1525 + ``` 1526 + 1527 + **Step 2: Add a comment explaining the SQLite override** 1528 + 1529 + Add a comment above the `services.postgres` block: 1530 + 1531 + ```nix 1532 + # PostgreSQL is enabled by default for development. 1533 + # To use SQLite instead, create devenv.local.nix with: 1534 + # { ... }: { services.postgres.enable = false; } 1535 + # Then set DATABASE_URL=file:./data/atbb.db in your .env file. 1536 + ``` 1537 + 1538 + **Step 3: Update .env.example to document the SQLite option** 1539 + 1540 + In `.env.example`, add a comment to the DATABASE_URL line: 1541 + 1542 + ```sh 1543 + # PostgreSQL (default, used with devenv): 1544 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb 1545 + 1546 + # SQLite alternative (no devenv postgres needed): 1547 + # DATABASE_URL=file:./data/atbb.db 1548 + # DATABASE_URL=file::memory: # in-memory, for tests only 1549 + ``` 1550 + 1551 + **Step 4: Commit** 1552 + 1553 + ```sh 1554 + git add devenv.nix .env.example 1555 + git commit -m "feat(devenv): make postgres optional via mkDefault, document SQLite alternative" 1556 + ``` 1557 + 1558 + --- 1559 + 1560 + ## Task 18: Final verification 1561 + 1562 + **Step 1: Run the full test suite with Postgres** 1563 + 1564 + ```sh 1565 + DATABASE_URL=postgres://atbb:atbb@localhost:5432/atbb pnpm test 1566 + ``` 1567 + 1568 + Expected: all tests pass. 1569 + 1570 + **Step 2: Run the full test suite with SQLite in-memory** 1571 + 1572 + ```sh 1573 + DATABASE_URL=file::memory: pnpm test 1574 + ``` 1575 + 1576 + Expected: all tests pass. This is the proof that the abstraction works end-to-end. 1577 + 1578 + **Step 3: Verify TypeScript builds cleanly** 1579 + 1580 + ```sh 1581 + pnpm build 1582 + ``` 1583 + 1584 + Expected: clean build with no TypeScript errors. 1585 + 1586 + **Step 4: Smoke test SQLite file deployment** 1587 + 1588 + ```sh 1589 + DATABASE_URL=file:./data/atbb.db pnpm --filter @atbb/appview db:migrate:sqlite 1590 + DATABASE_URL=file:./data/atbb.db pnpm --filter @atbb/appview dev 1591 + ``` 1592 + 1593 + Expected: server starts, migrations run, `/api/healthz` returns 200. 1594 + 1595 + **Step 5: Clean up the data directory** 1596 + 1597 + ```sh 1598 + rm -rf ./data/ 1599 + ``` 1600 + 1601 + **Step 6: Final commit** 1602 + 1603 + ```sh 1604 + git add . 1605 + git commit -m "feat: SQLite support complete — dual-dialect database with role_permissions join table" 1606 + ``` 1607 + 1608 + --- 1609 + 1610 + ## Operator upgrade instructions (reference) 1611 + 1612 + See `docs/plans/2026-02-24-sqlite-support-design.md` → "Operator Migration Instructions" for the three-step Postgres upgrade procedure and NixOS-specific instructions. The key safety rule: **run `migrate-permissions` between applying migration 0011 and migration 0012**.