Feed generator written in Golang

frantic notes from trying to tamp down db size

Changed files
+72
+72
notes.md
··· 1 + Some notes on how I handled manually cleaning up the database file. This needs to be automated later... 2 + 3 + ```bash 4 + apk add sqlite 5 + cd /data 6 + sqlite3 7 + .open coffee.db 8 + ATTACH DATABASE 'archive.db' AS archive; 9 + .schema post 10 + CREATE TABLE IF NOT EXISTS archive."post" ("uri" varchar primary key, "cid" varchar not null, "topic" varchar not null, "indexedAt" varchar not null); 11 + SELECT COUNT(*) FROM post WHERE indexedAt < '2025-06-01T00:00:00Z'; 12 + INSERT INTO archive.post SELECT * FROM post WHERE indexedAt < '2025-06-01T00:00:00Z'; 13 + DETACH DATABASE archive; 14 + DELETE FROM post WHERE indexedAt < '2025-06-01T00:00:00Z'; 15 + 16 + apk add sqlite 17 + cd /data 18 + rm archive.db archive.db-journal 19 + cp archive.db.bak archive.db 20 + sqlite3 21 + .open coffee.db 22 + ATTACH DATABASE 'archive.db' AS archive; 23 + INSERT INTO archive.post SELECT * FROM post WHERE indexedAt < '2025-06-15T00:00:00Z'; 24 + DETACH DATABASE archive; 25 + DELETE FROM post WHERE indexedAt < '2025-06-15T00:00:00Z'; 26 + .exit 27 + cp archive.db archive.db.bak 28 + 29 + ``` 30 + 31 + ```bash 32 + 32876d22b7d238:/data# apk add sqlite 33 + fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/main/x86_64/APKINDEX.tar.gz 34 + fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/community/x86_64/APKINDEX.tar.gz 35 + (1/4) Installing ncurses-terminfo-base (6.5_p20241006-r3) 36 + (2/4) Installing libncursesw (6.5_p20241006-r3) 37 + (3/4) Installing readline (8.2.13-r0) 38 + (4/4) Installing sqlite (3.48.0-r4) 39 + Executing busybox-1.37.0-r12.trigger 40 + OK: 10 MiB in 21 packages 41 + 32876d22b7d238:/data# sqlite3 42 + SQLite version 3.48.0 2025-01-14 11:05:00 43 + Enter ".help" for usage hints. 44 + Connected to a transient in-memory database. 45 + Use ".open FILENAME" to reopen on a persistent database. 46 + sqlite> .open coffee.db 47 + sqlite> ATTACH DATABASE 'archive.db' AS archive; 48 + sqlite> .schema post 49 + CREATE TABLE IF NOT EXISTS "post" ("uri" varchar primary key, "cid" varchar not null, "topic" varchar not null, "indexedAt" varchar not null); 50 + sqlite> CREATE TABLE IF NOT EXISTS archive."post" ("uri" varchar primary key, "cid" varchar not null, "topic" varchar not null, "indexedAt" varchar not null); 51 + sqlite> SELECT COUNT(*) FROM post WHERE indexedAt < '2025-06-01T00:00:00Z'; 52 + 35438 53 + sqlite> INSERT INTO archive.post SELECT * FROM post WHERE indexedAt < '2025-06-01T00:00:00Z'; 54 + sqlite> DETACH DATABASE archive; 55 + sqlite> .exit 56 + # because this crashed quite a few times... 57 + 32876d22b7d238:/data# cp archive.db archive.db.bak 58 + 59 + # ... later 60 + 32876d22b7d238:/data# sqlite3 61 + SQLite version 3.48.0 2025-01-14 11:05:00 62 + Enter ".help" for usage hints. 63 + Connected to a transient in-memory database. 64 + Use ".open FILENAME" to reopen on a persistent database. 65 + sqlite> .open coffee.db 66 + sqlite> ATTACH DATABASE 'archive.db' AS archive; 67 + sqlite> SELECT COUNT(*) FROM post WHERE indexedAt < '2025-06-29T02:43:00Z'; 68 + 188598 69 + sqlite> INSERT INTO archive.post SELECT * FROM post WHERE indexedAt < '2025-06-15T00:00:00Z'; 70 + sqlite> DETACH DATABASE archive; 71 + sqlite> DELETE FROM post WHERE indexedAt < '2025-06-15T00:00:00Z'; 72 + ```