chore: restructuring before push

+2
.gitignore
··· 1 + .cursor/ 2 + 1 3 # Dependencies 2 4 node_modules/ 3 5
+13 -148
README.md
··· 1 - # Bluesky to Leaflet.pub Reply Synchronization 1 + ## ATProto Experiments 2 2 3 - A real-time service that monitors Bluesky posts linking to leaflet.pub and synchronizes replies as comments. 3 + A monorepo of AT Protocol experiments, managed via Yarn workspaces. 4 4 5 - ## Overview 5 + ### Packages 6 6 7 - This application: 8 - - Monitors Bluesky Jetstream for real-time post events 9 - - Tracks Bluesky posts containing leaflet.pub links 10 - - Detects replies to tracked posts 11 - - Creates corresponding comments in leaflet.pub via AT Protocol's PDS 12 - - **Automatically deletes comments when Bluesky replies are deleted** (new!) 7 + - **[@atproto-experiments/bsky-leaflet-sync](./packages/bsky-leaflet-sync)** — Automatic syncing of Bluesky replies to Leaflet.pub posts as Leaflet.pub comments 13 8 14 - ## Prerequisites 15 - 16 - - Node.js 18 or higher 17 - - Yarn package manager 18 - - Bluesky account with app password 19 - - Leaflet.pub documents published on AT Protocol 20 - 21 - ## Installation 9 + ### Development 22 10 23 11 ```bash 24 - # Install dependencies 12 + # Install dependencies for all packages 25 13 yarn install 26 14 27 - # Build the project 15 + # Build all packages 28 16 yarn build 29 - ``` 30 - 31 - ## Configuration 32 - 33 - Create a `.env` file based on `.env.example`: 34 - 35 - ```env 36 - BLUESKY_HANDLE=your.handle.bsky.social 37 - BLUESKY_APP_PASSWORD=xxxx-xxxx-xxxx-xxxx 38 - JETSTREAM_URL=wss://jetstream2.us-east.bsky.network 39 - CURSOR_FILE=data/cursor.json 40 - LOG_LEVEL=info 41 - ``` 42 - 43 - ### Creating a Bluesky App Password 44 - 45 - 1. Log in to Bluesky 46 - 2. Go to Settings → App Passwords 47 - 3. Create a new app password 48 - 4. Copy the password (format: `xxxx-xxxx-xxxx-xxxx`) 49 - 5. Add it to your `.env` file 50 - 51 - ## Usage 52 - 53 - ```bash 54 - # Development mode (with auto-reload) 55 - yarn dev 56 - 57 - # Production mode 58 - yarn build 59 - yarn start 60 - ``` 61 - 62 - ## How It Works 63 - 64 - 1. **Startup**: Authenticates with Bluesky and loads saved cursor 65 - 2. **Initial Scan**: Queries leaflet.pub documents with `postRef` to build tracking map 66 - 3. **Real-time Monitoring**: Connects to Jetstream and processes events 67 - 4. **Post Tracking**: Adds new posts with leaflet.pub links to tracking 68 - 5. **Reply Detection**: Detects replies to tracked posts 69 - 6. **Comment Creation**: Creates comments in leaflet.pub with reply text and URL 70 - 7. **Deduplication**: Checks existing comments to prevent duplicates 71 - 8. **Reply Deletion**: Detects when Bluesky replies are deleted and removes corresponding comments 72 - 73 - ## Architecture 74 - 75 - - **Jetstream Client**: WebSocket connection with automatic reconnection 76 - - **Post Tracker**: In-memory Map for O(1) post lookup 77 - - **Event Processor**: Filters and routes Jetstream events (creates & deletes) 78 - - **Comment Checker**: Two-level caching for deduplication 79 - - **Comment Creator**: Creates PDS records for comments 80 - - **Comment Deleter**: Removes comments via URL search (no state needed!) 81 - - **Cursor Manager**: Persists cursor for gapless replay 82 - 83 - ### Deletion Strategy 84 17 85 - **Completely stateless** using [Constellation API](https://constellation.microcosm.blue/)! When a reply is deleted, we: 86 - 1. Query Constellation for each tracked post: "what replies to this post?" 87 - 2. Check if the deleted reply URI is in the backlinks 88 - 3. If found, delete the comment from that specific Leaflet document 89 - 90 - **Zero local state** - we query the AT Protocol network graph dynamically via Constellation's real-time index. 91 - 92 - See [REPLY_DELETION_TRACKING.md](./REPLY_DELETION_TRACKING.md) for the beautiful details. 93 - 94 - ## Logging 95 - 96 - Logs are written to: 97 - - Console (colorized for development) 98 - - `logs/combined.log` (all logs) 99 - - `logs/error.log` (errors only) 100 - 101 - ## Troubleshooting 102 - 103 - **Authentication fails:** 104 - - Verify `BLUESKY_HANDLE` is your full handle (e.g., `user.bsky.social`) 105 - - Ensure `BLUESKY_APP_PASSWORD` is correct 106 - - Check that app password has required permissions 107 - 108 - **No posts tracked:** 109 - - Verify you have leaflet.pub documents with `postRef` field 110 - - Check logs for errors during initial scan 111 - 112 - **Comments not created:** 113 - - Ensure replies are to tracked posts 114 - - Check logs for PDS record creation errors 115 - - Verify leaflet.pub comment schema is correct 116 - 117 - **Connection issues:** 118 - - Check `JETSTREAM_URL` is correct 119 - - Verify internet connection 120 - - Review logs for reconnection attempts 121 - 122 - ## Development 123 - 124 - ```bash 125 18 # Clean build artifacts 126 19 yarn clean 127 20 128 - # Build project 129 - yarn build 130 - 131 - # Run in development mode 132 - yarn dev 133 - ``` 134 - 135 - ## Project Structure 136 - 137 - ``` 138 - src/ 139 - ├── index.ts # Entry point 140 - ├── app.ts # Main application 141 - ├── config.ts # Configuration 142 - ├── auth.ts # Authentication 143 - ├── services/ 144 - │ ├── jetstreamClient.ts # WebSocket client 145 - │ ├── postTracker.ts # Post tracking 146 - │ ├── eventProcessor.ts # Event filtering 147 - │ ├── commentChecker.ts # Deduplication 148 - │ ├── commentCreator.ts # Comment creation 149 - │ └── commentDeleter.ts # Comment deletion (stateless!) 150 - ├── utils/ 151 - │ ├── cursorManager.ts # Cursor persistence 152 - │ ├── urlParser.ts # URL utilities 153 - │ └── logger.ts # Logging 154 - └── types/ 155 - ├── jetstream.ts # Jetstream types 156 - └── lexicons.ts # Lexicon types 157 - ``` 158 - 159 - ## License 160 - 161 - MIT 21 + # Run specific workspace 22 + yarn start # Runs bsky-leaflet-sync 23 + yarn dev # Dev mode for bsky-leaflet-sync 162 24 25 + # Run command in specific workspace 26 + yarn workspace @atproto-experiments/bsky-leaflet-sync <command> 27 + ```
+8 -13
package.json
··· 1 1 { 2 2 "name": "atproto-experiments", 3 3 "version": "1.0.0", 4 - "main": "dist/index.js", 4 + "private": true, 5 5 "author": "Kaushik Chakraborty <git@kaushikc.org>", 6 6 "license": "MIT", 7 + "workspaces": [ 8 + "packages/*" 9 + ], 7 10 "scripts": { 8 - "build": "tsc", 9 - "start": "yarn build && node dist/index.js", 10 - "dev": "ts-node src/index.ts", 11 - "clean": "rm -rf dist" 12 - }, 13 - "dependencies": { 14 - "@atproto/api": "^0.17.2", 15 - "dotenv": "^17.2.3", 16 - "winston": "^3.18.3", 17 - "ws": "^8.18.3" 11 + "build": "yarn workspaces run build", 12 + "clean": "yarn workspaces run clean", 13 + "start": "yarn workspace @atproto-experiments/bsky-leaflet-sync start", 14 + "dev": "yarn workspace @atproto-experiments/bsky-leaflet-sync dev" 18 15 }, 19 16 "devDependencies": { 20 17 "@types/node": "^24.7.2", 21 - "@types/ws": "^8.18.1", 22 - "ts-node": "^10.9.2", 23 18 "typescript": "^5.9.3" 24 19 } 25 20 }
+32
packages/bsky-leaflet-sync/README.md
··· 1 + # @atproto-experiments/bsky-leaflet-sync 2 + 3 + Automatic syncing of Bluesky replies to Leaflet.pub posts as Leaflet.pub comments. 4 + 5 + ## Setup 6 + 7 + 1. Copy `.env.example` to `.env` and configure: 8 + - Bluesky credentials 9 + - Leaflet.pub credentials 10 + 11 + 2. Install dependencies: 12 + ```bash 13 + yarn install 14 + ``` 15 + 16 + 3. Build: 17 + ```bash 18 + yarn build 19 + ``` 20 + 21 + 4. Run: 22 + ```bash 23 + yarn start 24 + ``` 25 + 26 + ## Development 27 + 28 + ```bash 29 + # Run in development mode 30 + yarn dev 31 + ``` 32 +
+24
packages/bsky-leaflet-sync/package.json
··· 1 + { 2 + "name": "@atproto-experiments/bsky-leaflet-sync", 3 + "version": "1.0.0", 4 + "private": true, 5 + "license": "MIT", 6 + "main": "dist/index.js", 7 + "scripts": { 8 + "build": "tsc", 9 + "start": "yarn build && node dist/index.js", 10 + "dev": "ts-node index.ts", 11 + "clean": "rm -rf dist" 12 + }, 13 + "dependencies": { 14 + "@atproto/api": "^0.17.2", 15 + "dotenv": "^17.2.3", 16 + "winston": "^3.18.3", 17 + "ws": "^8.18.3" 18 + }, 19 + "devDependencies": { 20 + "@types/ws": "^8.18.1", 21 + "ts-node": "^10.9.2" 22 + } 23 + } 24 +
+10
packages/bsky-leaflet-sync/tsconfig.json
··· 1 + { 2 + "extends": "../../tsconfig.json", 3 + "compilerOptions": { 4 + "outDir": "./dist", 5 + "rootDir": "." 6 + }, 7 + "include": ["./**/*"], 8 + "exclude": ["node_modules", "dist"] 9 + } 10 +
src/app.ts packages/bsky-leaflet-sync/app.ts
src/auth.ts packages/bsky-leaflet-sync/auth.ts
src/config.ts packages/bsky-leaflet-sync/config.ts
src/index.ts packages/bsky-leaflet-sync/index.ts
src/services/commentChecker.ts packages/bsky-leaflet-sync/services/commentChecker.ts
src/services/commentCreator.ts packages/bsky-leaflet-sync/services/commentCreator.ts
src/services/commentDeleter.ts packages/bsky-leaflet-sync/services/commentDeleter.ts
src/services/eventProcessor.ts packages/bsky-leaflet-sync/services/eventProcessor.ts
src/services/jetstreamClient.ts packages/bsky-leaflet-sync/services/jetstreamClient.ts
src/services/postTracker.ts packages/bsky-leaflet-sync/services/postTracker.ts
src/types/jetstream.ts packages/bsky-leaflet-sync/types/jetstream.ts
src/types/lexicons.ts packages/bsky-leaflet-sync/types/lexicons.ts
src/utils/cursorManager.ts packages/bsky-leaflet-sync/utils/cursorManager.ts
src/utils/logger.ts packages/bsky-leaflet-sync/utils/logger.ts
src/utils/urlParser.ts packages/bsky-leaflet-sync/utils/urlParser.ts
+1 -5
tsconfig.json
··· 3 3 "target": "ES2022", 4 4 "module": "commonjs", 5 5 "lib": ["ES2022"], 6 - "outDir": "./dist", 7 - "rootDir": "./src", 8 6 "strict": true, 9 7 "esModuleInterop": true, 10 8 "skipLibCheck": true, ··· 14 12 "declaration": true, 15 13 "declarationMap": true, 16 14 "sourceMap": true 17 - }, 18 - "include": ["src/**/*"], 19 - "exclude": ["node_modules", "dist"] 15 + } 20 16 } 21 17