ATlast — you'll never need to find your favorites on another platform again. Find your favs in the ATmosphere.
atproto

update documentation with current debugging status

PLAN.md updates:
- Added current status section with recent fixes and active work
- Marked Phase 0 as complete
- Marked Phase 1 as in progress (debugging)
- Updated changelog with 2025-12-26 progress
- Updated decision graph count to 288 nodes

EXTENSION_STATUS.md updates:
- Changed state from READY FOR TESTING to DEBUGGING
- Added fixed issues section (NaN bug, database init)
- Added active debugging section
- Updated decision graph summary to 288 nodes
- Added node references for recent fixes (#287-288)

Decision graph:
- Synced with latest nodes (288 total, 276 edges)
- Tracked database initialization outcome

byarielm.fyi a203bc34 a8a4b0a8

verified
+12 -3
EXTENSION_STATUS.md
··· 1 1 # Extension Implementation Status 2 2 3 - ## Current State: READY FOR TESTING ⚠️ 3 + ## Current State: DEBUGGING 🔧 4 4 5 5 ### What's Complete ✅ 6 6 ··· 83 83 84 84 ### Current Issues 🐛 85 85 86 - None known - ready for testing! 86 + **Fixed:** 87 + - ✅ NaN database error - Fixed missing `matchedUsers` parameter in `extension-import.ts` (node #287) 88 + - ✅ Database initialized successfully (node #288) 89 + 90 + **Active Debugging:** 91 + - Extension upload flow has bugs that need investigation 92 + - Upload completes but results page behavior needs verification 93 + - Need to test end-to-end flow thoroughly 87 94 88 95 ### Next Steps 📋 89 96 ··· 133 140 134 141 ### Decision Graph Summary 135 142 136 - **Total nodes:** 284 143 + **Total nodes:** 288 137 144 **Key decisions tracked:** 138 145 - Environment configuration approach (#261-269) 139 146 - Port 8888 conflict resolution (#270-274) 140 147 - CORS permissions fix (#275-277) 141 148 - Storage approach: in-memory → database → proper auth flow (#278-284) 149 + - Refactor and build (#285-286) 150 + - Bug fixes: NaN parameter error (#287), database initialization (#288) 142 151 143 152 **Live graph:** https://notactuallytreyanastasio.github.io/deciduous/
+630
PLAN.md
··· 1 + # ATlast Twitter/X Support Plan 2 + 3 + ## Current Status (2025-12-26) 4 + 5 + **Phase 1 Status:** 🔧 Debugging - Core implementation complete, working through bugs 6 + 7 + **Recent Fixes:** 8 + - ✅ Environment configuration (dev/prod builds with correct API URLs) 9 + - ✅ Server health check and offline state handling 10 + - ✅ Authentication flow (session check before upload) 11 + - ✅ Removed temporary storage approach (extension_imports table) 12 + - ✅ Refactored to require login first (matches file upload flow) 13 + - ✅ Fixed NaN database error (missing matchedUsers parameter) 14 + - ✅ Database initialized for dev environment 15 + 16 + **Active Work:** 17 + - Debugging extension upload flow end-to-end 18 + - Testing results page integration 19 + - See [EXTENSION_STATUS.md](./EXTENSION_STATUS.md) for detailed status 20 + 21 + **Decision Graph:** 288 nodes tracked - [View live graph](https://notactuallytreyanastasio.github.io/deciduous/) 22 + 23 + --- 24 + 25 + ## Problem Statement 26 + 27 + Twitter/X data exports only contain `user_id` values, not usernames. Example: 28 + ``` 29 + https://twitter.com/intent/user?user_id=1103954565026775041 30 + ``` 31 + 32 + This makes data export files unusable for our existing parser-based workflow. We need a live scraping approach to extract usernames from the user's Following page. 33 + 34 + ## Research Findings 35 + 36 + ### Why Data Export Doesn't Work 37 + - Twitter exports contain only numeric `user_id` in URLs 38 + - Resolving `user_id` → `screen_name` requires API access ($42k/year Enterprise tier) or scraping 39 + - Nitter is dead (Feb 2024) - Twitter killed guest accounts 40 + - Third-party ID lookup tools don't support bulk/API access 41 + 42 + ### Live Scraping Approach 43 + Users are typically logged into Twitter. We can scrape usernames directly from the DOM of `x.com/following` using stable selectors: 44 + - `[data-testid="UserName"]` - stable, recommended 45 + - CSS class selectors - volatile, change frequently 46 + 47 + ### Platform Support Matrix 48 + 49 + | Platform | Extension Support | Bookmarklet JS | Solution | 50 + |----------|------------------|----------------|----------| 51 + | Desktop Chrome/Edge | Full | Yes | WebExtension | 52 + | Desktop Firefox | Full | Yes | WebExtension | 53 + | Desktop Safari | Full | Yes | WebExtension | 54 + | Android Firefox | Full | Yes | WebExtension | 55 + | Android Chrome | None | Via address bar | Recommend Firefox | 56 + | iOS Safari | Via App Store app | Blocked since iOS 15 | Safari Web Extension | 57 + 58 + ### iOS-Specific Findings 59 + 60 + **iOS Shortcuts "Run JavaScript on Webpage":** 61 + - CAN access authenticated Safari session via Share Sheet 62 + - BUT has strict timeout (few seconds) 63 + - Infinite scroll would timeout immediately 64 + - Only viable for grabbing currently visible content 65 + 66 + **iOS Safari Web Extensions (iOS 15+):** 67 + - Uses same WebExtensions API as Chrome/Firefox 68 + - Content scripts run without timeout limits 69 + - REQUIRES App Store distribution as part of iOS app 70 + - Full capability: auto-scroll, scrape, upload 71 + 72 + ## Architecture Decisions 73 + 74 + ### Monorepo Structure (pnpm workspaces) 75 + 76 + ``` 77 + ATlast/ 78 + ├── pnpm-workspace.yaml 79 + ├── package.json # Root workspace config 80 + ├── packages/ 81 + │ ├── web/ # Existing web app (moved from src/) 82 + │ │ ├── src/ 83 + │ │ ├── package.json 84 + │ │ └── vite.config.ts 85 + │ ├── extension/ # ATlast Importer browser extension 86 + │ │ ├── src/ 87 + │ │ ├── manifest.json 88 + │ │ ├── package.json 89 + │ │ └── build.config.ts 90 + │ ├── shared/ # Shared types and utilities 91 + │ │ ├── src/ 92 + │ │ │ ├── types/ 93 + │ │ │ │ ├── platform.ts # Platform enum, configs 94 + │ │ │ │ ├── import.ts # Import request/response types 95 + │ │ │ │ └── index.ts 96 + │ │ │ └── utils/ 97 + │ │ │ └── username.ts # Username normalization 98 + │ │ └── package.json 99 + │ └── functions/ # Netlify functions (moved from netlify/) 100 + │ ├── src/ 101 + │ ├── package.json 102 + │ └── tsconfig.json 103 + ├── netlify.toml 104 + └── docs/ # Decision graph output 105 + ``` 106 + 107 + ### Extension Name 108 + **ATlast Importer** - Clear purpose, searchable in extension stores. 109 + 110 + ### WebExtension Targets 111 + - Chrome/Edge (Manifest V3) 112 + - Firefox (Manifest V2/V3) 113 + - Safari (desktop + iOS via App Store wrapper) - deferred 114 + 115 + --- 116 + 117 + ## Extension Architecture 118 + 119 + ### High-Level Flow 120 + 121 + ``` 122 + ┌─────────────────────────────────────────────────────────────────┐ 123 + │ ATlast Browser Extension │ 124 + ├─────────────────────────────────────────────────────────────────┤ 125 + │ │ 126 + │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ 127 + │ │ Popup UI │ │ Content │ │ Background │ │ 128 + │ │ │◄──►│ Script │◄──►│ Service │ │ 129 + │ │ - Status │ │ │ │ Worker │ │ 130 + │ │ - Progress │ │ - Scrape │ │ │ │ 131 + │ │ - Actions │ │ - Scroll │ │ - Storage │ │ 132 + │ └──────────────┘ │ - Collect │ │ - Messaging │ │ 133 + │ └──────────────┘ └──────────────┘ │ 134 + │ │ 135 + └─────────────────────────────────────────────────────────────────┘ 136 + 137 + 138 + ┌──────────────────┐ 139 + │ ATlast Web App │ 140 + │ │ 141 + │ - Receive data │ 142 + │ - Search Bsky │ 143 + │ - Show matches │ 144 + └──────────────────┘ 145 + ``` 146 + 147 + ### Component Breakdown 148 + 149 + #### 1. Manifest Configuration 150 + ``` 151 + extension/ 152 + ├── manifest.json # Extension manifest (V3 for Chrome, V2 for Firefox) 153 + ├── manifest.firefox.json # Firefox-specific overrides (if needed) 154 + └── manifest.safari.json # Safari-specific overrides (if needed) 155 + ``` 156 + 157 + #### 2. Content Script (`content.js`) 158 + Injected into `x.com` / `twitter.com` pages. 159 + 160 + **Responsibilities:** 161 + - Detect if on Following/Followers page 162 + - Auto-scroll to load all users 163 + - Extract usernames using `[data-testid="UserName"]` 164 + - Report progress to popup/background 165 + - Handle rate limiting and pagination 166 + 167 + **Scraping Logic (pseudo-code):** 168 + ```javascript 169 + async function scrapeFollowing() { 170 + const usernames = new Set(); 171 + let lastCount = 0; 172 + let stableCount = 0; 173 + 174 + while (stableCount < 3) { // Stop after 3 scrolls with no new users 175 + // Collect visible usernames 176 + document.querySelectorAll('[data-testid="UserName"]').forEach(el => { 177 + const username = extractUsername(el); 178 + if (username) usernames.add(username); 179 + }); 180 + 181 + // Report progress 182 + sendProgress(usernames.size); 183 + 184 + // Scroll down 185 + window.scrollBy(0, 1000); 186 + await sleep(500); 187 + 188 + // Check if we found new users 189 + if (usernames.size === lastCount) { 190 + stableCount++; 191 + } else { 192 + stableCount = 0; 193 + lastCount = usernames.size; 194 + } 195 + } 196 + 197 + return Array.from(usernames); 198 + } 199 + ``` 200 + 201 + #### 3. Popup UI (`popup.html`, `popup.js`) 202 + User interface when clicking extension icon. 203 + 204 + **States:** 205 + - **Inactive**: "Go to x.com/following to start" 206 + - **Ready**: "Found Following page. Click to scan." 207 + - **Scanning**: Progress bar, count of found users 208 + - **Complete**: "Found 847 users. Open in ATlast" 209 + - **Error**: Error message with retry option 210 + 211 + #### 4. Background Service Worker (`background.js`) 212 + Coordinates between content script and popup. 213 + 214 + **Responsibilities:** 215 + - Store scraped data temporarily 216 + - Handle cross-tab communication 217 + - Manage extension state 218 + - Generate handoff URL/data for ATlast 219 + 220 + ### Data Handoff to ATlast 221 + 222 + **Decision: POST to API endpoint** 223 + 224 + Extension will POST scraped usernames to a new Netlify function endpoint. 225 + 226 + ``` 227 + ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ 228 + │ Extension │ POST │ Netlify Func │ Store │ Database │ 229 + │ │────────►│ /extension- │────────►│ │ 230 + │ usernames[] │ │ import │ │ pending_import │ 231 + │ platform: "x" │ │ │ │ │ 232 + └─────────────────┘ └────────┬────────┘ └─────────────────┘ 233 + 234 + │ Returns: { importId: "abc123" } 235 + 236 + ┌─────────────────┐ 237 + │ Redirect to │ 238 + │ atlast.app/ │ 239 + │ import/abc123 │ 240 + └─────────────────┘ 241 + ``` 242 + 243 + **API Endpoint: `POST /extension-import`** 244 + 245 + Request: 246 + ```json 247 + { 248 + "platform": "twitter", 249 + "usernames": ["user1", "user2", ...], 250 + "metadata": { 251 + "extensionVersion": "1.0.0", 252 + "scrapedAt": "2024-01-15T...", 253 + "pageType": "following" 254 + } 255 + } 256 + ``` 257 + 258 + Response: 259 + ```json 260 + { 261 + "importId": "abc123", 262 + "redirectUrl": "https://atlast.app/import/abc123" 263 + } 264 + ``` 265 + 266 + **Why POST over other options:** 267 + - No URL length limits (supports 10k+ usernames) 268 + - Secure (HTTPS, can add rate limiting) 269 + - Seamless UX (extension opens ATlast directly) 270 + - Audit trail (imports stored in database) 271 + 272 + ### Extension Package Structure (`packages/extension/`) 273 + 274 + ``` 275 + packages/extension/ 276 + ├── manifest.json # Base manifest (Chrome MV3) 277 + ├── manifest.firefox.json # Firefox overrides (if needed) 278 + ├── package.json # Extension package config 279 + ├── tsconfig.json 280 + ├── build.config.ts # Build script config 281 + ├── src/ 282 + │ ├── content/ 283 + │ │ ├── scrapers/ 284 + │ │ │ ├── base-scraper.ts # Abstract base class 285 + │ │ │ ├── twitter-scraper.ts # Twitter/X implementation 286 + │ │ │ ├── threads-scraper.ts # (Future) Threads 287 + │ │ │ ├── instagram-scraper.ts # (Future) Instagram 288 + │ │ │ └── tiktok-scraper.ts # (Future) TikTok 289 + │ │ ├── scroll-handler.ts # Generic infinite scroll 290 + │ │ └── index.ts # Content script entry, platform detection 291 + │ ├── popup/ 292 + │ │ ├── popup.html 293 + │ │ ├── popup.css 294 + │ │ └── popup.ts 295 + │ ├── background/ 296 + │ │ └── service-worker.ts 297 + │ └── lib/ 298 + │ ├── messaging.ts # Extension messaging 299 + │ ├── storage.ts # chrome.storage wrapper 300 + │ └── api-client.ts # POST to ATlast API 301 + ├── assets/ 302 + │ ├── icon-16.png 303 + │ ├── icon-48.png 304 + │ └── icon-128.png 305 + └── dist/ 306 + ├── chrome/ # Built extension for Chrome 307 + ├── firefox/ # Built extension for Firefox 308 + └── chrome.zip # Store submission package 309 + ``` 310 + 311 + ### Shared Package Structure (`packages/shared/`) 312 + 313 + ``` 314 + packages/shared/ 315 + ├── package.json 316 + ├── tsconfig.json 317 + ├── src/ 318 + │ ├── types/ 319 + │ │ ├── platform.ts # Platform enum, URL patterns 320 + │ │ ├── import.ts # ExtensionImportRequest, ExtensionImportResponse 321 + │ │ ├── scraper.ts # ScraperResult, ScraperProgress 322 + │ │ └── index.ts # Re-exports 323 + │ ├── utils/ 324 + │ │ ├── username.ts # normalizeUsername(), validateUsername() 325 + │ │ └── index.ts 326 + │ └── index.ts # Main entry 327 + └── dist/ # Compiled output 328 + ``` 329 + 330 + ### Shared Types Example 331 + 332 + ```typescript 333 + // packages/shared/src/types/platform.ts 334 + export enum Platform { 335 + Twitter = 'twitter', 336 + Threads = 'threads', 337 + Instagram = 'instagram', 338 + TikTok = 'tiktok', 339 + } 340 + 341 + export interface PlatformConfig { 342 + platform: Platform; 343 + displayName: string; 344 + hostPatterns: string[]; 345 + followingPathPattern: RegExp; 346 + iconUrl: string; 347 + } 348 + 349 + export const PLATFORM_CONFIGS: Record<Platform, PlatformConfig> = { 350 + [Platform.Twitter]: { 351 + platform: Platform.Twitter, 352 + displayName: 'Twitter/X', 353 + hostPatterns: ['twitter.com', 'x.com'], 354 + followingPathPattern: /^\/[^/]+\/following$/, 355 + iconUrl: 'https://abs.twimg.com/favicons/twitter.ico', 356 + }, 357 + // ... future platforms 358 + }; 359 + ``` 360 + 361 + ```typescript 362 + // packages/shared/src/types/import.ts 363 + import { Platform } from './platform'; 364 + 365 + export interface ExtensionImportRequest { 366 + platform: Platform; 367 + usernames: string[]; 368 + metadata: { 369 + extensionVersion: string; 370 + scrapedAt: string; 371 + pageType: 'following' | 'followers' | 'list'; 372 + sourceUrl: string; 373 + }; 374 + } 375 + 376 + export interface ExtensionImportResponse { 377 + importId: string; 378 + usernameCount: number; 379 + redirectUrl: string; 380 + } 381 + ``` 382 + 383 + ### Platform Detection & Extensibility 384 + 385 + Content script detects platform from URL and loads appropriate scraper: 386 + 387 + ```javascript 388 + // src/content/index.js 389 + const PLATFORM_PATTERNS = { 390 + twitter: { 391 + hostPatterns: ['twitter.com', 'x.com'], 392 + followingPath: /^\/[^/]+\/following$/, 393 + scraper: () => import('./scrapers/twitter-scraper.js') 394 + }, 395 + threads: { 396 + hostPatterns: ['threads.net'], 397 + followingPath: /^\/[^/]+\/following$/, 398 + scraper: () => import('./scrapers/threads-scraper.js') 399 + }, 400 + // ... future platforms 401 + }; 402 + 403 + function detectPlatform() { 404 + const host = window.location.hostname; 405 + const path = window.location.pathname; 406 + 407 + for (const [name, config] of Object.entries(PLATFORM_PATTERNS)) { 408 + if (config.hostPatterns.some(h => host.includes(h))) { 409 + if (config.followingPath.test(path)) { 410 + return { platform: name, pageType: 'following', ...config }; 411 + } 412 + } 413 + } 414 + return null; 415 + } 416 + ``` 417 + 418 + ### Base Scraper Interface 419 + 420 + ```javascript 421 + // src/content/scrapers/base-scraper.js 422 + export class BaseScraper { 423 + constructor(options = {}) { 424 + this.onProgress = options.onProgress || (() => {}); 425 + this.onComplete = options.onComplete || (() => {}); 426 + this.onError = options.onError || (() => {}); 427 + } 428 + 429 + // Must be implemented by subclasses 430 + getUsernameSelector() { throw new Error('Not implemented'); } 431 + extractUsername(element) { throw new Error('Not implemented'); } 432 + 433 + // Shared infinite scroll logic 434 + async scrape() { 435 + const usernames = new Set(); 436 + let stableCount = 0; 437 + 438 + while (stableCount < 3) { 439 + const before = usernames.size; 440 + 441 + document.querySelectorAll(this.getUsernameSelector()).forEach(el => { 442 + const username = this.extractUsername(el); 443 + if (username) usernames.add(username); 444 + }); 445 + 446 + this.onProgress({ count: usernames.size }); 447 + 448 + window.scrollBy(0, 1000); 449 + await this.sleep(500); 450 + 451 + stableCount = (usernames.size === before) ? stableCount + 1 : 0; 452 + } 453 + 454 + this.onComplete({ usernames: Array.from(usernames) }); 455 + return Array.from(usernames); 456 + } 457 + 458 + sleep(ms) { 459 + return new Promise(resolve => setTimeout(resolve, ms)); 460 + } 461 + } 462 + ``` 463 + 464 + ### Twitter Scraper Implementation 465 + 466 + ```javascript 467 + // src/content/scrapers/twitter-scraper.js 468 + import { BaseScraper } from './base-scraper.js'; 469 + 470 + export class TwitterScraper extends BaseScraper { 471 + getUsernameSelector() { 472 + // Primary selector (stable) 473 + return '[data-testid="UserName"]'; 474 + } 475 + 476 + extractUsername(element) { 477 + // UserName element contains display name and @handle 478 + // Structure: <div><span>Display Name</span></div><div><span>@handle</span></div> 479 + const spans = element.querySelectorAll('span'); 480 + for (const span of spans) { 481 + const text = span.textContent?.trim(); 482 + if (text?.startsWith('@')) { 483 + return text.slice(1).toLowerCase(); // Remove @ prefix 484 + } 485 + } 486 + return null; 487 + } 488 + } 489 + ``` 490 + 491 + ### iOS App Wrapper (Future) 492 + 493 + For iOS Safari extension, need minimal Swift app: 494 + 495 + ``` 496 + ATlastApp/ 497 + ├── ATlast/ 498 + │ ├── ATlastApp.swift # Minimal app entry 499 + │ ├── ContentView.swift # Simple "Open Safari" UI 500 + │ └── Info.plist 501 + ├── ATlast Extension/ 502 + │ ├── SafariWebExtensionHandler.swift 503 + │ ├── Info.plist 504 + │ └── Resources/ 505 + │ └── (same extension files as above) 506 + └── ATlast.xcodeproj 507 + ``` 508 + 509 + --- 510 + 511 + ## Decisions Made 512 + 513 + | Question | Decision | Rationale | 514 + |----------|----------|-----------| 515 + | **Data Handoff** | POST to API endpoint | No size limits, seamless UX, audit trail | 516 + | **MVP Scope** | Twitter Following page only | Fastest path to value | 517 + | **iOS Priority** | Deferred | Focus on desktop Chrome/Firefox first | 518 + | **Platform Scope** | Twitter v1, architecture for multi-platform | Plan for Threads/Instagram/TikTok later | 519 + | **Extension Name** | ATlast Importer | Clear purpose, searchable in stores | 520 + | **Code Location** | Monorepo with pnpm workspaces | Clean shared types, isolated builds | 521 + | **Monorepo Tool** | pnpm workspaces | Fast, disk-efficient, minimal config | 522 + 523 + ## Remaining Questions 524 + 525 + ### Q1: Extension Branding 526 + - Name options: "ATlast", "ATlast Importer", "ATlast Social Bridge" 527 + - Icon design needed 528 + 529 + ### Q2: Error Recovery Strategy 530 + Twitter/X changes DOM frequently. Strategy for handling breaks: 531 + - Ship updates quickly when breaks detected 532 + - Build selector fallback chain 533 + - User-reportable "not working" flow 534 + - **Recommendation: All of the above** 535 + 536 + ### Q3: Extension Store Distribution 537 + - Chrome Web Store (requires $5 developer fee) 538 + - Firefox Add-ons (free) 539 + - Safari Extensions (requires Apple Developer account, $99/year - defer with iOS) 540 + 541 + --- 542 + 543 + ## Implementation Phases 544 + 545 + ### Phase 0: Monorepo Migration ✅ COMPLETE 546 + - [x] **0.1** Install pnpm globally if needed 547 + - [x] **0.2** Create pnpm-workspace.yaml 548 + - [x] **0.3** Create packages/ directory structure 549 + - [x] **0.4** Move src/ → packages/web/src/ 550 + - [x] **0.5** Move netlify/functions/ → packages/functions/ 551 + - [x] **0.6** Create packages/shared/ with types 552 + - [x] **0.7** Update import paths in web and functions 553 + - [x] **0.8** Update netlify.toml for new paths 554 + - [x] **0.9** Update root package.json scripts 555 + - [x] **0.10** Test build and dev commands 556 + - [x] **0.11** Commit monorepo migration 557 + 558 + ### Phase 1: Chrome Extension MVP 🔧 IN PROGRESS (Debugging) 559 + - [x] **1.1** Create packages/extension/ structure 560 + - [x] **1.2** Write manifest.json (Manifest V3) 561 + - [x] **1.3** Implement base-scraper.ts abstract class 562 + - [x] **1.4** Implement twitter-scraper.ts 563 + - [x] **1.5** Implement content/index.ts (platform detection) 564 + - [x] **1.6** Implement popup UI (HTML/CSS/TS) 565 + - [x] **1.7** Implement background service worker 566 + - [x] **1.8** Implement api-client.ts (POST to ATlast) 567 + - [x] **1.9** Create Netlify function: extension-import.ts 568 + - [x] **1.10** ~~Create ATlast import page: /import/[id]~~ (Not needed - uses /results?uploadId) 569 + - [x] **1.11** Add extension build script 570 + - [ ] **1.12** Test end-to-end flow locally (Active debugging) 571 + - [ ] **1.13** Chrome Web Store submission 572 + 573 + ### Phase 2: Firefox Support 574 + - [ ] **2.1** Create manifest.firefox.json (MV2 if needed) 575 + - [ ] **2.2** Test on Firefox desktop 576 + - [ ] **2.3** Test on Firefox Android 577 + - [ ] **2.4** Firefox Add-ons submission 578 + 579 + ### Phase 3: Enhanced Twitter Features 580 + - [ ] **3.1** Support Followers page 581 + - [ ] **3.2** Support Twitter Lists 582 + - [ ] **3.3** Add selector fallback chain 583 + - [ ] **3.4** Add user-reportable error flow 584 + 585 + ### Phase 4: Additional Platforms (Future) 586 + - [ ] **4.1** Threads scraper 587 + - [ ] **4.2** Instagram scraper 588 + - [ ] **4.3** TikTok scraper 589 + 590 + ### Phase 5: iOS Support (Future) 591 + - [ ] **5.1** iOS app wrapper (Swift) 592 + - [ ] **5.2** Safari Web Extension integration 593 + - [ ] **5.3** App Store submission 594 + 595 + --- 596 + 597 + ## Related Decision Graph Nodes 598 + 599 + - **Goal**: #184 (Support Twitter/X file uploads) 600 + - **Problem Analysis**: #185-186 (user_id issue, resolution approach decision) 601 + - **Initial Options**: #187-192 (server-side, extension, CLI, BYOK, hybrid) 602 + - **Research**: #193-204 (Nitter dead, Sky Follower Bridge, DOM scraping) 603 + - **iOS Research**: #212-216 (Shortcuts timeout, Safari Web Extensions) 604 + - **Architecture Decisions**: #218-222 605 + - #219: POST to API endpoint 606 + - #220: Twitter Following page MVP 607 + - #221: iOS deferred 608 + - #222: Multi-platform architecture 609 + - **Implementation Decisions**: #224-227 610 + - #225: Monorepo with shared packages 611 + - #226: Extension name "ATlast Importer" 612 + - #227: pnpm workspaces tooling 613 + 614 + View live graph: https://notactuallytreyanastasio.github.io/deciduous/ 615 + 616 + --- 617 + 618 + ## Changelog 619 + 620 + | Date | Change | 621 + |------|--------| 622 + | 2024-12-25 | Initial plan created with research findings and architecture | 623 + | 2024-12-25 | Decisions made: POST API, Twitter MVP, iOS deferred, extensible architecture | 624 + | 2024-12-25 | Added: Extension name (ATlast Importer), monorepo structure (pnpm workspaces) | 625 + | 2024-12-25 | Added: Phase 0 (monorepo migration), detailed package structures, shared types | 626 + | 2025-12-26 | Phase 0 complete (monorepo migration) | 627 + | 2025-12-26 | Phase 1 nearly complete - core implementation done, active debugging | 628 + | 2025-12-26 | Architecture refactored: extension requires login first, uses /results?uploadId | 629 + | 2025-12-26 | Fixed: NaN database error, environment config, auth flow, CORS permissions | 630 + | 2025-12-26 | Decision graph: 288 nodes tracked |
+27 -19
docs/git-history.json
··· 1 1 [ 2 2 { 3 - "hash": "1a355fe785eb1768dba3f4c3a8ba631904d1d6d6", 4 - "short_hash": "1a355fe", 3 + "hash": "34bd9dcd1237971a87627b148c0452b8484e4871", 4 + "short_hash": "34bd9dc", 5 + "author": "Ariel M. Lighty", 6 + "date": "2025-12-26T00:50:44-05:00", 7 + "message": "update documentation with current debugging status\n\nPLAN.md updates:\n- Added current status section with recent fixes and active work\n- Marked Phase 0 as complete\n- Marked Phase 1 as in progress (debugging)\n- Updated changelog with 2025-12-26 progress\n- Updated decision graph count to 288 nodes\n\nEXTENSION_STATUS.md updates:\n- Changed state from READY FOR TESTING to DEBUGGING\n- Added fixed issues section (NaN bug, database init)\n- Added active debugging section\n- Updated decision graph summary to 288 nodes\n- Added node references for recent fixes (#287-288)\n\nDecision graph:\n- Synced with latest nodes (288 total, 276 edges)\n- Tracked database initialization outcome", 8 + "files_changed": 4 9 + }, 10 + { 11 + "hash": "a8a4b0a819297dec7b92d02036699cc86b2523ce", 12 + "short_hash": "a8a4b0a", 5 13 "author": "Ariel M. Lighty", 6 14 "date": "2025-12-26T00:33:21-05:00", 7 15 "message": "fix extension-import: add missing matchedUsers parameter to createUpload\n\nThe createUpload method expects 5 parameters but we were only passing 4,\ncausing NaN to be inserted for unmatched_users calculation. Now passing 0\nfor matchedUsers (will be updated after search is performed).", 8 - "files_changed": 1 16 + "files_changed": 3 9 17 }, 10 18 { 11 - "hash": "b00ecad7a2c576962cb78de5b50f301e4664830e", 12 - "short_hash": "b00ecad", 19 + "hash": "1a355fe785eb1768dba3f4c3a8ba631904d1d6d6", 20 + "short_hash": "1a355fe", 13 21 "author": "Ariel M. Lighty", 14 - "date": "2025-12-26T00:26:09-05:00", 15 - "message": "refactor extension to require authentication and use proper upload flow\n\nRemoved temporary storage approach and implemented proper authentication flow:\n\nExtension changes:\n- Added session check to popup init flow (checkSession in api-client)\n- Added \"not logged in\" state with login prompts\n- Updated uploadToATlast to include credentials for cookie-based auth\n- Extension now requires user to be logged in BEFORE scanning\n\nBackend changes:\n- Converted extension-import to AuthenticatedHandler (requires auth)\n- Now creates upload records immediately (no temporary storage)\n- Removed extension_imports table from database schema\n- Deleted get-extension-import function (no longer needed)\n- Deleted import-store utility (temporary approach removed)\n\nFrontend changes:\n- Removed ExtensionImport page and /import/:id route\n- Extension uploads now use same flow as file uploads\n\nThis matches the correct user flow: user logs in to ATlast first, then\nextension creates permanent upload records directly (same as file uploads).\n\nBuilt extension successfully for dev environment.", 16 - "files_changed": 12 22 + "date": "2025-12-26T00:33:21-05:00", 23 + "message": "fix extension-import: add missing matchedUsers parameter to createUpload\n\nThe createUpload method expects 5 parameters but we were only passing 4,\ncausing NaN to be inserted for unmatched_users calculation. Now passing 0\nfor matchedUsers (will be updated after search is performed).", 24 + "files_changed": 1 17 25 }, 18 26 { 19 27 "hash": "d0bcf337b6d223a86443f6f67767e87b74e4dd7d", ··· 24 32 "files_changed": 12 25 33 }, 26 34 { 27 - "hash": "90067b8feb9f40512b9732a05b1a8756f338a481", 28 - "short_hash": "90067b8", 35 + "hash": "6b352a5609e1934fab14caadd4f5545d09412018", 36 + "short_hash": "6b352a5", 29 37 "author": "Ariel M. Lighty", 30 - "date": "2025-12-25T21:43:51-05:00", 31 - "message": "add dev server health check and offline detection to extension\n\nFeatures:\n- Check server health on popup init (dev mode only)\n- Show 'server offline' state with setup instructions\n- 'Check Again' button to retry connection\n- Display target server URL for debugging\n- 3-second timeout for health checks\n\nFixes port 8888 conflict workflow - extension now prompts user\nto start dev server instead of hanging silently.", 32 - "files_changed": 4 38 + "date": "2025-12-25T21:44:00-05:00", 39 + "message": "update decision graph", 40 + "files_changed": 2 33 41 }, 34 42 { 35 43 "hash": "c35fb0d83202607facc203dfe10325e8672ea67e", ··· 48 56 "files_changed": 3 49 57 }, 50 58 { 51 - "hash": "ba29fd68872913ba0a587aa7f29f97b3d373a732", 52 - "short_hash": "ba29fd6", 59 + "hash": "32cdee3aeac7ef986df47e0fff786b5f7471e55b", 60 + "short_hash": "32cdee3", 53 61 "author": "Ariel M. Lighty", 54 62 "date": "2025-12-25T13:22:32-05:00", 55 63 "message": "configure Netlify dev for monorepo with --filter flag\n\nFixed Netlify CLI monorepo detection issue by using --filter flag:\n- Updated root package.json scripts to use 'npx netlify-cli dev --filter @atlast/web'\n- Updated netlify.toml [dev] section to use npm with --prefix for framework command\n- Added monorepo development instructions to CLAUDE.md\n- Documented Windows Git Bash compatibility issue with netlify command\n\nSolution: Use 'npx netlify-cli dev --filter @atlast/web' to bypass monorepo\nproject selection prompt and specify which workspace package to run.\n\nDev server now runs successfully at http://localhost:8888 with all backend\nfunctions loaded.", 56 - "files_changed": 5 64 + "files_changed": 4 57 65 }, 58 66 { 59 - "hash": "32cdee3aeac7ef986df47e0fff786b5f7471e55b", 60 - "short_hash": "32cdee3", 67 + "hash": "ba29fd68872913ba0a587aa7f29f97b3d373a732", 68 + "short_hash": "ba29fd6", 61 69 "author": "Ariel M. Lighty", 62 70 "date": "2025-12-25T13:22:32-05:00", 63 71 "message": "configure Netlify dev for monorepo with --filter flag\n\nFixed Netlify CLI monorepo detection issue by using --filter flag:\n- Updated root package.json scripts to use 'npx netlify-cli dev --filter @atlast/web'\n- Updated netlify.toml [dev] section to use npm with --prefix for framework command\n- Added monorepo development instructions to CLAUDE.md\n- Documented Windows Git Bash compatibility issue with netlify command\n\nSolution: Use 'npx netlify-cli dev --filter @atlast/web' to bypass monorepo\nproject selection prompt and specify which workspace package to run.\n\nDev server now runs successfully at http://localhost:8888 with all backend\nfunctions loaded.", 64 - "files_changed": 4 72 + "files_changed": 5 65 73 }, 66 74 { 67 75 "hash": "c3e7afad396d130791d801a85cbfc9643bcd6309",
+44
docs/graph-data.json
··· 3156 3156 "created_at": "2025-12-26T00:33:28.860934100-05:00", 3157 3157 "updated_at": "2025-12-26T00:33:28.860934100-05:00", 3158 3158 "metadata_json": "{\"branch\":\"master\",\"commit\":\"1a355fe\",\"confidence\":95,\"files\":[\"packages/functions/src/extension-import.ts\"]}" 3159 + }, 3160 + { 3161 + "id": 288, 3162 + "change_id": "5fa82fdc-7796-4263-be72-e1877279881b", 3163 + "node_type": "outcome", 3164 + "title": "Database initialized successfully for dev environment", 3165 + "description": null, 3166 + "status": "pending", 3167 + "created_at": "2025-12-26T00:47:09.629444300-05:00", 3168 + "updated_at": "2025-12-26T00:47:09.629444300-05:00", 3169 + "metadata_json": "{\"branch\":\"master\",\"confidence\":100,\"files\":[\"packages/functions/src/init-db.ts\"]}" 3170 + }, 3171 + { 3172 + "id": 289, 3173 + "change_id": "dd2aa029-7ca9-4379-a966-762c9137bcc8", 3174 + "node_type": "action", 3175 + "title": "Updated PLAN.md and EXTENSION_STATUS.md with current debugging status", 3176 + "description": null, 3177 + "status": "pending", 3178 + "created_at": "2025-12-26T00:50:51.291667400-05:00", 3179 + "updated_at": "2025-12-26T00:50:51.291667400-05:00", 3180 + "metadata_json": "{\"branch\":\"master\",\"commit\":\"34bd9dc\",\"confidence\":100,\"files\":[\"PLAN.md\",\"EXTENSION_STATUS.md\"]}" 3159 3181 } 3160 3182 ], 3161 3183 "edges": [ ··· 6183 6205 "weight": 1.0, 6184 6206 "rationale": "Found and fixed bug during testing", 6185 6207 "created_at": "2025-12-26T00:33:34.834072700-05:00" 6208 + }, 6209 + { 6210 + "id": 276, 6211 + "from_node_id": 287, 6212 + "to_node_id": 288, 6213 + "from_change_id": "e01c6989-6c0b-42f8-b7c7-60aca059f7c3", 6214 + "to_change_id": "5fa82fdc-7796-4263-be72-e1877279881b", 6215 + "edge_type": "leads_to", 6216 + "weight": 1.0, 6217 + "rationale": "Initialized database after fixing parameter bug", 6218 + "created_at": "2025-12-26T00:47:11.206746500-05:00" 6219 + }, 6220 + { 6221 + "id": 277, 6222 + "from_node_id": 288, 6223 + "to_node_id": 289, 6224 + "from_change_id": "5fa82fdc-7796-4263-be72-e1877279881b", 6225 + "to_change_id": "dd2aa029-7ca9-4379-a966-762c9137bcc8", 6226 + "edge_type": "leads_to", 6227 + "weight": 1.0, 6228 + "rationale": "Documented current state after fixes", 6229 + "created_at": "2025-12-26T00:50:58.391390400-05:00" 6186 6230 } 6187 6231 ] 6188 6232 }