byarielm.fyi
ATlast — you'll never need to find your favorites on another platform again. Find your favs in the ATmosphere.
atproto
1# ATlast Twitter/X Support Plan
2
3## Current Status (2025-12-26)
4
5**Phase 1 Status:** ✅ Ready for Testing - Core implementation complete, all bugs fixed
6
7**Recent Fixes:**
8- ✅ Environment configuration (dev/prod builds with correct API URLs)
9- ✅ Server health check and offline state handling
10- ✅ Authentication flow (session check before upload)
11- ✅ Removed temporary storage approach (extension_imports table)
12- ✅ Refactored to require login first (matches file upload flow)
13- ✅ Fixed NaN database error (missing matchedUsers parameter)
14- ✅ Database initialized for dev environment
15- ✅ Fixed API response unwrapping (uploadToATlast and checkSession)
16
17**Active Work:**
18- End-to-end testing of complete flow
19- Verification of results page integration
20- See [EXTENSION_STATUS.md](./EXTENSION_STATUS.md) for detailed status
21
22**Decision Graph:** 295 nodes tracked - [View live graph](https://notactuallytreyanastasio.github.io/deciduous/)
23
24---
25
26## Problem Statement
27
28Twitter/X data exports only contain `user_id` values, not usernames. Example:
29```
30https://twitter.com/intent/user?user_id=1103954565026775041
31```
32
33This makes data export files unusable for our existing parser-based workflow. We need a live scraping approach to extract usernames from the user's Following page.
34
35## Research Findings
36
37### Why Data Export Doesn't Work
38- Twitter exports contain only numeric `user_id` in URLs
39- Resolving `user_id` → `screen_name` requires API access ($42k/year Enterprise tier) or scraping
40- Nitter is dead (Feb 2024) - Twitter killed guest accounts
41- Third-party ID lookup tools don't support bulk/API access
42
43### Live Scraping Approach
44Users are typically logged into Twitter. We can scrape usernames directly from the DOM of `x.com/following` using stable selectors:
45- `[data-testid="UserName"]` - stable, recommended
46- CSS class selectors - volatile, change frequently
47
48### Platform Support Matrix
49
50| Platform | Extension Support | Bookmarklet JS | Solution |
51|----------|------------------|----------------|----------|
52| Desktop Chrome/Edge | Full | Yes | WebExtension |
53| Desktop Firefox | Full | Yes | WebExtension |
54| Desktop Safari | Full | Yes | WebExtension |
55| Android Firefox | Full | Yes | WebExtension |
56| Android Chrome | None | Via address bar | Recommend Firefox |
57| iOS Safari | Via App Store app | Blocked since iOS 15 | Safari Web Extension |
58
59### iOS-Specific Findings
60
61**iOS Shortcuts "Run JavaScript on Webpage":**
62- CAN access authenticated Safari session via Share Sheet
63- BUT has strict timeout (few seconds)
64- Infinite scroll would timeout immediately
65- Only viable for grabbing currently visible content
66
67**iOS Safari Web Extensions (iOS 15+):**
68- Uses same WebExtensions API as Chrome/Firefox
69- Content scripts run without timeout limits
70- REQUIRES App Store distribution as part of iOS app
71- Full capability: auto-scroll, scrape, upload
72
73## Architecture Decisions
74
75### Monorepo Structure (pnpm workspaces)
76
77```
78ATlast/
79├── pnpm-workspace.yaml
80├── package.json # Root workspace config
81├── packages/
82│ ├── web/ # Existing web app (moved from src/)
83│ │ ├── src/
84│ │ ├── package.json
85│ │ └── vite.config.ts
86│ ├── extension/ # ATlast Importer browser extension
87│ │ ├── src/
88│ │ ├── manifest.json
89│ │ ├── package.json
90│ │ └── build.config.ts
91│ ├── shared/ # Shared types and utilities
92│ │ ├── src/
93│ │ │ ├── types/
94│ │ │ │ ├── platform.ts # Platform enum, configs
95│ │ │ │ ├── import.ts # Import request/response types
96│ │ │ │ └── index.ts
97│ │ │ └── utils/
98│ │ │ └── username.ts # Username normalization
99│ │ └── package.json
100│ └── functions/ # Netlify functions (moved from netlify/)
101│ ├── src/
102│ ├── package.json
103│ └── tsconfig.json
104├── netlify.toml
105└── docs/ # Decision graph output
106```
107
108### Extension Name
109**ATlast Importer** - Clear purpose, searchable in extension stores.
110
111### WebExtension Targets
112- Chrome/Edge (Manifest V3)
113- Firefox (Manifest V2/V3)
114- Safari (desktop + iOS via App Store wrapper) - deferred
115
116---
117
118## Extension Architecture
119
120### High-Level Flow
121
122```
123┌─────────────────────────────────────────────────────────────────┐
124│ ATlast Browser Extension │
125├─────────────────────────────────────────────────────────────────┤
126│ │
127│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
128│ │ Popup UI │ │ Content │ │ Background │ │
129│ │ │◄──►│ Script │◄──►│ Service │ │
130│ │ - Status │ │ │ │ Worker │ │
131│ │ - Progress │ │ - Scrape │ │ │ │
132│ │ - Actions │ │ - Scroll │ │ - Storage │ │
133│ └──────────────┘ │ - Collect │ │ - Messaging │ │
134│ └──────────────┘ └──────────────┘ │
135│ │
136└─────────────────────────────────────────────────────────────────┘
137 │
138 ▼
139 ┌──────────────────┐
140 │ ATlast Web App │
141 │ │
142 │ - Receive data │
143 │ - Search Bsky │
144 │ - Show matches │
145 └──────────────────┘
146```
147
148### Component Breakdown
149
150#### 1. Manifest Configuration
151```
152extension/
153├── manifest.json # Extension manifest (V3 for Chrome, V2 for Firefox)
154├── manifest.firefox.json # Firefox-specific overrides (if needed)
155└── manifest.safari.json # Safari-specific overrides (if needed)
156```
157
158#### 2. Content Script (`content.js`)
159Injected into `x.com` / `twitter.com` pages.
160
161**Responsibilities:**
162- Detect if on Following/Followers page
163- Auto-scroll to load all users
164- Extract usernames using `[data-testid="UserName"]`
165- Report progress to popup/background
166- Handle rate limiting and pagination
167
168**Scraping Logic (pseudo-code):**
169```javascript
170async function scrapeFollowing() {
171 const usernames = new Set();
172 let lastCount = 0;
173 let stableCount = 0;
174
175 while (stableCount < 3) { // Stop after 3 scrolls with no new users
176 // Collect visible usernames
177 document.querySelectorAll('[data-testid="UserName"]').forEach(el => {
178 const username = extractUsername(el);
179 if (username) usernames.add(username);
180 });
181
182 // Report progress
183 sendProgress(usernames.size);
184
185 // Scroll down
186 window.scrollBy(0, 1000);
187 await sleep(500);
188
189 // Check if we found new users
190 if (usernames.size === lastCount) {
191 stableCount++;
192 } else {
193 stableCount = 0;
194 lastCount = usernames.size;
195 }
196 }
197
198 return Array.from(usernames);
199}
200```
201
202#### 3. Popup UI (`popup.html`, `popup.js`)
203User interface when clicking extension icon.
204
205**States:**
206- **Inactive**: "Go to x.com/following to start"
207- **Ready**: "Found Following page. Click to scan."
208- **Scanning**: Progress bar, count of found users
209- **Complete**: "Found 847 users. Open in ATlast"
210- **Error**: Error message with retry option
211
212#### 4. Background Service Worker (`background.js`)
213Coordinates between content script and popup.
214
215**Responsibilities:**
216- Store scraped data temporarily
217- Handle cross-tab communication
218- Manage extension state
219- Generate handoff URL/data for ATlast
220
221### Data Handoff to ATlast
222
223**Decision: POST to API endpoint**
224
225Extension will POST scraped usernames to a new Netlify function endpoint.
226
227```
228┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
229│ Extension │ POST │ Netlify Func │ Store │ Database │
230│ │────────►│ /extension- │────────►│ │
231│ usernames[] │ │ import │ │ pending_import │
232│ platform: "x" │ │ │ │ │
233└─────────────────┘ └────────┬────────┘ └─────────────────┘
234 │
235 │ Returns: { importId: "abc123" }
236 ▼
237 ┌─────────────────┐
238 │ Redirect to │
239 │ atlast.app/ │
240 │ import/abc123 │
241 └─────────────────┘
242```
243
244**API Endpoint: `POST /extension-import`**
245
246Request:
247```json
248{
249 "platform": "twitter",
250 "usernames": ["user1", "user2", ...],
251 "metadata": {
252 "extensionVersion": "1.0.0",
253 "scrapedAt": "2024-01-15T...",
254 "pageType": "following"
255 }
256}
257```
258
259Response:
260```json
261{
262 "importId": "abc123",
263 "redirectUrl": "https://atlast.app/import/abc123"
264}
265```
266
267**Why POST over other options:**
268- No URL length limits (supports 10k+ usernames)
269- Secure (HTTPS, can add rate limiting)
270- Seamless UX (extension opens ATlast directly)
271- Audit trail (imports stored in database)
272
273### Extension Package Structure (`packages/extension/`)
274
275```
276packages/extension/
277├── manifest.json # Base manifest (Chrome MV3)
278├── manifest.firefox.json # Firefox overrides (if needed)
279├── package.json # Extension package config
280├── tsconfig.json
281├── build.config.ts # Build script config
282├── src/
283│ ├── content/
284│ │ ├── scrapers/
285│ │ │ ├── base-scraper.ts # Abstract base class
286│ │ │ ├── twitter-scraper.ts # Twitter/X implementation
287│ │ │ ├── threads-scraper.ts # (Future) Threads
288│ │ │ ├── instagram-scraper.ts # (Future) Instagram
289│ │ │ └── tiktok-scraper.ts # (Future) TikTok
290│ │ ├── scroll-handler.ts # Generic infinite scroll
291│ │ └── index.ts # Content script entry, platform detection
292│ ├── popup/
293│ │ ├── popup.html
294│ │ ├── popup.css
295│ │ └── popup.ts
296│ ├── background/
297│ │ └── service-worker.ts
298│ └── lib/
299│ ├── messaging.ts # Extension messaging
300│ ├── storage.ts # chrome.storage wrapper
301│ └── api-client.ts # POST to ATlast API
302├── assets/
303│ ├── icon-16.png
304│ ├── icon-48.png
305│ └── icon-128.png
306└── dist/
307 ├── chrome/ # Built extension for Chrome
308 ├── firefox/ # Built extension for Firefox
309 └── chrome.zip # Store submission package
310```
311
312### Shared Package Structure (`packages/shared/`)
313
314```
315packages/shared/
316├── package.json
317├── tsconfig.json
318├── src/
319│ ├── types/
320│ │ ├── platform.ts # Platform enum, URL patterns
321│ │ ├── import.ts # ExtensionImportRequest, ExtensionImportResponse
322│ │ ├── scraper.ts # ScraperResult, ScraperProgress
323│ │ └── index.ts # Re-exports
324│ ├── utils/
325│ │ ├── username.ts # normalizeUsername(), validateUsername()
326│ │ └── index.ts
327│ └── index.ts # Main entry
328└── dist/ # Compiled output
329```
330
331### Shared Types Example
332
333```typescript
334// packages/shared/src/types/platform.ts
335export enum Platform {
336 Twitter = 'twitter',
337 Threads = 'threads',
338 Instagram = 'instagram',
339 TikTok = 'tiktok',
340}
341
342export interface PlatformConfig {
343 platform: Platform;
344 displayName: string;
345 hostPatterns: string[];
346 followingPathPattern: RegExp;
347 iconUrl: string;
348}
349
350export const PLATFORM_CONFIGS: Record<Platform, PlatformConfig> = {
351 [Platform.Twitter]: {
352 platform: Platform.Twitter,
353 displayName: 'Twitter/X',
354 hostPatterns: ['twitter.com', 'x.com'],
355 followingPathPattern: /^\/[^/]+\/following$/,
356 iconUrl: 'https://abs.twimg.com/favicons/twitter.ico',
357 },
358 // ... future platforms
359};
360```
361
362```typescript
363// packages/shared/src/types/import.ts
364import { Platform } from './platform';
365
366export interface ExtensionImportRequest {
367 platform: Platform;
368 usernames: string[];
369 metadata: {
370 extensionVersion: string;
371 scrapedAt: string;
372 pageType: 'following' | 'followers' | 'list';
373 sourceUrl: string;
374 };
375}
376
377export interface ExtensionImportResponse {
378 importId: string;
379 usernameCount: number;
380 redirectUrl: string;
381}
382```
383
384### Platform Detection & Extensibility
385
386Content script detects platform from URL and loads appropriate scraper:
387
388```javascript
389// src/content/index.js
390const PLATFORM_PATTERNS = {
391 twitter: {
392 hostPatterns: ['twitter.com', 'x.com'],
393 followingPath: /^\/[^/]+\/following$/,
394 scraper: () => import('./scrapers/twitter-scraper.js')
395 },
396 threads: {
397 hostPatterns: ['threads.net'],
398 followingPath: /^\/[^/]+\/following$/,
399 scraper: () => import('./scrapers/threads-scraper.js')
400 },
401 // ... future platforms
402};
403
404function detectPlatform() {
405 const host = window.location.hostname;
406 const path = window.location.pathname;
407
408 for (const [name, config] of Object.entries(PLATFORM_PATTERNS)) {
409 if (config.hostPatterns.some(h => host.includes(h))) {
410 if (config.followingPath.test(path)) {
411 return { platform: name, pageType: 'following', ...config };
412 }
413 }
414 }
415 return null;
416}
417```
418
419### Base Scraper Interface
420
421```javascript
422// src/content/scrapers/base-scraper.js
423export class BaseScraper {
424 constructor(options = {}) {
425 this.onProgress = options.onProgress || (() => {});
426 this.onComplete = options.onComplete || (() => {});
427 this.onError = options.onError || (() => {});
428 }
429
430 // Must be implemented by subclasses
431 getUsernameSelector() { throw new Error('Not implemented'); }
432 extractUsername(element) { throw new Error('Not implemented'); }
433
434 // Shared infinite scroll logic
435 async scrape() {
436 const usernames = new Set();
437 let stableCount = 0;
438
439 while (stableCount < 3) {
440 const before = usernames.size;
441
442 document.querySelectorAll(this.getUsernameSelector()).forEach(el => {
443 const username = this.extractUsername(el);
444 if (username) usernames.add(username);
445 });
446
447 this.onProgress({ count: usernames.size });
448
449 window.scrollBy(0, 1000);
450 await this.sleep(500);
451
452 stableCount = (usernames.size === before) ? stableCount + 1 : 0;
453 }
454
455 this.onComplete({ usernames: Array.from(usernames) });
456 return Array.from(usernames);
457 }
458
459 sleep(ms) {
460 return new Promise(resolve => setTimeout(resolve, ms));
461 }
462}
463```
464
465### Twitter Scraper Implementation
466
467```javascript
468// src/content/scrapers/twitter-scraper.js
469import { BaseScraper } from './base-scraper.js';
470
471export class TwitterScraper extends BaseScraper {
472 getUsernameSelector() {
473 // Primary selector (stable)
474 return '[data-testid="UserName"]';
475 }
476
477 extractUsername(element) {
478 // UserName element contains display name and @handle
479 // Structure: <div><span>Display Name</span></div><div><span>@handle</span></div>
480 const spans = element.querySelectorAll('span');
481 for (const span of spans) {
482 const text = span.textContent?.trim();
483 if (text?.startsWith('@')) {
484 return text.slice(1).toLowerCase(); // Remove @ prefix
485 }
486 }
487 return null;
488 }
489}
490```
491
492### iOS App Wrapper (Future)
493
494For iOS Safari extension, need minimal Swift app:
495
496```
497ATlastApp/
498├── ATlast/
499│ ├── ATlastApp.swift # Minimal app entry
500│ ├── ContentView.swift # Simple "Open Safari" UI
501│ └── Info.plist
502├── ATlast Extension/
503│ ├── SafariWebExtensionHandler.swift
504│ ├── Info.plist
505│ └── Resources/
506│ └── (same extension files as above)
507└── ATlast.xcodeproj
508```
509
510---
511
512## Decisions Made
513
514| Question | Decision | Rationale |
515|----------|----------|-----------|
516| **Data Handoff** | POST to API endpoint | No size limits, seamless UX, audit trail |
517| **MVP Scope** | Twitter Following page only | Fastest path to value |
518| **iOS Priority** | Deferred | Focus on desktop Chrome/Firefox first |
519| **Platform Scope** | Twitter v1, architecture for multi-platform | Plan for Threads/Instagram/TikTok later |
520| **Extension Name** | ATlast Importer | Clear purpose, searchable in stores |
521| **Code Location** | Monorepo with pnpm workspaces | Clean shared types, isolated builds |
522| **Monorepo Tool** | pnpm workspaces | Fast, disk-efficient, minimal config |
523
524## Remaining Questions
525
526### Q1: Extension Branding
527- Name options: "ATlast", "ATlast Importer", "ATlast Social Bridge"
528- Icon design needed
529
530### Q2: Error Recovery Strategy
531Twitter/X changes DOM frequently. Strategy for handling breaks:
532- Ship updates quickly when breaks detected
533- Build selector fallback chain
534- User-reportable "not working" flow
535- **Recommendation: All of the above**
536
537### Q3: Extension Store Distribution
538- Chrome Web Store (requires $5 developer fee)
539- Firefox Add-ons (free)
540- Safari Extensions (requires Apple Developer account, $99/year - defer with iOS)
541
542---
543
544## Implementation Phases
545
546### Phase 0: Monorepo Migration ✅ COMPLETE
547- [x] **0.1** Install pnpm globally if needed
548- [x] **0.2** Create pnpm-workspace.yaml
549- [x] **0.3** Create packages/ directory structure
550- [x] **0.4** Move src/ → packages/web/src/
551- [x] **0.5** Move netlify/functions/ → packages/functions/
552- [x] **0.6** Create packages/shared/ with types
553- [x] **0.7** Update import paths in web and functions
554- [x] **0.8** Update netlify.toml for new paths
555- [x] **0.9** Update root package.json scripts
556- [x] **0.10** Test build and dev commands
557- [x] **0.11** Commit monorepo migration
558
559### Phase 1: Chrome Extension MVP 🔧 IN PROGRESS (Debugging)
560- [x] **1.1** Create packages/extension/ structure
561- [x] **1.2** Write manifest.json (Manifest V3)
562- [x] **1.3** Implement base-scraper.ts abstract class
563- [x] **1.4** Implement twitter-scraper.ts
564- [x] **1.5** Implement content/index.ts (platform detection)
565- [x] **1.6** Implement popup UI (HTML/CSS/TS)
566- [x] **1.7** Implement background service worker
567- [x] **1.8** Implement api-client.ts (POST to ATlast)
568- [x] **1.9** Create Netlify function: extension-import.ts
569- [x] **1.10** ~~Create ATlast import page: /import/[id]~~ (Not needed - uses /results?uploadId)
570- [x] **1.11** Add extension build script
571- [ ] **1.12** Test end-to-end flow locally (Active debugging)
572- [ ] **1.13** Chrome Web Store submission
573
574### Phase 2: Firefox Support
575- [ ] **2.1** Create manifest.firefox.json (MV2 if needed)
576- [ ] **2.2** Test on Firefox desktop
577- [ ] **2.3** Test on Firefox Android
578- [ ] **2.4** Firefox Add-ons submission
579
580### Phase 3: Enhanced Twitter Features
581- [ ] **3.1** Support Followers page
582- [ ] **3.2** Support Twitter Lists
583- [ ] **3.3** Add selector fallback chain
584- [ ] **3.4** Add user-reportable error flow
585
586### Phase 4: Additional Platforms (Future)
587- [ ] **4.1** Threads scraper
588- [ ] **4.2** Instagram scraper
589- [ ] **4.3** TikTok scraper
590
591### Phase 5: iOS Support (Future)
592- [ ] **5.1** iOS app wrapper (Swift)
593- [ ] **5.2** Safari Web Extension integration
594- [ ] **5.3** App Store submission
595
596---
597
598## Related Decision Graph Nodes
599
600- **Goal**: #184 (Support Twitter/X file uploads)
601- **Problem Analysis**: #185-186 (user_id issue, resolution approach decision)
602- **Initial Options**: #187-192 (server-side, extension, CLI, BYOK, hybrid)
603- **Research**: #193-204 (Nitter dead, Sky Follower Bridge, DOM scraping)
604- **iOS Research**: #212-216 (Shortcuts timeout, Safari Web Extensions)
605- **Architecture Decisions**: #218-222
606 - #219: POST to API endpoint
607 - #220: Twitter Following page MVP
608 - #221: iOS deferred
609 - #222: Multi-platform architecture
610- **Implementation Decisions**: #224-227
611 - #225: Monorepo with shared packages
612 - #226: Extension name "ATlast Importer"
613 - #227: pnpm workspaces tooling
614
615View live graph: https://notactuallytreyanastasio.github.io/deciduous/
616
617---
618
619## Changelog
620
621| Date | Change |
622|------|--------|
623| 2024-12-25 | Initial plan created with research findings and architecture |
624| 2024-12-25 | Decisions made: POST API, Twitter MVP, iOS deferred, extensible architecture |
625| 2024-12-25 | Added: Extension name (ATlast Importer), monorepo structure (pnpm workspaces) |
626| 2024-12-25 | Added: Phase 0 (monorepo migration), detailed package structures, shared types |
627| 2025-12-26 | Phase 0 complete (monorepo migration) |
628| 2025-12-26 | Phase 1 nearly complete - core implementation done, active debugging |
629| 2025-12-26 | Architecture refactored: extension requires login first, uses /results?uploadId |
630| 2025-12-26 | Fixed: NaN database error, environment config, auth flow, CORS permissions |
631| 2025-12-26 | Fixed: API response unwrapping - extension now correctly handles ApiResponse structure |
632| 2025-12-26 | Phase 1 ready for testing - all bugs resolved, decision graph: 295 nodes tracked |