ATlast — you'll never need to find your favorites on another platform again. Find your favs in the ATmosphere.
byarielm.fyi
atproto
1# ATlast Twitter/X Support Plan
2
3## Current Status (2025-12-26)
4
5**Phase 1 Status:** 🔧 Debugging - Core implementation complete, working through bugs
6
7**Recent Fixes:**
8- ✅ Environment configuration (dev/prod builds with correct API URLs)
9- ✅ Server health check and offline state handling
10- ✅ Authentication flow (session check before upload)
11- ✅ Removed temporary storage approach (extension_imports table)
12- ✅ Refactored to require login first (matches file upload flow)
13- ✅ Fixed NaN database error (missing matchedUsers parameter)
14- ✅ Database initialized for dev environment
15
16**Active Work:**
17- Debugging extension upload flow end-to-end
18- Testing results page integration
19- See [EXTENSION_STATUS.md](./EXTENSION_STATUS.md) for detailed status
20
21**Decision Graph:** 288 nodes tracked - [View live graph](https://notactuallytreyanastasio.github.io/deciduous/)
22
23---
24
25## Problem Statement
26
27Twitter/X data exports only contain `user_id` values, not usernames. Example:
28```
29https://twitter.com/intent/user?user_id=1103954565026775041
30```
31
32This makes data export files unusable for our existing parser-based workflow. We need a live scraping approach to extract usernames from the user's Following page.
33
34## Research Findings
35
36### Why Data Export Doesn't Work
37- Twitter exports contain only numeric `user_id` in URLs
38- Resolving `user_id` → `screen_name` requires API access ($42k/year Enterprise tier) or scraping
39- Nitter is dead (Feb 2024) - Twitter killed guest accounts
40- Third-party ID lookup tools don't support bulk/API access
41
42### Live Scraping Approach
43Users are typically logged into Twitter. We can scrape usernames directly from the DOM of `x.com/following` using stable selectors:
44- `[data-testid="UserName"]` - stable, recommended
45- CSS class selectors - volatile, change frequently
46
47### Platform Support Matrix
48
49| Platform | Extension Support | Bookmarklet JS | Solution |
50|----------|------------------|----------------|----------|
51| Desktop Chrome/Edge | Full | Yes | WebExtension |
52| Desktop Firefox | Full | Yes | WebExtension |
53| Desktop Safari | Full | Yes | WebExtension |
54| Android Firefox | Full | Yes | WebExtension |
55| Android Chrome | None | Via address bar | Recommend Firefox |
56| iOS Safari | Via App Store app | Blocked since iOS 15 | Safari Web Extension |
57
58### iOS-Specific Findings
59
60**iOS Shortcuts "Run JavaScript on Webpage":**
61- CAN access authenticated Safari session via Share Sheet
62- BUT has strict timeout (few seconds)
63- Infinite scroll would timeout immediately
64- Only viable for grabbing currently visible content
65
66**iOS Safari Web Extensions (iOS 15+):**
67- Uses same WebExtensions API as Chrome/Firefox
68- Content scripts run without timeout limits
69- REQUIRES App Store distribution as part of iOS app
70- Full capability: auto-scroll, scrape, upload
71
72## Architecture Decisions
73
74### Monorepo Structure (pnpm workspaces)
75
76```
77ATlast/
78├── pnpm-workspace.yaml
79├── package.json # Root workspace config
80├── packages/
81│ ├── web/ # Existing web app (moved from src/)
82│ │ ├── src/
83│ │ ├── package.json
84│ │ └── vite.config.ts
85│ ├── extension/ # ATlast Importer browser extension
86│ │ ├── src/
87│ │ ├── manifest.json
88│ │ ├── package.json
89│ │ └── build.config.ts
90│ ├── shared/ # Shared types and utilities
91│ │ ├── src/
92│ │ │ ├── types/
93│ │ │ │ ├── platform.ts # Platform enum, configs
94│ │ │ │ ├── import.ts # Import request/response types
95│ │ │ │ └── index.ts
96│ │ │ └── utils/
97│ │ │ └── username.ts # Username normalization
98│ │ └── package.json
99│ └── functions/ # Netlify functions (moved from netlify/)
100│ ├── src/
101│ ├── package.json
102│ └── tsconfig.json
103├── netlify.toml
104└── docs/ # Decision graph output
105```
106
107### Extension Name
108**ATlast Importer** - Clear purpose, searchable in extension stores.
109
110### WebExtension Targets
111- Chrome/Edge (Manifest V3)
112- Firefox (Manifest V2/V3)
113- Safari (desktop + iOS via App Store wrapper) - deferred
114
115---
116
117## Extension Architecture
118
119### High-Level Flow
120
121```
122┌─────────────────────────────────────────────────────────────────┐
123│ ATlast Browser Extension │
124├─────────────────────────────────────────────────────────────────┤
125│ │
126│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
127│ │ Popup UI │ │ Content │ │ Background │ │
128│ │ │◄──►│ Script │◄──►│ Service │ │
129│ │ - Status │ │ │ │ Worker │ │
130│ │ - Progress │ │ - Scrape │ │ │ │
131│ │ - Actions │ │ - Scroll │ │ - Storage │ │
132│ └──────────────┘ │ - Collect │ │ - Messaging │ │
133│ └──────────────┘ └──────────────┘ │
134│ │
135└─────────────────────────────────────────────────────────────────┘
136 │
137 ▼
138 ┌──────────────────┐
139 │ ATlast Web App │
140 │ │
141 │ - Receive data │
142 │ - Search Bsky │
143 │ - Show matches │
144 └──────────────────┘
145```
146
147### Component Breakdown
148
149#### 1. Manifest Configuration
150```
151extension/
152├── manifest.json # Extension manifest (V3 for Chrome, V2 for Firefox)
153├── manifest.firefox.json # Firefox-specific overrides (if needed)
154└── manifest.safari.json # Safari-specific overrides (if needed)
155```
156
157#### 2. Content Script (`content.js`)
158Injected into `x.com` / `twitter.com` pages.
159
160**Responsibilities:**
161- Detect if on Following/Followers page
162- Auto-scroll to load all users
163- Extract usernames using `[data-testid="UserName"]`
164- Report progress to popup/background
165- Handle rate limiting and pagination
166
167**Scraping Logic (pseudo-code):**
168```javascript
169async function scrapeFollowing() {
170 const usernames = new Set();
171 let lastCount = 0;
172 let stableCount = 0;
173
174 while (stableCount < 3) { // Stop after 3 scrolls with no new users
175 // Collect visible usernames
176 document.querySelectorAll('[data-testid="UserName"]').forEach(el => {
177 const username = extractUsername(el);
178 if (username) usernames.add(username);
179 });
180
181 // Report progress
182 sendProgress(usernames.size);
183
184 // Scroll down
185 window.scrollBy(0, 1000);
186 await sleep(500);
187
188 // Check if we found new users
189 if (usernames.size === lastCount) {
190 stableCount++;
191 } else {
192 stableCount = 0;
193 lastCount = usernames.size;
194 }
195 }
196
197 return Array.from(usernames);
198}
199```
200
201#### 3. Popup UI (`popup.html`, `popup.js`)
202User interface when clicking extension icon.
203
204**States:**
205- **Inactive**: "Go to x.com/following to start"
206- **Ready**: "Found Following page. Click to scan."
207- **Scanning**: Progress bar, count of found users
208- **Complete**: "Found 847 users. Open in ATlast"
209- **Error**: Error message with retry option
210
211#### 4. Background Service Worker (`background.js`)
212Coordinates between content script and popup.
213
214**Responsibilities:**
215- Store scraped data temporarily
216- Handle cross-tab communication
217- Manage extension state
218- Generate handoff URL/data for ATlast
219
220### Data Handoff to ATlast
221
222**Decision: POST to API endpoint**
223
224Extension will POST scraped usernames to a new Netlify function endpoint.
225
226```
227┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
228│ Extension │ POST │ Netlify Func │ Store │ Database │
229│ │────────►│ /extension- │────────►│ │
230│ usernames[] │ │ import │ │ pending_import │
231│ platform: "x" │ │ │ │ │
232└─────────────────┘ └────────┬────────┘ └─────────────────┘
233 │
234 │ Returns: { importId: "abc123" }
235 ▼
236 ┌─────────────────┐
237 │ Redirect to │
238 │ atlast.app/ │
239 │ import/abc123 │
240 └─────────────────┘
241```
242
243**API Endpoint: `POST /extension-import`**
244
245Request:
246```json
247{
248 "platform": "twitter",
249 "usernames": ["user1", "user2", ...],
250 "metadata": {
251 "extensionVersion": "1.0.0",
252 "scrapedAt": "2024-01-15T...",
253 "pageType": "following"
254 }
255}
256```
257
258Response:
259```json
260{
261 "importId": "abc123",
262 "redirectUrl": "https://atlast.app/import/abc123"
263}
264```
265
266**Why POST over other options:**
267- No URL length limits (supports 10k+ usernames)
268- Secure (HTTPS, can add rate limiting)
269- Seamless UX (extension opens ATlast directly)
270- Audit trail (imports stored in database)
271
272### Extension Package Structure (`packages/extension/`)
273
274```
275packages/extension/
276├── manifest.json # Base manifest (Chrome MV3)
277├── manifest.firefox.json # Firefox overrides (if needed)
278├── package.json # Extension package config
279├── tsconfig.json
280├── build.config.ts # Build script config
281├── src/
282│ ├── content/
283│ │ ├── scrapers/
284│ │ │ ├── base-scraper.ts # Abstract base class
285│ │ │ ├── twitter-scraper.ts # Twitter/X implementation
286│ │ │ ├── threads-scraper.ts # (Future) Threads
287│ │ │ ├── instagram-scraper.ts # (Future) Instagram
288│ │ │ └── tiktok-scraper.ts # (Future) TikTok
289│ │ ├── scroll-handler.ts # Generic infinite scroll
290│ │ └── index.ts # Content script entry, platform detection
291│ ├── popup/
292│ │ ├── popup.html
293│ │ ├── popup.css
294│ │ └── popup.ts
295│ ├── background/
296│ │ └── service-worker.ts
297│ └── lib/
298│ ├── messaging.ts # Extension messaging
299│ ├── storage.ts # chrome.storage wrapper
300│ └── api-client.ts # POST to ATlast API
301├── assets/
302│ ├── icon-16.png
303│ ├── icon-48.png
304│ └── icon-128.png
305└── dist/
306 ├── chrome/ # Built extension for Chrome
307 ├── firefox/ # Built extension for Firefox
308 └── chrome.zip # Store submission package
309```
310
311### Shared Package Structure (`packages/shared/`)
312
313```
314packages/shared/
315├── package.json
316├── tsconfig.json
317├── src/
318│ ├── types/
319│ │ ├── platform.ts # Platform enum, URL patterns
320│ │ ├── import.ts # ExtensionImportRequest, ExtensionImportResponse
321│ │ ├── scraper.ts # ScraperResult, ScraperProgress
322│ │ └── index.ts # Re-exports
323│ ├── utils/
324│ │ ├── username.ts # normalizeUsername(), validateUsername()
325│ │ └── index.ts
326│ └── index.ts # Main entry
327└── dist/ # Compiled output
328```
329
330### Shared Types Example
331
332```typescript
333// packages/shared/src/types/platform.ts
334export enum Platform {
335 Twitter = 'twitter',
336 Threads = 'threads',
337 Instagram = 'instagram',
338 TikTok = 'tiktok',
339}
340
341export interface PlatformConfig {
342 platform: Platform;
343 displayName: string;
344 hostPatterns: string[];
345 followingPathPattern: RegExp;
346 iconUrl: string;
347}
348
349export const PLATFORM_CONFIGS: Record<Platform, PlatformConfig> = {
350 [Platform.Twitter]: {
351 platform: Platform.Twitter,
352 displayName: 'Twitter/X',
353 hostPatterns: ['twitter.com', 'x.com'],
354 followingPathPattern: /^\/[^/]+\/following$/,
355 iconUrl: 'https://abs.twimg.com/favicons/twitter.ico',
356 },
357 // ... future platforms
358};
359```
360
361```typescript
362// packages/shared/src/types/import.ts
363import { Platform } from './platform';
364
365export interface ExtensionImportRequest {
366 platform: Platform;
367 usernames: string[];
368 metadata: {
369 extensionVersion: string;
370 scrapedAt: string;
371 pageType: 'following' | 'followers' | 'list';
372 sourceUrl: string;
373 };
374}
375
376export interface ExtensionImportResponse {
377 importId: string;
378 usernameCount: number;
379 redirectUrl: string;
380}
381```
382
383### Platform Detection & Extensibility
384
385Content script detects platform from URL and loads appropriate scraper:
386
387```javascript
388// src/content/index.js
389const PLATFORM_PATTERNS = {
390 twitter: {
391 hostPatterns: ['twitter.com', 'x.com'],
392 followingPath: /^\/[^/]+\/following$/,
393 scraper: () => import('./scrapers/twitter-scraper.js')
394 },
395 threads: {
396 hostPatterns: ['threads.net'],
397 followingPath: /^\/[^/]+\/following$/,
398 scraper: () => import('./scrapers/threads-scraper.js')
399 },
400 // ... future platforms
401};
402
403function detectPlatform() {
404 const host = window.location.hostname;
405 const path = window.location.pathname;
406
407 for (const [name, config] of Object.entries(PLATFORM_PATTERNS)) {
408 if (config.hostPatterns.some(h => host.includes(h))) {
409 if (config.followingPath.test(path)) {
410 return { platform: name, pageType: 'following', ...config };
411 }
412 }
413 }
414 return null;
415}
416```
417
418### Base Scraper Interface
419
420```javascript
421// src/content/scrapers/base-scraper.js
422export class BaseScraper {
423 constructor(options = {}) {
424 this.onProgress = options.onProgress || (() => {});
425 this.onComplete = options.onComplete || (() => {});
426 this.onError = options.onError || (() => {});
427 }
428
429 // Must be implemented by subclasses
430 getUsernameSelector() { throw new Error('Not implemented'); }
431 extractUsername(element) { throw new Error('Not implemented'); }
432
433 // Shared infinite scroll logic
434 async scrape() {
435 const usernames = new Set();
436 let stableCount = 0;
437
438 while (stableCount < 3) {
439 const before = usernames.size;
440
441 document.querySelectorAll(this.getUsernameSelector()).forEach(el => {
442 const username = this.extractUsername(el);
443 if (username) usernames.add(username);
444 });
445
446 this.onProgress({ count: usernames.size });
447
448 window.scrollBy(0, 1000);
449 await this.sleep(500);
450
451 stableCount = (usernames.size === before) ? stableCount + 1 : 0;
452 }
453
454 this.onComplete({ usernames: Array.from(usernames) });
455 return Array.from(usernames);
456 }
457
458 sleep(ms) {
459 return new Promise(resolve => setTimeout(resolve, ms));
460 }
461}
462```
463
464### Twitter Scraper Implementation
465
466```javascript
467// src/content/scrapers/twitter-scraper.js
468import { BaseScraper } from './base-scraper.js';
469
470export class TwitterScraper extends BaseScraper {
471 getUsernameSelector() {
472 // Primary selector (stable)
473 return '[data-testid="UserName"]';
474 }
475
476 extractUsername(element) {
477 // UserName element contains display name and @handle
478 // Structure: <div><span>Display Name</span></div><div><span>@handle</span></div>
479 const spans = element.querySelectorAll('span');
480 for (const span of spans) {
481 const text = span.textContent?.trim();
482 if (text?.startsWith('@')) {
483 return text.slice(1).toLowerCase(); // Remove @ prefix
484 }
485 }
486 return null;
487 }
488}
489```
490
491### iOS App Wrapper (Future)
492
493For iOS Safari extension, need minimal Swift app:
494
495```
496ATlastApp/
497├── ATlast/
498│ ├── ATlastApp.swift # Minimal app entry
499│ ├── ContentView.swift # Simple "Open Safari" UI
500│ └── Info.plist
501├── ATlast Extension/
502│ ├── SafariWebExtensionHandler.swift
503│ ├── Info.plist
504│ └── Resources/
505│ └── (same extension files as above)
506└── ATlast.xcodeproj
507```
508
509---
510
511## Decisions Made
512
513| Question | Decision | Rationale |
514|----------|----------|-----------|
515| **Data Handoff** | POST to API endpoint | No size limits, seamless UX, audit trail |
516| **MVP Scope** | Twitter Following page only | Fastest path to value |
517| **iOS Priority** | Deferred | Focus on desktop Chrome/Firefox first |
518| **Platform Scope** | Twitter v1, architecture for multi-platform | Plan for Threads/Instagram/TikTok later |
519| **Extension Name** | ATlast Importer | Clear purpose, searchable in stores |
520| **Code Location** | Monorepo with pnpm workspaces | Clean shared types, isolated builds |
521| **Monorepo Tool** | pnpm workspaces | Fast, disk-efficient, minimal config |
522
523## Remaining Questions
524
525### Q1: Extension Branding
526- Name options: "ATlast", "ATlast Importer", "ATlast Social Bridge"
527- Icon design needed
528
529### Q2: Error Recovery Strategy
530Twitter/X changes DOM frequently. Strategy for handling breaks:
531- Ship updates quickly when breaks detected
532- Build selector fallback chain
533- User-reportable "not working" flow
534- **Recommendation: All of the above**
535
536### Q3: Extension Store Distribution
537- Chrome Web Store (requires $5 developer fee)
538- Firefox Add-ons (free)
539- Safari Extensions (requires Apple Developer account, $99/year - defer with iOS)
540
541---
542
543## Implementation Phases
544
545### Phase 0: Monorepo Migration ✅ COMPLETE
546- [x] **0.1** Install pnpm globally if needed
547- [x] **0.2** Create pnpm-workspace.yaml
548- [x] **0.3** Create packages/ directory structure
549- [x] **0.4** Move src/ → packages/web/src/
550- [x] **0.5** Move netlify/functions/ → packages/functions/
551- [x] **0.6** Create packages/shared/ with types
552- [x] **0.7** Update import paths in web and functions
553- [x] **0.8** Update netlify.toml for new paths
554- [x] **0.9** Update root package.json scripts
555- [x] **0.10** Test build and dev commands
556- [x] **0.11** Commit monorepo migration
557
558### Phase 1: Chrome Extension MVP 🔧 IN PROGRESS (Debugging)
559- [x] **1.1** Create packages/extension/ structure
560- [x] **1.2** Write manifest.json (Manifest V3)
561- [x] **1.3** Implement base-scraper.ts abstract class
562- [x] **1.4** Implement twitter-scraper.ts
563- [x] **1.5** Implement content/index.ts (platform detection)
564- [x] **1.6** Implement popup UI (HTML/CSS/TS)
565- [x] **1.7** Implement background service worker
566- [x] **1.8** Implement api-client.ts (POST to ATlast)
567- [x] **1.9** Create Netlify function: extension-import.ts
568- [x] **1.10** ~~Create ATlast import page: /import/[id]~~ (Not needed - uses /results?uploadId)
569- [x] **1.11** Add extension build script
570- [ ] **1.12** Test end-to-end flow locally (Active debugging)
571- [ ] **1.13** Chrome Web Store submission
572
573### Phase 2: Firefox Support
574- [ ] **2.1** Create manifest.firefox.json (MV2 if needed)
575- [ ] **2.2** Test on Firefox desktop
576- [ ] **2.3** Test on Firefox Android
577- [ ] **2.4** Firefox Add-ons submission
578
579### Phase 3: Enhanced Twitter Features
580- [ ] **3.1** Support Followers page
581- [ ] **3.2** Support Twitter Lists
582- [ ] **3.3** Add selector fallback chain
583- [ ] **3.4** Add user-reportable error flow
584
585### Phase 4: Additional Platforms (Future)
586- [ ] **4.1** Threads scraper
587- [ ] **4.2** Instagram scraper
588- [ ] **4.3** TikTok scraper
589
590### Phase 5: iOS Support (Future)
591- [ ] **5.1** iOS app wrapper (Swift)
592- [ ] **5.2** Safari Web Extension integration
593- [ ] **5.3** App Store submission
594
595---
596
597## Related Decision Graph Nodes
598
599- **Goal**: #184 (Support Twitter/X file uploads)
600- **Problem Analysis**: #185-186 (user_id issue, resolution approach decision)
601- **Initial Options**: #187-192 (server-side, extension, CLI, BYOK, hybrid)
602- **Research**: #193-204 (Nitter dead, Sky Follower Bridge, DOM scraping)
603- **iOS Research**: #212-216 (Shortcuts timeout, Safari Web Extensions)
604- **Architecture Decisions**: #218-222
605 - #219: POST to API endpoint
606 - #220: Twitter Following page MVP
607 - #221: iOS deferred
608 - #222: Multi-platform architecture
609- **Implementation Decisions**: #224-227
610 - #225: Monorepo with shared packages
611 - #226: Extension name "ATlast Importer"
612 - #227: pnpm workspaces tooling
613
614View live graph: https://notactuallytreyanastasio.github.io/deciduous/
615
616---
617
618## Changelog
619
620| Date | Change |
621|------|--------|
622| 2024-12-25 | Initial plan created with research findings and architecture |
623| 2024-12-25 | Decisions made: POST API, Twitter MVP, iOS deferred, extensible architecture |
624| 2024-12-25 | Added: Extension name (ATlast Importer), monorepo structure (pnpm workspaces) |
625| 2024-12-25 | Added: Phase 0 (monorepo migration), detailed package structures, shared types |
626| 2025-12-26 | Phase 0 complete (monorepo migration) |
627| 2025-12-26 | Phase 1 nearly complete - core implementation done, active debugging |
628| 2025-12-26 | Architecture refactored: extension requires login first, uses /results?uploadId |
629| 2025-12-26 | Fixed: NaN database error, environment config, auth flow, CORS permissions |
630| 2025-12-26 | Decision graph: 288 nodes tracked |