ATlast Twitter/X Support Plan#

Current Status (2025-12-27)#

Phase 1 Status: ✅ COMPLETE - Ready for production testing and Chrome Web Store submission

All Completed (Dec 2024 - Jan 2025):

• ✅ Environment configuration (dev/prod builds with correct API URLs)
• ✅ Server health check and offline state handling
• ✅ Authentication flow (session check before upload)
• ✅ Removed temporary storage approach (extension_imports table)
• ✅ Refactored to require login first (matches file upload flow)
• ✅ Fixed NaN database error (missing matchedUsers parameter)
• ✅ Database initialized for dev environment
• ✅ Fixed API response unwrapping (uploadToATlast and checkSession)
• ✅ Loading screen during extension upload search
• ✅ Timezone fixes with TIMESTAMPTZ
• ✅ Vite dev server optimization
• ✅ Decision graph integrity fixes (18 orphan nodes resolved)
• ✅ Documentation improvements (CLAUDE.md with lifecycle management)

Ready For:

• Production testing
• Chrome Web Store submission
• Firefox Add-ons development

Decision Graph: 332 nodes, 333 edges - View live graph

Problem Statement#

Twitter/X data exports only contain user_id values, not usernames. Example:

https://twitter.com/intent/user?user_id=1103954565026775041

This makes data export files unusable for our existing parser-based workflow. We need a live scraping approach to extract usernames from the user's Following page.

Research Findings#

Why Data Export Doesn't Work#

• Twitter exports contain only numeric user_id in URLs
• Resolving user_id → screen_name requires API access ($42k/year Enterprise tier) or scraping
• Nitter is dead (Feb 2024) - Twitter killed guest accounts
• Third-party ID lookup tools don't support bulk/API access

Live Scraping Approach#

Users are typically logged into Twitter. We can scrape usernames directly from the DOM of x.com/following using stable selectors:

• [data-testid="UserName"] - stable, recommended
• CSS class selectors - volatile, change frequently

Platform Support Matrix#

iOS-Specific Findings#

iOS Shortcuts "Run JavaScript on Webpage":

• CAN access authenticated Safari session via Share Sheet
• BUT has strict timeout (few seconds)
• Infinite scroll would timeout immediately
• Only viable for grabbing currently visible content

iOS Safari Web Extensions (iOS 15+):

• Uses same WebExtensions API as Chrome/Firefox
• Content scripts run without timeout limits
• REQUIRES App Store distribution as part of iOS app
• Full capability: auto-scroll, scrape, upload

Architecture Decisions#

Monorepo Structure (pnpm workspaces)#

ATlast/
├── pnpm-workspace.yaml
├── package.json                      # Root workspace config
├── packages/
│   ├── web/                          # Existing web app (moved from src/)
│   │   ├── src/
│   │   ├── package.json
│   │   └── vite.config.ts
│   ├── extension/                    # ATlast Importer browser extension
│   │   ├── src/
│   │   ├── manifest.json
│   │   ├── package.json
│   │   └── build.config.ts
│   ├── shared/                       # Shared types and utilities
│   │   ├── src/
│   │   │   ├── types/
│   │   │   │   ├── platform.ts       # Platform enum, configs
│   │   │   │   ├── import.ts         # Import request/response types
│   │   │   │   └── index.ts
│   │   │   └── utils/
│   │   │       └── username.ts       # Username normalization
│   │   └── package.json
│   └── functions/                    # Netlify functions (moved from netlify/)
│       ├── src/
│       ├── package.json
│       └── tsconfig.json
├── netlify.toml
└── docs/                             # Decision graph output

Extension Name#

ATlast Importer - Clear purpose, searchable in extension stores.

WebExtension Targets#

• Chrome/Edge (Manifest V3)
• Firefox (Manifest V2/V3)
• Safari (desktop + iOS via App Store wrapper) - deferred

Extension Architecture#

High-Level Flow#

┌─────────────────────────────────────────────────────────────────┐
│                     ATlast Browser Extension                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐       │
│  │   Popup UI   │    │   Content    │    │  Background  │       │
│  │              │◄──►│   Script     │◄──►│   Service    │       │
│  │  - Status    │    │              │    │   Worker     │       │
│  │  - Progress  │    │  - Scrape    │    │              │       │
│  │  - Actions   │    │  - Scroll    │    │  - Storage   │       │
│  └──────────────┘    │  - Collect   │    │  - Messaging │       │
│                      └──────────────┘    └──────────────┘       │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │   ATlast Web App │
                    │                  │
                    │  - Receive data  │
                    │  - Search Bsky   │
                    │  - Show matches  │
                    └──────────────────┘

Component Breakdown#

1. Manifest Configuration#

extension/
├── manifest.json          # Extension manifest (V3 for Chrome, V2 for Firefox)
├── manifest.firefox.json  # Firefox-specific overrides (if needed)
└── manifest.safari.json   # Safari-specific overrides (if needed)

2. Content Script (content.js)#

Injected into x.com / twitter.com pages.

Responsibilities:

• Detect if on Following/Followers page
• Auto-scroll to load all users
• Extract usernames using [data-testid="UserName"]
• Report progress to popup/background
• Handle rate limiting and pagination

Scraping Logic (pseudo-code):

async function scrapeFollowing() {
  const usernames = new Set();
  let lastCount = 0;
  let stableCount = 0;

  while (stableCount < 3) {  // Stop after 3 scrolls with no new users
    // Collect visible usernames
    document.querySelectorAll('[data-testid="UserName"]').forEach(el => {
      const username = extractUsername(el);
      if (username) usernames.add(username);
    });

    // Report progress
    sendProgress(usernames.size);

    // Scroll down
    window.scrollBy(0, 1000);
    await sleep(500);

    // Check if we found new users
    if (usernames.size === lastCount) {
      stableCount++;
    } else {
      stableCount = 0;
      lastCount = usernames.size;
    }
  }

  return Array.from(usernames);
}

3. Popup UI (popup.html, popup.js)#

User interface when clicking extension icon.

States:

• Inactive: "Go to x.com/following to start"
• Ready: "Found Following page. Click to scan."
• Scanning: Progress bar, count of found users
• Complete: "Found 847 users. Open in ATlast"
• Error: Error message with retry option

4. Background Service Worker (background.js)#

Coordinates between content script and popup.

Responsibilities:

• Store scraped data temporarily
• Handle cross-tab communication
• Manage extension state
• Generate handoff URL/data for ATlast

Data Handoff to ATlast#

Decision: POST to API endpoint

Extension will POST scraped usernames to a new Netlify function endpoint.

┌─────────────────┐         ┌─────────────────┐         ┌─────────────────┐
│    Extension    │  POST   │  Netlify Func   │  Store  │    Database     │
│                 │────────►│  /extension-    │────────►│                 │
│  usernames[]    │         │   import        │         │  pending_import │
│  platform: "x"  │         │                 │         │                 │
└─────────────────┘         └────────┬────────┘         └─────────────────┘
                                     │
                                     │ Returns: { importId: "abc123" }
                                     ▼
                            ┌─────────────────┐
                            │   Redirect to   │
                            │  atlast.app/    │
                            │  import/abc123  │
                            └─────────────────┘

API Endpoint: POST /extension-import

Request:

{
  "platform": "twitter",
  "usernames": ["user1", "user2", ...],
  "metadata": {
    "extensionVersion": "1.0.0",
    "scrapedAt": "2024-01-15T...",
    "pageType": "following"
  }
}

Response:

{
  "importId": "abc123",
  "redirectUrl": "https://atlast.app/import/abc123"
}

Why POST over other options:

• No URL length limits (supports 10k+ usernames)
• Secure (HTTPS, can add rate limiting)
• Seamless UX (extension opens ATlast directly)
• Audit trail (imports stored in database)

Extension Package Structure (packages/extension/)#

packages/extension/
├── manifest.json                    # Base manifest (Chrome MV3)
├── manifest.firefox.json            # Firefox overrides (if needed)
├── package.json                     # Extension package config
├── tsconfig.json
├── build.config.ts                  # Build script config
├── src/
│   ├── content/
│   │   ├── scrapers/
│   │   │   ├── base-scraper.ts      # Abstract base class
│   │   │   ├── twitter-scraper.ts   # Twitter/X implementation
│   │   │   ├── threads-scraper.ts   # (Future) Threads
│   │   │   ├── instagram-scraper.ts # (Future) Instagram
│   │   │   └── tiktok-scraper.ts    # (Future) TikTok
│   │   ├── scroll-handler.ts        # Generic infinite scroll
│   │   └── index.ts                 # Content script entry, platform detection
│   ├── popup/
│   │   ├── popup.html
│   │   ├── popup.css
│   │   └── popup.ts
│   ├── background/
│   │   └── service-worker.ts
│   └── lib/
│       ├── messaging.ts             # Extension messaging
│       ├── storage.ts               # chrome.storage wrapper
│       └── api-client.ts            # POST to ATlast API
├── assets/
│   ├── icon-16.png
│   ├── icon-48.png
│   └── icon-128.png
└── dist/
    ├── chrome/                      # Built extension for Chrome
    ├── firefox/                     # Built extension for Firefox
    └── chrome.zip                   # Store submission package

Shared Package Structure (packages/shared/)#

packages/shared/
├── package.json
├── tsconfig.json
├── src/
│   ├── types/
│   │   ├── platform.ts              # Platform enum, URL patterns
│   │   ├── import.ts                # ExtensionImportRequest, ExtensionImportResponse
│   │   ├── scraper.ts               # ScraperResult, ScraperProgress
│   │   └── index.ts                 # Re-exports
│   ├── utils/
│   │   ├── username.ts              # normalizeUsername(), validateUsername()
│   │   └── index.ts
│   └── index.ts                     # Main entry
└── dist/                            # Compiled output

Shared Types Example#

// packages/shared/src/types/platform.ts
export enum Platform {
  Twitter = 'twitter',
  Threads = 'threads',
  Instagram = 'instagram',
  TikTok = 'tiktok',
}

export interface PlatformConfig {
  platform: Platform;
  displayName: string;
  hostPatterns: string[];
  followingPathPattern: RegExp;
  iconUrl: string;
}

export const PLATFORM_CONFIGS: Record<Platform, PlatformConfig> = {
  [Platform.Twitter]: {
    platform: Platform.Twitter,
    displayName: 'Twitter/X',
    hostPatterns: ['twitter.com', 'x.com'],
    followingPathPattern: /^\/[^/]+\/following$/,
    iconUrl: 'https://abs.twimg.com/favicons/twitter.ico',
  },
  // ... future platforms
};

// packages/shared/src/types/import.ts
import { Platform } from './platform';

export interface ExtensionImportRequest {
  platform: Platform;
  usernames: string[];
  metadata: {
    extensionVersion: string;
    scrapedAt: string;
    pageType: 'following' | 'followers' | 'list';
    sourceUrl: string;
  };
}

export interface ExtensionImportResponse {
  importId: string;
  usernameCount: number;
  redirectUrl: string;
}

Platform Detection & Extensibility#

Content script detects platform from URL and loads appropriate scraper:

// src/content/index.js
const PLATFORM_PATTERNS = {
  twitter: {
    hostPatterns: ['twitter.com', 'x.com'],
    followingPath: /^\/[^/]+\/following$/,
    scraper: () => import('./scrapers/twitter-scraper.js')
  },
  threads: {
    hostPatterns: ['threads.net'],
    followingPath: /^\/[^/]+\/following$/,
    scraper: () => import('./scrapers/threads-scraper.js')
  },
  // ... future platforms
};

function detectPlatform() {
  const host = window.location.hostname;
  const path = window.location.pathname;

  for (const [name, config] of Object.entries(PLATFORM_PATTERNS)) {
    if (config.hostPatterns.some(h => host.includes(h))) {
      if (config.followingPath.test(path)) {
        return { platform: name, pageType: 'following', ...config };
      }
    }
  }
  return null;
}

Base Scraper Interface#

// src/content/scrapers/base-scraper.js
export class BaseScraper {
  constructor(options = {}) {
    this.onProgress = options.onProgress || (() => {});
    this.onComplete = options.onComplete || (() => {});
    this.onError = options.onError || (() => {});
  }

  // Must be implemented by subclasses
  getUsernameSelector() { throw new Error('Not implemented'); }
  extractUsername(element) { throw new Error('Not implemented'); }

  // Shared infinite scroll logic
  async scrape() {
    const usernames = new Set();
    let stableCount = 0;

    while (stableCount < 3) {
      const before = usernames.size;

      document.querySelectorAll(this.getUsernameSelector()).forEach(el => {
        const username = this.extractUsername(el);
        if (username) usernames.add(username);
      });

      this.onProgress({ count: usernames.size });

      window.scrollBy(0, 1000);
      await this.sleep(500);

      stableCount = (usernames.size === before) ? stableCount + 1 : 0;
    }

    this.onComplete({ usernames: Array.from(usernames) });
    return Array.from(usernames);
  }

  sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

Twitter Scraper Implementation#

// src/content/scrapers/twitter-scraper.js
import { BaseScraper } from './base-scraper.js';

export class TwitterScraper extends BaseScraper {
  getUsernameSelector() {
    // Primary selector (stable)
    return '[data-testid="UserName"]';
  }

  extractUsername(element) {
    // UserName element contains display name and @handle
    // Structure: <div><span>Display Name</span></div><div><span>@handle</span></div>
    const spans = element.querySelectorAll('span');
    for (const span of spans) {
      const text = span.textContent?.trim();
      if (text?.startsWith('@')) {
        return text.slice(1).toLowerCase(); // Remove @ prefix
      }
    }
    return null;
  }
}

iOS App Wrapper (Future)#

For iOS Safari extension, need minimal Swift app:

ATlastApp/
├── ATlast/
│   ├── ATlastApp.swift           # Minimal app entry
│   ├── ContentView.swift         # Simple "Open Safari" UI
│   └── Info.plist
├── ATlast Extension/
│   ├── SafariWebExtensionHandler.swift
│   ├── Info.plist
│   └── Resources/
│       └── (same extension files as above)
└── ATlast.xcodeproj

Decisions Made#

Remaining Questions#

Q1: Extension Branding#

• Name options: "ATlast", "ATlast Importer", "ATlast Social Bridge"
• Icon design needed

Q2: Error Recovery Strategy#

Twitter/X changes DOM frequently. Strategy for handling breaks:

• Ship updates quickly when breaks detected
• Build selector fallback chain
• User-reportable "not working" flow
• Recommendation: All of the above

Q3: Extension Store Distribution#

• Chrome Web Store (requires $5 developer fee)
• Firefox Add-ons (free)
• Safari Extensions (requires Apple Developer account, $99/year - defer with iOS)

Implementation Phases#

Phase 0: Monorepo Migration ✅ COMPLETE#

• 0.1 Install pnpm globally if needed
• 0.2 Create pnpm-workspace.yaml
• 0.3 Create packages/ directory structure
• 0.4 Move src/ → packages/web/src/
• 0.5 Move netlify/functions/ → packages/functions/
• 0.6 Create packages/shared/ with types
• 0.7 Update import paths in web and functions
• 0.8 Update netlify.toml for new paths
• 0.9 Update root package.json scripts
• 0.10 Test build and dev commands
• 0.11 Commit monorepo migration

Phase 1: Chrome Extension MVP ✅ COMPLETE#

• 1.1 Create packages/extension/ structure
• 1.2 Write manifest.json (Manifest V3)
• 1.3 Implement base-scraper.ts abstract class
• 1.4 Implement twitter-scraper.ts
• 1.5 Implement content/index.ts (platform detection)
• 1.6 Implement popup UI (HTML/CSS/TS)
• 1.7 Implement background service worker
• 1.8 Implement api-client.ts (POST to ATlast)
• 1.9 Create Netlify function: extension-import.ts
• 1.10 Create ATlast import page: /import/[id] (Not needed - uses /results?uploadId)
• 1.11 Add extension build script
• 1.12 Test end-to-end flow locally - All bugs resolved
• 1.13 Chrome Web Store submission - Next step

Phase 2: Firefox Support#

• 2.1 Create manifest.firefox.json (MV2 if needed)
• 2.2 Test on Firefox desktop
• 2.3 Test on Firefox Android
• 2.4 Firefox Add-ons submission

Phase 3: Enhanced Twitter Features#

• 3.1 Support Followers page
• 3.2 Support Twitter Lists
• 3.3 Add selector fallback chain
• 3.4 Add user-reportable error flow

Phase 4: Additional Platforms (Future)#

• 4.1 Threads scraper
• 4.2 Instagram scraper
• 4.3 TikTok scraper

Phase 5: iOS Support (Future)#

• 5.1 iOS app wrapper (Swift)
• 5.2 Safari Web Extension integration
• 5.3 App Store submission

Related Decision Graph Nodes#

• Goal: #184 (Support Twitter/X file uploads)
• Problem Analysis: #185-186 (user_id issue, resolution approach decision)
• Initial Options: #187-192 (server-side, extension, CLI, BYOK, hybrid)
• Research: #193-204 (Nitter dead, Sky Follower Bridge, DOM scraping)
• iOS Research: #212-216 (Shortcuts timeout, Safari Web Extensions)
• Architecture Decisions: #218-222
  • #219: POST to API endpoint
  • #220: Twitter Following page MVP
  • #221: iOS deferred
  • #222: Multi-platform architecture
• Implementation Decisions: #224-227
  • #225: Monorepo with shared packages
  • #226: Extension name "ATlast Importer"
  • #227: pnpm workspaces tooling

View live graph: https://notactuallytreyanastasio.github.io/deciduous/

Changelog#