WebSocket Module Loader#

Problem#

On localhost, the HTTP proxy chain (Caddy → Netlify Dev → browser) causes:

• ERR_CONTENT_LENGTH_MISMATCH errors
• ERR_INCOMPLETE_CHUNKED_ENCODING errors
• Slow module loading due to proxy overhead
• Unreliable hot reloading

Current workarounds in Caddyfile:

• Disabled compression (encode zstd gzip commented out)
• Strip Accept-Encoding header
• Force Connection: close

These help but don't fully solve the issue.

Solution: WebSocket Module Streaming#

Establish an early WebSocket connection to session-server and use it to:

1. Stream JS module text directly to the client
2. Cache modules locally (IndexedDB/Cache API)
3. Serve cached modules on subsequent loads
4. Prefetch modules in parallel with execution

Architecture#

┌─────────────┐                    ┌──────────────────┐
│  boot.mjs   │───WebSocket───────▶│  session-server  │
│  (early)    │                    │                  │
└─────────────┘                    │  /ws/modules     │
       │                           │  - disk.mjs      │
       ▼                           │  - bios.mjs      │
┌─────────────┐                    │  - graph.mjs     │
│ IndexedDB   │                    │  - ...           │
│ Module Cache│                    └──────────────────┘
└─────────────┘
       │
       ▼
┌─────────────┐
│ Blob URL    │  URL.createObjectURL(new Blob([moduleText]))
│ Import      │  import(blobUrl)
└─────────────┘

Implementation Plan#

Phase 1: Session Server Module Endpoint#

File: session-server/session.mjs

Add a module streaming protocol:

// New message types
ws.on('message', (data) => {
  const msg = JSON.parse(data);
  
  if (msg.type === 'module:request') {
    // Request: { type: 'module:request', path: 'lib/disk.mjs' }
    const modulePath = path.join(PUBLIC_DIR, 'aesthetic.computer', msg.path);
    const content = fs.readFileSync(modulePath, 'utf8');
    const hash = crypto.createHash('sha256').update(content).digest('hex').slice(0, 16);
    
    ws.send(JSON.stringify({
      type: 'module:response',
      path: msg.path,
      hash,
      content
    }));
  }
  
  if (msg.type === 'module:check') {
    // Check if module changed: { type: 'module:check', path: 'lib/disk.mjs', hash: '...' }
    const modulePath = path.join(PUBLIC_DIR, 'aesthetic.computer', msg.path);
    const content = fs.readFileSync(modulePath, 'utf8');
    const currentHash = crypto.createHash('sha256').update(content).digest('hex').slice(0, 16);
    
    ws.send(JSON.stringify({
      type: 'module:status',
      path: msg.path,
      changed: currentHash !== msg.hash,
      hash: currentHash
    }));
  }
});

Phase 2: Boot.mjs Early Connection#

File: system/public/aesthetic.computer/boot.mjs

// Very early - before any other imports
const MODULE_CACHE_NAME = 'ac-modules-v1';
const SESSION_WS_URL = location.hostname === 'localhost' 
  ? 'ws://localhost:8889'
  : 'wss://session-server.aesthetic.computer';

class ModuleLoader {
  constructor() {
    this.ws = null;
    this.cache = null;
    this.pending = new Map(); // path -> Promise resolvers
    this.modules = new Map(); // path -> { hash, blobUrl }
  }
  
  async init() {
    // Open IndexedDB cache
    this.cache = await this.openCache();
    
    // Connect WebSocket
    this.ws = new WebSocket(SESSION_WS_URL);
    
    return new Promise((resolve, reject) => {
      this.ws.onopen = () => {
        this.ws.onmessage = (e) => this.handleMessage(JSON.parse(e.data));
        resolve();
      };
      this.ws.onerror = reject;
      setTimeout(() => reject(new Error('WS timeout')), 5000);
    });
  }
  
  async openCache() {
    return new Promise((resolve, reject) => {
      const req = indexedDB.open('ac-module-cache', 1);
      req.onupgradeneeded = (e) => {
        const db = e.target.result;
        db.createObjectStore('modules', { keyPath: 'path' });
      };
      req.onsuccess = () => resolve(req.result);
      req.onerror = reject;
    });
  }
  
  handleMessage(msg) {
    if (msg.type === 'module:response') {
      const resolver = this.pending.get(msg.path);
      if (resolver) {
        // Create blob URL
        const blob = new Blob([msg.content], { type: 'application/javascript' });
        const blobUrl = URL.createObjectURL(blob);
        
        // Cache it
        this.modules.set(msg.path, { hash: msg.hash, blobUrl });
        this.cacheModule(msg.path, msg.hash, msg.content);
        
        resolver.resolve(blobUrl);
        this.pending.delete(msg.path);
      }
    }
    
    if (msg.type === 'module:status') {
      const resolver = this.pending.get(`check:${msg.path}`);
      if (resolver) {
        resolver.resolve(msg);
        this.pending.delete(`check:${msg.path}`);
      }
    }
  }
  
  async load(path) {
    // Check local cache first
    const cached = await this.getCached(path);
    if (cached) {
      // Verify hash in background
      this.checkHash(path, cached.hash);
      return cached.blobUrl;
    }
    
    // Request from server
    return new Promise((resolve, reject) => {
      this.pending.set(path, { resolve, reject });
      this.ws.send(JSON.stringify({ type: 'module:request', path }));
      setTimeout(() => reject(new Error(`Module timeout: ${path}`)), 10000);
    });
  }
  
  async getCached(path) {
    return new Promise((resolve) => {
      const tx = this.cache.transaction('modules', 'readonly');
      const req = tx.objectStore('modules').get(path);
      req.onsuccess = () => {
        if (req.result) {
          const blob = new Blob([req.result.content], { type: 'application/javascript' });
          resolve({ hash: req.result.hash, blobUrl: URL.createObjectURL(blob) });
        } else {
          resolve(null);
        }
      };
      req.onerror = () => resolve(null);
    });
  }
  
  cacheModule(path, hash, content) {
    const tx = this.cache.transaction('modules', 'readwrite');
    tx.objectStore('modules').put({ path, hash, content });
  }
  
  async checkHash(path, cachedHash) {
    // Check if server version changed
    return new Promise((resolve) => {
      this.pending.set(`check:${path}`, { resolve });
      this.ws.send(JSON.stringify({ type: 'module:check', path, hash: cachedHash }));
    });
  }
  
  // Prefetch modules we know we'll need
  prefetch(paths) {
    for (const path of paths) {
      if (!this.modules.has(path) && !this.pending.has(path)) {
        this.load(path).catch(() => {}); // Fire and forget
      }
    }
  }
}

// Global instance
window.acModuleLoader = new ModuleLoader();

// Export for use in other modules
export { ModuleLoader };
export const moduleLoader = window.acModuleLoader;

Phase 3: Integration with Boot Sequence#

File: system/public/aesthetic.computer/boot.mjs (updated)

// At the very top of boot.mjs
import { moduleLoader } from './module-loader.mjs';

async function boot() {
  // 1. Initialize module loader (WebSocket + IndexedDB)
  try {
    await moduleLoader.init();
    console.log('🔌 Module loader connected');
    
    // 2. Prefetch critical modules immediately
    moduleLoader.prefetch([
      'lib/disk.mjs',
      'lib/graph.mjs',
      'lib/num.mjs',
      'lib/geo.mjs',
      'lib/parse.mjs',
      'lib/help.mjs',
      'bios.mjs'
    ]);
  } catch (e) {
    console.warn('⚠️ Module loader failed, falling back to HTTP:', e);
    // Fall back to normal HTTP imports
  }
  
  // 3. Load disk.mjs (will use cache if available)
  const diskUrl = await moduleLoader.load('lib/disk.mjs').catch(() => './lib/disk.mjs');
  const { boot: diskBoot } = await import(diskUrl);
  
  // ... rest of boot sequence
}

Phase 4: Hot Reload via WebSocket#

Session server can push module updates:

// session-server: Watch for file changes
const watcher = fs.watch(PUBLIC_DIR, { recursive: true }, (eventType, filename) => {
  if (filename.endsWith('.mjs') || filename.endsWith('.js')) {
    const relativePath = filename.replace(/\\/g, '/');
    const content = fs.readFileSync(path.join(PUBLIC_DIR, filename), 'utf8');
    const hash = crypto.createHash('sha256').update(content).digest('hex').slice(0, 16);
    
    // Broadcast to all connected clients
    broadcast({
      type: 'module:updated',
      path: relativePath,
      hash
    });
  }
});

// Client side: Listen for updates
moduleLoader.ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === 'module:updated') {
    // Invalidate cache
    moduleLoader.modules.delete(msg.path);
    // Optionally trigger hot reload
    if (window.acHotReload) {
      window.acHotReload(msg.path);
    }
  }
};

Benefits#

Development#

1. Bypasses proxy chain - Direct WebSocket to session-server, no Caddy/Netlify issues
2. No more ERR_CONTENT_LENGTH_MISMATCH - WebSocket is a clean binary channel
3. Hot reload - Push module updates instantly via existing connection
4. Faster iteration - Module changes arrive in milliseconds

Production#

1. Persistent connection - No TCP/TLS handshake per module (already connected for real-time)
2. Parallel prefetching - Download next modules while current ones execute
3. Local caching - Instant loads after first visit (IndexedDB survives refresh)
4. Hash validation - Know when cache is stale, only re-download changed modules
5. Single connection - Reuse for module loading + real-time features + UDP setup

Both#

1. Graceful fallback - If WebSocket is slow/offline, HTTP works exactly as before
2. Progressive enhancement - Zero breakage, only speed improvements
3. Transparent - Code doesn't need to know where modules came from

Fallback Strategy#

The loader races WebSocket against a timeout:

async load(path) {
  // Race: WebSocket vs timeout
  const wsPromise = this.loadViaWebSocket(path);
  const timeoutMs = 500; // Half second max wait
  
  try {
    return await Promise.race([
      wsPromise,
      new Promise((_, reject) => 
        setTimeout(() => reject(new Error('WS slow')), timeoutMs)
      )
    ]);
  } catch {
    // WebSocket slow or failed - use normal HTTP
    console.log(`⚡ Falling back to HTTP for ${path}`);
    return `./${path}`; // Normal relative import path
  }
}

This means:

• WebSocket fast (< 500ms): Use cached/streamed module ✅
• WebSocket slow (> 500ms): Fall back to HTTP, no delay ✅
• WebSocket offline: Immediate fallback to HTTP ✅
• Cached locally: Instant, no network at all ✅

Migration Path#

1. Phase 1: Add module endpoint to session-server (no client changes)
2. Phase 2: Add ModuleLoader class to boot.mjs with HTTP fallback
3. Phase 3: Gradually migrate critical modules to use loader
4. Phase 4: Add hot reload support
5. Phase 5: Consider for production (with CDN cache headers)

Considerations#

• CORS: WebSocket doesn't have CORS issues
• Binary transfer: Could use binary WebSocket frames for larger modules
• Compression: WebSocket can use per-message deflate
• Fallback: Always fall back to HTTP if WebSocket fails
• Production: Could still be useful for faster initial load + hot reload

File Changes#

Status#

• Phase 1: Session server module endpoint
• Phase 2: ModuleLoader class
• Phase 3: Boot.mjs integration
• Phase 4: Hot reload
• Phase 5: Production evaluation