aesthetic.computer
Monorepo for Aesthetic.Computer
aesthetic.computer
1# WebSocket Module Loader
2
3## Problem
4
5On localhost, the HTTP proxy chain (Caddy → Netlify Dev → browser) causes:
6- `ERR_CONTENT_LENGTH_MISMATCH` errors
7- `ERR_INCOMPLETE_CHUNKED_ENCODING` errors
8- Slow module loading due to proxy overhead
9- Unreliable hot reloading
10
11Current workarounds in `Caddyfile`:
12- Disabled compression (`encode zstd gzip` commented out)
13- Strip `Accept-Encoding` header
14- Force `Connection: close`
15
16These help but don't fully solve the issue.
17
18## Solution: WebSocket Module Streaming
19
20Establish an early WebSocket connection to session-server and use it to:
211. Stream JS module text directly to the client
222. Cache modules locally (IndexedDB/Cache API)
233. Serve cached modules on subsequent loads
244. Prefetch modules in parallel with execution
25
26## Architecture
27
28```
29┌─────────────┐ ┌──────────────────┐
30│ boot.mjs │───WebSocket───────▶│ session-server │
31│ (early) │ │ │
32└─────────────┘ │ /ws/modules │
33 │ │ - disk.mjs │
34 ▼ │ - bios.mjs │
35┌─────────────┐ │ - graph.mjs │
36│ IndexedDB │ │ - ... │
37│ Module Cache│ └──────────────────┘
38└─────────────┘
39 │
40 ▼
41┌─────────────┐
42│ Blob URL │ URL.createObjectURL(new Blob([moduleText]))
43│ Import │ import(blobUrl)
44└─────────────┘
45```
46
47## Implementation Plan
48
49### Phase 1: Session Server Module Endpoint
50
51**File**: `session-server/session.mjs`
52
53Add a module streaming protocol:
54
55```javascript
56// New message types
57ws.on('message', (data) => {
58 const msg = JSON.parse(data);
59
60 if (msg.type === 'module:request') {
61 // Request: { type: 'module:request', path: 'lib/disk.mjs' }
62 const modulePath = path.join(PUBLIC_DIR, 'aesthetic.computer', msg.path);
63 const content = fs.readFileSync(modulePath, 'utf8');
64 const hash = crypto.createHash('sha256').update(content).digest('hex').slice(0, 16);
65
66 ws.send(JSON.stringify({
67 type: 'module:response',
68 path: msg.path,
69 hash,
70 content
71 }));
72 }
73
74 if (msg.type === 'module:check') {
75 // Check if module changed: { type: 'module:check', path: 'lib/disk.mjs', hash: '...' }
76 const modulePath = path.join(PUBLIC_DIR, 'aesthetic.computer', msg.path);
77 const content = fs.readFileSync(modulePath, 'utf8');
78 const currentHash = crypto.createHash('sha256').update(content).digest('hex').slice(0, 16);
79
80 ws.send(JSON.stringify({
81 type: 'module:status',
82 path: msg.path,
83 changed: currentHash !== msg.hash,
84 hash: currentHash
85 }));
86 }
87});
88```
89
90### Phase 2: Boot.mjs Early Connection
91
92**File**: `system/public/aesthetic.computer/boot.mjs`
93
94```javascript
95// Very early - before any other imports
96const MODULE_CACHE_NAME = 'ac-modules-v1';
97const SESSION_WS_URL = location.hostname === 'localhost'
98 ? 'ws://localhost:8889'
99 : 'wss://session-server.aesthetic.computer';
100
101class ModuleLoader {
102 constructor() {
103 this.ws = null;
104 this.cache = null;
105 this.pending = new Map(); // path -> Promise resolvers
106 this.modules = new Map(); // path -> { hash, blobUrl }
107 }
108
109 async init() {
110 // Open IndexedDB cache
111 this.cache = await this.openCache();
112
113 // Connect WebSocket
114 this.ws = new WebSocket(SESSION_WS_URL);
115
116 return new Promise((resolve, reject) => {
117 this.ws.onopen = () => {
118 this.ws.onmessage = (e) => this.handleMessage(JSON.parse(e.data));
119 resolve();
120 };
121 this.ws.onerror = reject;
122 setTimeout(() => reject(new Error('WS timeout')), 5000);
123 });
124 }
125
126 async openCache() {
127 return new Promise((resolve, reject) => {
128 const req = indexedDB.open('ac-module-cache', 1);
129 req.onupgradeneeded = (e) => {
130 const db = e.target.result;
131 db.createObjectStore('modules', { keyPath: 'path' });
132 };
133 req.onsuccess = () => resolve(req.result);
134 req.onerror = reject;
135 });
136 }
137
138 handleMessage(msg) {
139 if (msg.type === 'module:response') {
140 const resolver = this.pending.get(msg.path);
141 if (resolver) {
142 // Create blob URL
143 const blob = new Blob([msg.content], { type: 'application/javascript' });
144 const blobUrl = URL.createObjectURL(blob);
145
146 // Cache it
147 this.modules.set(msg.path, { hash: msg.hash, blobUrl });
148 this.cacheModule(msg.path, msg.hash, msg.content);
149
150 resolver.resolve(blobUrl);
151 this.pending.delete(msg.path);
152 }
153 }
154
155 if (msg.type === 'module:status') {
156 const resolver = this.pending.get(`check:${msg.path}`);
157 if (resolver) {
158 resolver.resolve(msg);
159 this.pending.delete(`check:${msg.path}`);
160 }
161 }
162 }
163
164 async load(path) {
165 // Check local cache first
166 const cached = await this.getCached(path);
167 if (cached) {
168 // Verify hash in background
169 this.checkHash(path, cached.hash);
170 return cached.blobUrl;
171 }
172
173 // Request from server
174 return new Promise((resolve, reject) => {
175 this.pending.set(path, { resolve, reject });
176 this.ws.send(JSON.stringify({ type: 'module:request', path }));
177 setTimeout(() => reject(new Error(`Module timeout: ${path}`)), 10000);
178 });
179 }
180
181 async getCached(path) {
182 return new Promise((resolve) => {
183 const tx = this.cache.transaction('modules', 'readonly');
184 const req = tx.objectStore('modules').get(path);
185 req.onsuccess = () => {
186 if (req.result) {
187 const blob = new Blob([req.result.content], { type: 'application/javascript' });
188 resolve({ hash: req.result.hash, blobUrl: URL.createObjectURL(blob) });
189 } else {
190 resolve(null);
191 }
192 };
193 req.onerror = () => resolve(null);
194 });
195 }
196
197 cacheModule(path, hash, content) {
198 const tx = this.cache.transaction('modules', 'readwrite');
199 tx.objectStore('modules').put({ path, hash, content });
200 }
201
202 async checkHash(path, cachedHash) {
203 // Check if server version changed
204 return new Promise((resolve) => {
205 this.pending.set(`check:${path}`, { resolve });
206 this.ws.send(JSON.stringify({ type: 'module:check', path, hash: cachedHash }));
207 });
208 }
209
210 // Prefetch modules we know we'll need
211 prefetch(paths) {
212 for (const path of paths) {
213 if (!this.modules.has(path) && !this.pending.has(path)) {
214 this.load(path).catch(() => {}); // Fire and forget
215 }
216 }
217 }
218}
219
220// Global instance
221window.acModuleLoader = new ModuleLoader();
222
223// Export for use in other modules
224export { ModuleLoader };
225export const moduleLoader = window.acModuleLoader;
226```
227
228### Phase 3: Integration with Boot Sequence
229
230**File**: `system/public/aesthetic.computer/boot.mjs` (updated)
231
232```javascript
233// At the very top of boot.mjs
234import { moduleLoader } from './module-loader.mjs';
235
236async function boot() {
237 // 1. Initialize module loader (WebSocket + IndexedDB)
238 try {
239 await moduleLoader.init();
240 console.log('🔌 Module loader connected');
241
242 // 2. Prefetch critical modules immediately
243 moduleLoader.prefetch([
244 'lib/disk.mjs',
245 'lib/graph.mjs',
246 'lib/num.mjs',
247 'lib/geo.mjs',
248 'lib/parse.mjs',
249 'lib/help.mjs',
250 'bios.mjs'
251 ]);
252 } catch (e) {
253 console.warn('⚠️ Module loader failed, falling back to HTTP:', e);
254 // Fall back to normal HTTP imports
255 }
256
257 // 3. Load disk.mjs (will use cache if available)
258 const diskUrl = await moduleLoader.load('lib/disk.mjs').catch(() => './lib/disk.mjs');
259 const { boot: diskBoot } = await import(diskUrl);
260
261 // ... rest of boot sequence
262}
263```
264
265### Phase 4: Hot Reload via WebSocket
266
267Session server can push module updates:
268
269```javascript
270// session-server: Watch for file changes
271const watcher = fs.watch(PUBLIC_DIR, { recursive: true }, (eventType, filename) => {
272 if (filename.endsWith('.mjs') || filename.endsWith('.js')) {
273 const relativePath = filename.replace(/\\/g, '/');
274 const content = fs.readFileSync(path.join(PUBLIC_DIR, filename), 'utf8');
275 const hash = crypto.createHash('sha256').update(content).digest('hex').slice(0, 16);
276
277 // Broadcast to all connected clients
278 broadcast({
279 type: 'module:updated',
280 path: relativePath,
281 hash
282 });
283 }
284});
285
286// Client side: Listen for updates
287moduleLoader.ws.onmessage = (e) => {
288 const msg = JSON.parse(e.data);
289 if (msg.type === 'module:updated') {
290 // Invalidate cache
291 moduleLoader.modules.delete(msg.path);
292 // Optionally trigger hot reload
293 if (window.acHotReload) {
294 window.acHotReload(msg.path);
295 }
296 }
297};
298```
299
300## Benefits
301
302### Development
3031. **Bypasses proxy chain** - Direct WebSocket to session-server, no Caddy/Netlify issues
3042. **No more `ERR_CONTENT_LENGTH_MISMATCH`** - WebSocket is a clean binary channel
3053. **Hot reload** - Push module updates instantly via existing connection
3064. **Faster iteration** - Module changes arrive in milliseconds
307
308### Production
3091. **Persistent connection** - No TCP/TLS handshake per module (already connected for real-time)
3102. **Parallel prefetching** - Download next modules while current ones execute
3113. **Local caching** - Instant loads after first visit (IndexedDB survives refresh)
3124. **Hash validation** - Know when cache is stale, only re-download changed modules
3135. **Single connection** - Reuse for module loading + real-time features + UDP setup
314
315### Both
3161. **Graceful fallback** - If WebSocket is slow/offline, HTTP works exactly as before
3172. **Progressive enhancement** - Zero breakage, only speed improvements
3183. **Transparent** - Code doesn't need to know where modules came from
319
320## Fallback Strategy
321
322The loader races WebSocket against a timeout:
323
324```javascript
325async load(path) {
326 // Race: WebSocket vs timeout
327 const wsPromise = this.loadViaWebSocket(path);
328 const timeoutMs = 500; // Half second max wait
329
330 try {
331 return await Promise.race([
332 wsPromise,
333 new Promise((_, reject) =>
334 setTimeout(() => reject(new Error('WS slow')), timeoutMs)
335 )
336 ]);
337 } catch {
338 // WebSocket slow or failed - use normal HTTP
339 console.log(`⚡ Falling back to HTTP for ${path}`);
340 return `./${path}`; // Normal relative import path
341 }
342}
343```
344
345This means:
346- **WebSocket fast (< 500ms)**: Use cached/streamed module ✅
347- **WebSocket slow (> 500ms)**: Fall back to HTTP, no delay ✅
348- **WebSocket offline**: Immediate fallback to HTTP ✅
349- **Cached locally**: Instant, no network at all ✅
350
351## Migration Path
352
3531. **Phase 1**: Add module endpoint to session-server (no client changes)
3542. **Phase 2**: Add ModuleLoader class to boot.mjs with HTTP fallback
3553. **Phase 3**: Gradually migrate critical modules to use loader
3564. **Phase 4**: Add hot reload support
3575. **Phase 5**: Consider for production (with CDN cache headers)
358
359## Considerations
360
361- **CORS**: WebSocket doesn't have CORS issues
362- **Binary transfer**: Could use binary WebSocket frames for larger modules
363- **Compression**: WebSocket can use per-message deflate
364- **Fallback**: Always fall back to HTTP if WebSocket fails
365- **Production**: Could still be useful for faster initial load + hot reload
366
367## File Changes
368
369| File | Change |
370|------|--------|
371| `session-server/session.mjs` | Add module streaming handlers |
372| `system/public/aesthetic.computer/module-loader.mjs` | New file - ModuleLoader class |
373| `system/public/aesthetic.computer/boot.mjs` | Integrate ModuleLoader early |
374| `system/public/aesthetic.computer/lib/disk.mjs` | Use moduleLoader for dynamic imports |
375
376## Status
377
378- [ ] Phase 1: Session server module endpoint
379- [ ] Phase 2: ModuleLoader class
380- [ ] Phase 3: Boot.mjs integration
381- [ ] Phase 4: Hot reload
382- [ ] Phase 5: Production evaluation