aesthetic.computer
Monorepo for Aesthetic.Computer
aesthetic.computer
1# Grab Worker - Browser Rendering Service Plan
2
3## 📋 Overview
4
5Replace the current Netlify screenshot.js function with a Cloudflare Worker that uses Cloudflare's Browser Rendering API with Durable Objects for screenshot generation and caching.
6
7**Deployment Target:** `grab.aesthetic.computer`
8
9**Status:** ✅ **DEPLOYED & WORKING!**
10- Worker URL: https://aesthetic-grab.aesthetic-computer.workers.dev
11- Version: 511d31bc-c2e8-4f00-aea7-9b9d95e08733
12- Screenshots generating successfully with query parameter support
13
14**Tutorial Reference:** https://developers.cloudflare.com/browser-rendering/get-started/
15
16---
17
18## ✅ Completed Tasks
19
20### Phase 1: Setup & Implementation ✅
21- [x] Created `/grab` directory structure
22- [x] Set up vault directory for secrets at `/aesthetic-computer-vault/grab/`
23- [x] Created comprehensive documentation (PLAN.md, README.md, DEPLOYMENT.md, MIGRATION.md)
24- [x] Implemented TypeScript worker following Cloudflare's official example
25- [x] Configured wrangler.toml with Browser binding and Durable Objects
26- [x] Installed dependencies (wrangler 4.42.2, @cloudflare/puppeteer latest)
27- [x] Created deployment scripts (deploy.fish)
28
29### Phase 2: Deployment & Debugging ✅
30- [x] Fixed TypeScript compilation errors
31- [x] Added Buffer polyfill for Puppeteer compatibility
32- [x] Updated @cloudflare/puppeteer from 0.0.1 to latest (critical fix!)
33- [x] Replaced `page.waitForTimeout()` with standard `setTimeout()`
34- [x] Successfully deployed to Cloudflare Workers
35- [x] Fixed URL construction to use query parameters (`?icon=WxH`, `?preview=WxH`)
36- [x] Verified screenshots generating correctly (128x128, 256x256, 1200x630 tested)
37- [x] **Added video recording endpoint** using CDP screencast API
38 - `/video/WxH/piece.mp4?duration=N` endpoint
39 - Captures frames over 1-30 seconds
40 - Returns last frame + metadata (frame count, duration, resolution)
41 - POC working: 162 frames in 5s (~32 fps)
42 - Full video encoding requires future infrastructure
43
44### Key Fixes Applied
451. **Browser Binding Syntax**: Changed from `[[browser]]` to `[browser]` in wrangler.toml
462. **Durable Object Migration**: Added v1→v2 migration for ScreenshotDO→Browser rename
473. **Puppeteer Update**: Version 0.0.1 was returning empty screenshots; latest version works
484. **Query Parameters**: Now properly uses `?icon=128x128` or `?preview=1200x630` like original
495. **Timeout Replacement**: Modern Puppeteer doesn't have waitForTimeout, using Promise setTimeout
50
51---
52
53## 🚧 Remaining Tasks
54
55### Phase 3: Custom Domain & DNS 🔄
56- [ ] **Configure Custom Domain** (Choose one method)
57
58 **Method 1: DNS Settings (Recommended - Faster!)**
59 - Navigate to: https://dash.cloudflare.com/a23b54e8877a833a1cf8db7765bce3ca/aesthetic.computer/dns/records
60 - Add CNAME: `grab` → `aesthetic-grab.aesthetic-computer.workers.dev` (Proxied)
61 - Wait: ~30 seconds
62 - See: `DNS-SETUP.md` for details
63
64 **Method 2: Workers Dashboard (Alternative)**
65 - Navigate to: Workers & Pages > aesthetic-grab > Settings > Domains & Routes
66 - Click: "Add Custom Domain"
67 - Enter: `grab.aesthetic.computer`
68 - Wait: 2-5 minutes
69
70 **Verification:**
71 - Test: `curl -I "https://grab.aesthetic.computer/icon/128x128/prompt.png"`
72 - Should return: HTTP/2 200
73
74### Phase 4: Integration & Migration ✅
75- [x] **Update parse.mjs** to use grab worker URLs
76 - File: `/workspaces/aesthetic-computer/system/public/aesthetic.computer/lib/parse.mjs`
77 - Lines 487-494: Updated to use `https://grab.aesthetic.computer/icon/...` and `https://grab.aesthetic.computer/preview/...`
78
79- [x] **Update netlify.toml** redirects
80 - File: `/workspaces/aesthetic-computer/system/netlify.toml`
81 - Lines 285-290 & 306-310: Redirects now point to grab.aesthetic.computer
82 - Backwards compatibility maintained for existing paths
83
84- [x] **Create DEV-SETUP.md** with development workflow
85 - Local testing with `wrangler dev`
86 - Integration with netlify dev
87 - Emacs web panels setup instructions
88 - Debugging and monitoring guides
89
90- [ ] **Production Validation** 🔄
91 - Test og:image meta tags on live site
92 - Test favicon generation
93 - Monitor for errors in Cloudflare dashboard
94 - Validate across different pieces
95
96- [ ] **Cleanup** (after 1 week validation period)
97 - Remove `/system/netlify/functions/screenshot.js`
98 - Remove Puppeteer dependencies from Netlify
99 - Update documentation
100
101### Phase 5: Development Integration ✅
102- [x] **Create Development Documentation**
103 - DEV-SETUP.md: Local testing with wrangler dev
104 - Netlify dev integration patterns
105 - Emacs web panel configuration
106 - Debugging and monitoring guides
107
108- [x] **Create Integration Documentation**
109 - INTEGRATION-SUMMARY.md: Complete overview of changes
110 - CODE-CHANGES.md: Detailed diff of modified files
111 - CUSTOM-DOMAIN-SETUP.md: Quick reference for domain setup
112
113- [ ] **Add to Dev Stack** (Optional)
114 - Create Emacs web panel configuration for grab worker
115 - Add wrangler dev command to startup scripts
116 - Test localhost screenshot generation
117
118---
119
120## 📚 Documentation Index
121
122All documentation lives in `/workspaces/aesthetic-computer/grab/`:
123
124- **PLAN.md** (this file) - Project roadmap and progress tracking
125- **README.md** - Architecture, API reference, and technical overview
126- **DEPLOYMENT.md** - Deployment guide with custom domain instructions
127- **DEV-SETUP.md** - Local development workflow and debugging
128- **MIGRATION.md** - Migration strategy from Netlify to Cloudflare
129- **INTEGRATION-SUMMARY.md** - What's complete, what's pending, next steps
130- **CODE-CHANGES.md** - Detailed file-by-file change documentation
131- **CUSTOM-DOMAIN-SETUP.md** - Quick reference for domain configuration
132
133---
134
135## 🎯 Current Status
136
137**Worker:** ✅ Deployed and functional
138**Integration:** ✅ Code updated, redirects configured
139**Custom Domain:** ⚠️ Pending manual setup via Dashboard
140**Testing:** ⚠️ Awaiting custom domain for full validation
141**Production:** 🕐 Ready to go live after custom domain setup
142
143---
144
145## 🚀 Immediate Next Steps
146
1471. **Get Cloudflare API Token** (1 minute)
148 - Go to: https://dash.cloudflare.com/profile/api-tokens
149 - Create token with "Edit zone DNS" template
150 - Add to `/aesthetic-computer-vault/grab/.env` as `CLOUDFLARE_API_TOKEN`
151
1522. **Deploy with Automatic DNS** (2 minutes)
153 ```fish
154 cd /workspaces/aesthetic-computer/grab
155 ./scripts/deploy-with-dns.fish
156 ```
157 This one command handles:
158 - Worker deployment
159 - DNS CNAME record creation
160 - DNS propagation wait
161 - Deployment verification
162
1633. **Test Production Integration** (10 minutes)
164 - Test og:images on live site
165 - Verify social media previews
166 - Check favicon loading
167 - Monitor Cloudflare dashboard
168
1694. **Begin Validation Period** (1 week)
170 - Monitor error rates
171 - Compare performance with old implementation
172 - Test with various pieces
173 - Collect user feedback
174
1755. **Cleanup** (after validation)
176 - Remove `/system/netlify/functions/screenshot.js`
177 - Remove Puppeteer dependencies
178 - Update final documentation
179
180---
181
182## 📁 Directory Structure
183
184```
185/workspaces/aesthetic-computer/grab/
186├── PLAN.md # This file
187├── README.md # Documentation and usage
188├── DEPLOYMENT.md # Deployment instructions
189├── package.json # Dependencies
190├── tsconfig.json # TypeScript config
191├── wrangler.toml # Cloudflare config (gitignored)
192├── wrangler.production.toml # Production config template
193├── src/
194│ ├── worker.ts # Main worker entry point
195│ ├── screenshot-do.ts # Durable Object for screenshot coordination
196│ ├── browser.ts # Browser rendering logic
197│ ├── cache.ts # Cache management utilities
198│ └── types.ts # TypeScript types
199├── scripts/
200│ ├── deploy.fish # Deployment script
201│ └── setup-browser-binding.fish # Initial setup script
202└── test/
203 └── worker.test.ts # Tests
204```
205
206---
207
208## 🔧 Architecture
209
210### Current Implementation (Netlify)
211- Serverless function using Puppeteer Core
212- Chrome browser via `@sparticuz/chromium`
213- Local file cache in dev mode
214- ETag-based HTTP caching
215- Accepts: `128x128` (icon), `1200x630` (og:image), `1800x900` (twitter:image)
216
217### New Implementation (Cloudflare)
218- **Worker**: Handle incoming requests, parse parameters
219- **Durable Object**: Coordinate screenshot generation per URL
220- **Browser Rendering API**: Puppeteer-compatible interface
221- **R2 Storage** (optional): Long-term screenshot cache
222- **Cache API**: Fast edge caching with ETags
223- **Browser Binding**: Native Cloudflare browser sessions
224
225### Request Flow
226```
227Client Request
228 ↓
229Worker (grab.aesthetic.computer)
230 ↓
231Parse /icon/ or /preview/ path
232 ↓
233Check Cache API (ETag)
234 ↓ (miss)
235Durable Object for URL
236 ↓
237Browser Rendering API
238 ↓
239Take Screenshot
240 ↓
241Store in Cache + R2
242 ↓
243Return PNG
244```
245
246---
247
248## 🔐 Environment Variables & Secrets
249
250### Stored in `/aesthetic-computer-vault/grab/.env`
251
252```bash
253# Cloudflare Configuration
254CLOUDFLARE_ACCOUNT_ID=<account-id>
255CLOUDFLARE_API_KEY=<api-key>
256CLOUDFLARE_EMAIL=me@jas.life
257
258# Worker Configuration
259GRAB_WORKER_NAME=aesthetic-grab
260GRAB_DOMAIN=grab.aesthetic.computer
261
262# Browser Binding Configuration
263BROWSER_BINDING_NAME=BROWSER
264BROWSER_BINDING_ID=<browser-binding-id>
265
266# Durable Object Configuration
267DO_NAMESPACE_NAME=SCREENSHOT_DO
268DO_NAMESPACE_ID=<do-namespace-id>
269
270# R2 Bucket (optional)
271R2_BUCKET_NAME=aesthetic-screenshots
272R2_BUCKET_ID=<r2-bucket-id>
273
274# Cache Configuration
275CACHE_TTL_SECONDS=3600 # 1 hour for browser cache
276CDN_CACHE_TTL_SECONDS=86400 # 24 hours for CDN cache
277DEV_CACHE_TTL_SECONDS=60 # 1 minute for dev
278
279# Screenshot Configuration
280MAX_SCREENSHOT_AGE_MS=604800000 # 7 days
281BROWSER_TIMEOUT_MS=30000 # 30 seconds
282```
283
284### Secrets (set via `wrangler secret put`)
285- `BROWSER_API_KEY` - If needed for additional auth
286
287---
288
289## 📦 Dependencies
290
291```json
292{
293 "name": "@aesthetic-computer/grab",
294 "version": "1.0.0",
295 "type": "module",
296 "scripts": {
297 "dev": "wrangler dev",
298 "deploy": "./scripts/deploy.fish",
299 "deploy:production": "./scripts/deploy.fish production",
300 "type-check": "tsc --noEmit",
301 "test": "vitest"
302 },
303 "dependencies": {
304 "@cloudflare/puppeteer": "^0.0.1",
305 "@cloudflare/workers-types": "^4.20231218.0"
306 },
307 "devDependencies": {
308 "typescript": "^5.3.3",
309 "wrangler": "^3.80.0",
310 "vitest": "^1.0.4"
311 }
312}
313```
314
315---
316
317## 🚀 Deployment Process
318
319### 1. Initial Setup (One-time)
320
321```fish
322cd /workspaces/aesthetic-computer/grab
323
324# Install dependencies
325npm install
326
327# Create Browser Rendering binding
328./scripts/setup-browser-binding.fish
329
330# Create Durable Object namespace
331wrangler dispatch-namespace create SCREENSHOT_DO
332
333# Create R2 bucket (optional)
334wrangler r2 bucket create aesthetic-screenshots
335
336# Set secrets
337wrangler secret put BROWSER_API_KEY
338```
339
340### 2. Development
341
342```fish
343# Copy secrets from vault
344cd /workspaces/aesthetic-computer/aesthetic-computer-vault
345./devault.fish
346
347# Start dev server
348cd /workspaces/aesthetic-computer/grab
349npm run dev
350```
351
352### 3. Production Deployment
353
354```fish
355# Deploy to production
356cd /workspaces/aesthetic-computer/grab
357npm run deploy:production
358
359# Configure custom domain (one-time)
360# Via Cloudflare Dashboard:
361# Workers & Pages > aesthetic-grab > Settings > Domains & Routes
362# Add custom domain: grab.aesthetic.computer
363```
364
365---
366
367## 🔄 Migration from Netlify
368
369### Current Endpoint Usage
370
371**Icon Generation:**
372```
373https://aesthetic.computer/icon/128x128/prompt~wipe.png
374```
375
376**Preview Generation (OG Image):**
377```
378https://aesthetic.computer/preview/1200x630/prompt~wipe.png
379```
380
381**Preview Generation (Twitter):**
382```
383https://aesthetic.computer/preview/1800x900/prompt~wipe.png
384```
385
386### Migration Steps
387
3881. **Phase 1: Deploy new worker**
389 - Deploy to `grab.aesthetic.computer`
390 - Test all resolutions
391 - Verify caching behavior
392
3932. **Phase 2: Update references**
394 - Update `/system/netlify/functions/index.mjs` (main HTML generator)
395 - Update any other references to screenshot.js
396 - Add redirects in `netlify.toml`
397
3983. **Phase 3: Remove old function**
399 - Remove `/system/netlify/functions/screenshot.js`
400 - Remove Puppeteer dependencies from package.json
401 - Clean up old dev cache directory
402
403### Files to Update
404
405**Primary:**
406- `/workspaces/aesthetic-computer/system/netlify/functions/index.mjs`
407 - Lines 367-391 (icon and og:image generation)
408
409**Search for references:**
410```fish
411grep -r "screenshot\.js" /workspaces/aesthetic-computer/
412grep -r "/icon/" /workspaces/aesthetic-computer/system/
413grep -r "/preview/" /workspaces/aesthetic-computer/system/
414```
415
416**Add to netlify.toml:**
417```toml
418[[redirects]]
419 from = "/icon/*"
420 to = "https://grab.aesthetic.computer/icon/:splat"
421 status = 200
422 force = true
423
424[[redirects]]
425 from = "/preview/*"
426 to = "https://grab.aesthetic.computer/preview/:splat"
427 status = 200
428 force = true
429```
430
431---
432
433## 💾 Caching Strategy
434
435### Three-tier Caching
436
4371. **Browser Cache** (Client-side)
438 - `Cache-Control: public, max-age=3600`
439 - ETag-based revalidation
440 - 1 hour default
441
4422. **Cloudflare CDN Cache**
443 - `Cloudflare-CDN-Cache-Control: public, durable, max-age=86400`
444 - 24 hours for icon/preview images
445 - Edge caching globally
446
4473. **R2 Storage** (Optional long-term)
448 - Store screenshots for 7+ days
449 - Metadata: URL, resolution, timestamp
450 - Eviction policy: LRU or time-based
451
452### ETag Generation
453```typescript
454// Same as current implementation
455const etagSource = `${resolution}-${filepath}`;
456const etag = `"${crypto.subtle.digestSync('MD5',
457 new TextEncoder().encode(etagSource)
458).toString('hex')}"`;
459```
460
461---
462
463## 🧪 Testing Plan
464
465### Unit Tests
466- URL parsing
467- Resolution validation
468- ETag generation
469- Cache key generation
470
471### Integration Tests
472- Browser rendering
473- Screenshot capture
474- Durable Object coordination
475- Cache retrieval
476
477### E2E Tests
478- Full request/response cycle
479- Multiple resolutions
480- Cache hit/miss scenarios
481- Error handling
482
483### Load Testing
484- Concurrent requests
485- Browser session limits
486- Memory usage
487- Response times
488
489---
490
491## 📊 Monitoring
492
493### Metrics to Track
494- Request count per resolution
495- Cache hit rate
496- Browser session duration
497- Screenshot generation time
498- Error rate
499- P50/P95/P99 latency
500
501### Logging
502- Request details (resolution, URL)
503- Cache hits/misses
504- Browser errors
505- Durable Object coordination
506
507### Alerts
508- Error rate > 5%
509- P95 latency > 10s
510- Browser session failures
511- Cache errors
512
513---
514
515## 🔒 Security Considerations
516
5171. **Rate Limiting**
518 - Limit requests per IP
519 - Limit browser sessions per DO
520 - Prevent DoS attacks
521
5222. **URL Validation**
523 - Only allow aesthetic.computer URLs
524 - Sanitize paths
525 - Prevent SSRF
526
5273. **Resource Limits**
528 - Max screenshot size
529 - Browser timeout
530 - Memory limits per session
531
5324. **Secrets Management**
533 - Store in vault
534 - Use wrangler secrets
535 - Never commit to repo
536
537---
538
539## 🎨 API Design
540
541### Endpoints
542
543**Icon Endpoint:**
544```
545GET /icon/{resolution}/{piece-path}.png
546```
547
548**Preview Endpoint:**
549```
550GET /preview/{resolution}/{piece-path}.png
551```
552
553### Supported Resolutions
554- `128x128` - Favicon/icon
555- `1200x630` - Open Graph image
556- `1800x900` - Twitter card image
557
558### Response Headers
559```
560Content-Type: image/png
561Content-Length: {size}
562ETag: "{hash}"
563Cache-Control: public, max-age=3600
564Cloudflare-CDN-Cache-Control: public, durable, max-age=86400
565X-Cache-Status: HIT | MISS
566X-Screenshot-Time: {ms}
567```
568
569### Error Responses
570
571**400 Bad Request**
572```json
573{
574 "error": "Invalid resolution",
575 "accepted": ["128x128", "1200x630", "1800x900"]
576}
577```
578
579**500 Internal Server Error**
580```json
581{
582 "error": "Screenshot generation failed",
583 "message": "Browser timeout"
584}
585```
586
587**503 Service Unavailable**
588```json
589{
590 "error": "Browser unavailable",
591 "message": "All browser sessions in use"
592}
593```
594
595---
596
597## 🔮 Future Enhancements
598
5991. **Dynamic Resolution Support**
600 - Allow any resolution within limits
601 - Maintain aspect ratio
602
6032. **Video Thumbnails**
604 - Capture frames from videos
605 - Generate GIF previews
606
6073. **PDF Generation**
608 - Full-page PDFs
609 - Print stylesheets
610
6114. **Advanced Caching**
612 - Predictive pre-rendering
613 - Smart eviction policies
614
6155. **A/B Testing**
616 - Multiple screenshot variants
617 - Performance comparison
618
6196. **Analytics**
620 - Most requested pieces
621 - Popular resolutions
622 - Geographic distribution
623
624---
625
626## 📚 References
627
628- [Cloudflare Browser Rendering](https://developers.cloudflare.com/browser-rendering/)
629- [Browser Rendering with Durable Objects](https://developers.cloudflare.com/browser-rendering/workers-bindings/browser-rendering-with-do/)
630- [Puppeteer Documentation](https://pptr.dev/)
631- [Cloudflare Workers Docs](https://developers.cloudflare.com/workers/)
632- [Durable Objects](https://developers.cloudflare.com/durable-objects/)
633- [R2 Storage](https://developers.cloudflare.com/r2/)
634
635---
636
637## ✅ Checklist
638
639### Setup Phase
640- [ ] Create `/grab` directory structure
641- [ ] Set up TypeScript configuration
642- [ ] Install dependencies
643- [ ] Create wrangler.toml from template
644- [ ] Set up vault directory for secrets
645
646### Development Phase
647- [ ] Implement worker entry point
648- [ ] Implement Durable Object
649- [ ] Implement browser rendering logic
650- [ ] Implement caching layer
651- [ ] Add ETag support
652- [ ] Add error handling
653- [ ] Write unit tests
654- [ ] Write integration tests
655
656### Deployment Phase
657- [ ] Create Browser Rendering binding
658- [ ] Create Durable Object namespace
659- [ ] Create R2 bucket (optional)
660- [ ] Set secrets via wrangler
661- [ ] Deploy to dev environment
662- [ ] Test in dev
663- [ ] Deploy to production
664- [ ] Configure custom domain
665- [ ] Test in production
666
667### Migration Phase
668- [ ] Update index.mjs references
669- [ ] Add netlify.toml redirects
670- [ ] Test icon generation
671- [ ] Test preview generation
672- [ ] Verify caching works
673- [ ] Monitor for errors
674- [ ] Compare performance metrics
675
676### Cleanup Phase
677- [ ] Remove screenshot.js
678- [ ] Remove Puppeteer dependencies
679- [ ] Clean up dev cache directory
680- [ ] Update documentation
681- [ ] Archive old implementation
682
683---
684
685**Status:** 📝 Planning Complete
686**Next Steps:** Begin implementation of worker.ts and screenshot-do.ts