aesthetic.computer
Monorepo for Aesthetic.Computer
aesthetic.computer
1# Cross-Platform Sample Storage Plan
2
3## Key Insight: Samples ARE Paintings
4
5AC already has `pixel-sample.mjs` which encodes audio samples as RGB pixel data
6in painting bitmaps. `stample.mjs` uses this today. Paintings have full
7infrastructure: upload, CDN, short codes (`#k3d`), @handle ownership, track-media
8API, and cross-platform rendering.
9
10**Samples use the same `#` sigil and PNG storage format as paintings, but live in
11a separate `samples` MongoDB collection with audio-specific metadata.**
12
13---
14
15## Architecture Decision: Shared Sigil, Separate Collection
16
17### Why `#` (not a new sigil like `^`)
18- Samples and paintings share the same PNG storage format
19- A painting can BE a sample (load any image as audio via stample)
20- A sample IS a painting (the pixel-encoded waveform is visible art)
21- `%` and `&` are URL-unfriendly (`%` is URL escape, `&` is query separator)
22- Keeps the sigil set small: `@` people, `$` code, `#` media, `*` time
23
24### Why separate collection (not a flag on paintings)
25- Clean querying: "list my samples" vs "list my paintings" without filters
26- Separate indexes optimized for audio metadata (duration, sampleRate, etc.)
27- Separate counts/quotas per media type
28- Future-proof: audio-specific features (waveform preview, BPM detection)
29
30### Shared code namespace
31- `#` codes must be unique across BOTH `paintings` AND `samples` collections
32- `generateUniqueCode()` checks both collections before assigning
33- Other sigils (`$` KidLisp, `*` clock, tapes) have independent namespaces
34- The code resolver checks both collections to find which type a `#code` refers to
35
36---
37
38## MongoDB Schema
39
40### Collection: `samples`
41```js
42{
43 _id: ObjectId,
44 user: "auth0|63effeeb...", // owner (null for guest)
45 slug: "kick-drum", // user-friendly name
46 code: "k3d", // unique short code (shared with paintings)
47 when: ISODate, // upload timestamp
48
49 // Sample-specific metadata
50 v: 1, // pixel-sample encoding version
51 sampleRate: 48000,
52 sampleLength: 240000, // exact sample count
53 duration: 5.0, // seconds
54 channels: 1,
55 source: "native", // or "web"
56
57 // Standard media fields
58 ext: "png", // storage format
59 width: 256, // image dimensions
60 height: 313,
61}
62```
63
64### Indexes
65```js
66// Unique code (shared namespace with paintings — enforced at generation time)
67await samples.createIndex({ code: 1 }, { unique: true, sparse: true });
68
69// User queries: "list my samples"
70await samples.createIndex({ user: 1 });
71
72// Audio-specific queries
73await samples.createIndex({ duration: 1 }); // sort by length
74await samples.createIndex({ "source": 1 }); // native vs web
75await samples.createIndex({ v: 1 }); // encoding version (for migration)
76await samples.createIndex({ when: -1 }); // recent first
77await samples.createIndex({ user: 1, slug: 1 }, { unique: true }); // per-user slugs
78```
79
80### Encoding Version Contract
81```
82v1 (current — pixel-sample.mjs):
83 - 3 audio samples per pixel (R, G, B channels)
84 - Float range -1.0..+1.0 mapped to 0..255
85 - A channel: 255 (opaque)
86 - Width: 256px (configurable)
87 - Height: ceil(sampleLength / 3 / width)
88 - Mono only
89 - The `v` field MUST be stored on every record so decoders know which algorithm to use
90```
91
92---
93
94## Implementation Plan
95
96### Phase 1: Backend — track-media + code generation
97**Files to modify:**
98
991. **`system/netlify/functions/track-media.mjs`**
100 - Add `mediaType: "sample"` branch for PNG uploads with sample metadata
101 - Route to `samples` collection instead of `paintings`
102 - Store `v`, `sampleRate`, `sampleLength`, `duration`, `channels`, `source`
103
1042. **`system/backend/generate-short-code.mjs`**
105 - `generateUniqueCode()` accepts optional `siblingCollections` array
106 - For `#` codes: checks both `paintings` AND `samples` before assigning
107 - Other types unchanged (single collection check)
108
1093. **`system/netlify/functions/painting-code.mjs`** (or equivalent resolver)
110 - When resolving `#code`, check `paintings` first, then `samples`
111 - Return `{ type: "painting" | "sample", ...record }` so the client knows
112
1134. **New: `system/netlify/functions/list-samples.mjs`** (or extend existing)
114 - `GET /api/samples/@handle` → list user's samples
115 - `GET /api/samples/@handle/:slug` → get specific sample
116 - Returns CDN URLs + metadata
117
118### Phase 2: Native pixel-sample bridge
119**Files to create/modify:**
120
1211. **`fedac/native/pieces/lib/pixel-sample-native.mjs`**
122 - Pure-JS encode/decode (no DOM, no Canvas — works in QuickJS)
123 - Same algorithm as web `pixel-sample.mjs`
124 - `encodeSampleToBitmap(float32Array, width)` → RGBA pixel array
125 - `decodeBitmapToSample(rgbaArray, width, height, sampleLength)` → float32[]
126
1272. **PNG write from native**
128 - Option A: Add `stb_image_write.h` (single-header, ~1KB) for PNG encoding in C
129 - Option B: Minimal PNG writer in JS (deflate + PNG header — ~100 lines)
130 - Option C: BMP format (simpler, no compression, server converts to PNG on upload)
131
132### Phase 3: Upload from native
133**Flow:**
134```
135Record audio → float32[] → encodeSampleToBitmap → PNG bytes
136 → POST /api/track-media { ext:"png", mediaType:"sample", sampleMeta:{v:1,...} }
137 → GET presigned URL → PUT PNG to Spaces → #code returned
138```
139
140**Files:**
141- `fedac/native/pieces/samples.mjs` — add upload key (`u`)
142- `fedac/native/src/js-bindings.c` — `system.uploadMedia()` binding if needed
143- Or use existing `system.fetch()` + `system.fetchPost()` for the API calls
144
145### Phase 4: Download to native
146**Flow:**
147```
148Type #code → resolve via /api/painting-code → get CDN URL
149 → fetchBinary(url, /tmp/sample.png) → decode PNG → decodeBitmapToSample
150 → sound.sample.loadData(float32, rate) → play
151```
152
153**Requires:** PNG decoding in native
154- Option A: `stb_image.h` (single-header PNG decoder)
155- Option B: Decode in JS (pure-JS PNG inflate)
156- Option C: Server endpoint that returns raw PCM (avoid client-side PNG decode)
157
158### Phase 5: Unified experience
159- **notepat.mjs**: type `#code` to load a sample from cloud into sample bank
160- **samples.mjs**: browse/record/upload/download on web and native
161- **stample.mjs**: already works, becomes the web sample player
162- **Gallery**: shows speaker icon on `#` codes that are samples
163- **Profile**: separate "samples" tab alongside "paintings"
164
165---
166
167## Data Flow
168
169```
170Native Record Web Record
171 | |
172 float32[] float32[]
173 | |
174 encodeSampleToBitmap encodeSampleToBitmap
175 | |
176 PNG bytes PNG bytes (via Canvas)
177 | |
178 POST /api/track-media POST /api/track-media
179 { mediaType:"sample" } { mediaType:"sample" }
180 | |
181 +----→ DO Spaces ←------+
182 + MongoDB "samples"
183 + short code (#abc)
184 |
185 +------------+------------+
186 | |
187 download PNG download PNG
188 | |
189 decodeBitmapToSample decodeBitmapToSample
190 | |
191 float32[] float32[]
192 | |
193 audio playback audio playback
194```
195
196## Storage Math
197
198| Duration | Samples @48kHz | Pixels (3/px) | Image (256w) | PNG size |
199|----------|---------------|---------------|--------------|----------|
200| 1 sec | 48,000 | 16,000 | 256×63 | ~20 KB |
201| 5 sec | 240,000 | 80,000 | 256×313 | ~100 KB |
202| 10 sec | 480,000 | 160,000 | 256×625 | ~200 KB |
203
204## Key Files
205
206| Existing | Purpose |
207|----------|---------|
208| `system/public/aesthetic.computer/lib/pixel-sample.mjs` | Encode/decode samples↔bitmaps |
209| `system/public/aesthetic.computer/disks/stample.mjs` | Web sample-painting player |
210| `system/netlify/functions/track-media.mjs` | Media upload API |
211| `system/netlify/functions/painting-code.mjs` | Short code → slug resolver |
212| `system/backend/generate-short-code.mjs` | Unique code generation |
213| `system/netlify/functions/presigned-url.js` | CDN upload/download URLs |
214
215| New/Modified | Purpose |
216|-------------|---------|
217| `fedac/native/pieces/lib/pixel-sample-native.mjs` | Pure-JS encode/decode for QuickJS |
218| `fedac/native/pieces/samples.mjs` | Native sample browser + upload/download |
219| `system/public/aesthetic.computer/disks/samples.mjs` | Web sample browser piece |
220| `system/netlify/functions/track-media.mjs` | Add sample branch |
221| `system/backend/generate-short-code.mjs` | Cross-collection check for # codes |