aesthetic.computer
Monorepo for Aesthetic.Computer
aesthetic.computer
plan: cross-platform samples — store samples as paintings
Samples encoded as RGB pixel bitmaps using existing pixel-sample.mjs,
stored via the painting infrastructure (CDN, short codes, @handles).
Record on native → upload as painting → share code → play on web.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
+199
+199
plans/cross-platform-samples.md
+199
plans/cross-platform-samples.md
···
1
+
# Cross-Platform Sample Storage Plan
2
+
3
+
## Key Insight: Samples ARE Paintings
4
+
5
+
AC already has `pixel-sample.mjs` which encodes audio samples as RGB pixel data
6
+
in painting bitmaps. `stample.mjs` uses this today. Paintings have full
7
+
infrastructure: upload, CDN, short codes (`#k3d`), @handle ownership, track-media
8
+
API, and cross-platform rendering.
9
+
10
+
**We don't need a new storage system. Samples should be stored as paintings.**
11
+
12
+
Record audio → encode as bitmap → save as painting → get a short code → share.
13
+
Anyone loads the painting back as audio on web or native.
14
+
15
+
---
16
+
17
+
## Current State
18
+
19
+
### Web (stample.mjs + pixel-sample.mjs)
20
+
- `encodeSampleToBitmap(data, width)` — float32[] → RGB pixels (3 samples/pixel)
21
+
- `decodeBitmapToSample(bitmap, meta)` — RGB pixels → float32[]
22
+
- `loadPaintingAsAudio(source, opts)` — load a painting code/object as playable audio
23
+
- Paintings upload via `track-media.mjs` → DO Spaces + MongoDB + short code
24
+
- Download via `/media/@handle/painting/slug.png` or `/media/paintings/CODE.png`
25
+
26
+
### Native (ac-native)
27
+
- Samples stored as raw float32 PCM (rate + length + data) at `/mnt/ac-sample.raw`
28
+
- `audio_sample_save()` / `audio_sample_load()` in C
29
+
- `sound.sample.saveTo(path)` / `sound.sample.loadFrom(path)` JS bindings
30
+
- `sound.sample.getData()` → Float32Array, `sound.sample.loadData(f32, rate)` → load
31
+
- `/mnt/samples/` directory with `manifest.json` for local sample library
32
+
- No pixel-sample encoding yet, no painting upload capability
33
+
34
+
---
35
+
36
+
## Plan
37
+
38
+
### Phase 1: Native Pixel-Sample Bridge
39
+
40
+
**Add pixel-sample encoding/decoding to native JS pieces.**
41
+
42
+
Since native pieces run in QuickJS (no DOM, no Canvas), implement a pure-JS
43
+
version of the encode/decode that works without browser APIs:
44
+
45
+
**File: `fedac/native/pieces/lib/pixel-sample-native.mjs`**
46
+
- `encodeSampleToBitmap(float32Array, width)` — same algorithm as web version
47
+
(float32 → 8-bit RGB, 3 samples per pixel)
48
+
- `decodeBitmapToSample(rgbaArray, width, height, sampleLength)` — reverse
49
+
- These are pure math — no DOM needed
50
+
51
+
**Modify: `fedac/native/pieces/samples.mjs`**
52
+
- When saving, also encode sample as bitmap PNG
53
+
- Store both `.raw` (for instant native reload) and `.png` (for upload/sharing)
54
+
55
+
**Requires:** A way to write PNG from native. Options:
56
+
1. Use the existing `graph.c` framebuffer snapshot capability
57
+
2. Add a minimal PNG writer in C (stb_image_write.h is ~1KB)
58
+
3. Encode as BMP (simpler header, no compression) — paintings can be any image format
59
+
60
+
### Phase 2: Native Painting Upload
61
+
62
+
**Let native devices upload paintings to the AC cloud.**
63
+
64
+
**Add C binding: `system.uploadPainting(localPath, handle, token)`**
65
+
- POST to `/api/track-media` with painting metadata
66
+
- GET presigned upload URL
67
+
- PUT the image file to DO Spaces
68
+
- Returns short code on success
69
+
70
+
**Or simpler: POST the bitmap directly.**
71
+
- Encode sample as pixel data in JS
72
+
- Use `system.fetchPost()` to send base64-encoded bitmap to a new
73
+
`/api/sample-painting` endpoint that:
74
+
1. Decodes the base64 bitmap
75
+
2. Renders it as PNG via sharp/canvas
76
+
3. Uploads to DO Spaces via existing painting pipeline
77
+
4. Returns a short code
78
+
79
+
**Update samples.mjs:** Add `u` (upload) key that encodes the current sample
80
+
as a painting bitmap and uploads it. Shows the short code on success.
81
+
82
+
### Phase 3: Native Painting Download (Load Remote Samples)
83
+
84
+
**Let native devices load samples from painting codes.**
85
+
86
+
**Flow:**
87
+
1. User types a painting code (e.g., `#k3d`) in samples.mjs
88
+
2. `system.fetch("/api/painting-code?code=k3d")` → get slug + handle
89
+
3. `system.fetchBinary("https://aesthetic.computer/media/paintings/k3d.png", "/tmp/sample.png")`
90
+
4. Decode PNG → extract RGB pixels → `decodeBitmapToSample()` → load into audio
91
+
92
+
**Requires:** PNG decoding in native. Options:
93
+
1. Add `stb_image.h` to the C build (single-header PNG decoder, tiny)
94
+
2. Decode in JS using a pure-JS PNG decoder
95
+
3. Use the existing `graph.c` image loading if it supports PNG
96
+
97
+
### Phase 4: Unified samples.mjs (Web + Native)
98
+
99
+
**Create a web `samples.mjs` piece that mirrors the native one.**
100
+
101
+
**File: `system/public/aesthetic.computer/disks/samples.mjs`**
102
+
103
+
Uses the existing painting infrastructure:
104
+
- List user's paintings that are tagged as samples (metadata flag)
105
+
- Record new sample via `Microphone` API
106
+
- Encode → upload as painting via `track-media`
107
+
- Browse + play back via `loadPaintingAsAudio()`
108
+
- Share via short code
109
+
- Compatible with native — same painting, same code, playable on both
110
+
111
+
### Phase 5: notepat Cross-Platform Samples
112
+
113
+
**Both web and native notepat can load sample paintings.**
114
+
115
+
**Native notepat.mjs:**
116
+
- Add `stample CODE` command or sample bank that loads from painting codes
117
+
- After recording, offer to save as painting (upload to cloud)
118
+
- Saved samples show their short code in the status bar
119
+
120
+
**Web notepat.mjs:**
121
+
- Already has `stample.mjs` as a sibling piece
122
+
- Add sample mode that uses `loadPaintingAsAudio()` to load from codes
123
+
- Shared sample format means a sample recorded on bare metal can be
124
+
played in the browser and vice versa
125
+
126
+
---
127
+
128
+
## Data Flow
129
+
130
+
```
131
+
Native Record Web Record
132
+
| |
133
+
float32[] float32[]
134
+
| |
135
+
encodeSampleToBitmap encodeSampleToBitmap
136
+
| |
137
+
RGB pixels RGB pixels
138
+
| |
139
+
upload as painting upload as painting
140
+
| |
141
+
+----→ DO Spaces ←------+
142
+
+ MongoDB
143
+
+ short code (#k3d)
144
+
|
145
+
+------------+------------+
146
+
| |
147
+
download PNG download PNG
148
+
| |
149
+
decodeBitmapToSample decodeBitmapToSample
150
+
| |
151
+
float32[] float32[]
152
+
| |
153
+
audio playback audio playback
154
+
```
155
+
156
+
## Why Paintings, Not WAV
157
+
158
+
1. **Infrastructure exists** — upload, CDN, short codes, @handle, MongoDB, all done
159
+
2. **Visual** — you can SEE the sample as an image, share it as art
160
+
3. **Compact** — 8-bit RGB is ~4x smaller than float32 WAV for the same data
161
+
4. **Cross-platform** — PNG/image works everywhere, WAV needs special handling
162
+
5. **Social** — paintings are already the shareable unit in AC, samples become paintings
163
+
6. **stample.mjs already does this** — proven format, just extend it
164
+
165
+
## Storage Format
166
+
167
+
```
168
+
Painting PNG (256px wide, height varies):
169
+
R channel: sample[i*3+0] → 8-bit (mapped from -1..1 to 0..255)
170
+
G channel: sample[i*3+1]
171
+
B channel: sample[i*3+2]
172
+
A channel: 255 (opaque)
173
+
174
+
Metadata (in MongoDB painting record):
175
+
type: "sample"
176
+
sampleLength: number (exact sample count)
177
+
sampleRate: 48000
178
+
durationSecs: number
179
+
source: "native" | "web"
180
+
```
181
+
182
+
A 5-second 48kHz sample = 240,000 samples = 80,000 pixels = 256×313 PNG ≈ 100KB.
183
+
184
+
## Key Files
185
+
186
+
| Existing | Purpose |
187
+
|----------|---------|
188
+
| `system/public/aesthetic.computer/lib/pixel-sample.mjs` | Encode/decode samples↔paintings |
189
+
| `system/public/aesthetic.computer/disks/stample.mjs` | Web sample-painting player |
190
+
| `system/netlify/functions/track-media.mjs` | Painting upload API |
191
+
| `system/netlify/functions/painting-code.mjs` | Short code → slug resolver |
192
+
| `system/netlify/functions/presigned-url.js` | CDN upload/download URLs |
193
+
194
+
| New/Modified | Purpose |
195
+
|-------------|---------|
196
+
| `fedac/native/pieces/lib/pixel-sample-native.mjs` | Pure-JS encode/decode for native |
197
+
| `fedac/native/pieces/samples.mjs` | Add upload/download/code support |
198
+
| `system/public/aesthetic.computer/disks/samples.mjs` | Web sample browser piece |
199
+
| `fedac/native/src/js-bindings.c` | PNG read/write bindings if needed |