aesthetic.computer
Monorepo for Aesthetic.Computer
aesthetic.computer
1# Fuzz
2
3Automated fuzzing/generative testing for Aesthetic Computer pieces using browser automation.
4
5## Overview
6
7This directory contains documentation and tools for automating interactions with Aesthetic Computer pieces through Chrome DevTools Protocol (CDP). The goal is to create generative, randomized inputs that explore the creative space of interactive pieces.
8
9## What We've Learned
10
11### Browser Automation with Aesthetic Computer
12
13Aesthetic Computer is a canvas-based, keyboard-driven environment. Traditional DOM manipulation doesn't work well because most interactions happen through keyboard events rather than clickable elements.
14
15**Key insights:**
16- Always take a screenshot first to understand the visual state
17- Focus on keyboard event dispatching rather than DOM clicks
18- Use `document.dispatchEvent()` with proper KeyboardEvent configuration
19- Include `bubbles: true` and `cancelable: true` for events to propagate
20- Some pieces may require focusing a specific element first (like a textarea)
21
22### Notepat: Musical Fuzzing
23
24Notepat is a musical sequencer piece that responds to keyboard input. We successfully created a fuzzing algorithm that generates randomized musical compositions.
25
26#### Controls
27
28**Notes:** `c`, `d`, `e`, `f`, `g`, `a`, `b`, `h`, `i`, `j`, `k`, `l`, `m`, `n`
29- Each key plays a different note in the scale
30- Hold duration affects the note length
31
32**Octaves:** `3`, `4`, `5`, `6`, `7`, `8` (number keys)
33- Changes the pitch range of subsequent notes
34- 6 octave ranges available
35
36**Wavetypes:** `Tab` key
37- Cycles through 6 different waveform types
38- Changes the timbre/sound character
39
40#### Fuzzing Algorithm
41
42```javascript
43async () => {
44 const notes = ['c', 'd', 'e', 'f', 'g', 'a', 'b', 'h', 'i', 'j', 'k', 'l', 'm', 'n'];
45 const octaves = ['3', '4', '5', '6', '7', '8'];
46
47 let running = true;
48 window.stopNotepat = () => { running = false; };
49
50 while (running) {
51 // Randomly select octave
52 const octave = octaves[Math.floor(Math.random() * octaves.length)];
53 document.dispatchEvent(new KeyboardEvent('keydown', {
54 key: octave,
55 code: `Digit${octave}`,
56 keyCode: 48 + parseInt(octave),
57 bubbles: true,
58 cancelable: true
59 }));
60 await new Promise(r => setTimeout(r, 30));
61 document.dispatchEvent(new KeyboardEvent('keyup', {
62 key: octave,
63 code: `Digit${octave}`,
64 keyCode: 48 + parseInt(octave),
65 bubbles: true,
66 cancelable: true
67 }));
68 await new Promise(r => setTimeout(r, 50));
69
70 // Randomly change wavetype (50% chance)
71 if (Math.random() > 0.5) {
72 document.dispatchEvent(new KeyboardEvent('keydown', {
73 key: 'Tab',
74 code: 'Tab',
75 keyCode: 9,
76 bubbles: true,
77 cancelable: true
78 }));
79 await new Promise(r => setTimeout(r, 30));
80 document.dispatchEvent(new KeyboardEvent('keyup', {
81 key: 'Tab',
82 code: 'Tab',
83 keyCode: 9,
84 bubbles: true,
85 cancelable: true
86 }));
87 await new Promise(r => setTimeout(r, 50));
88 }
89
90 // Play random portion of scale
91 const goingUp = Math.random() > 0.5;
92 const scaleNotes = goingUp ? notes : [...notes].reverse();
93 const numNotes = 3 + Math.floor(Math.random() * 6); // 3-8 notes
94 const startIdx = Math.floor(Math.random() * (scaleNotes.length - numNotes));
95 const notesToPlay = scaleNotes.slice(startIdx, startIdx + numNotes);
96
97 for (const note of notesToPlay) {
98 if (!running) break;
99
100 document.dispatchEvent(new KeyboardEvent('keydown', {
101 key: note,
102 code: `Key${note.toUpperCase()}`,
103 keyCode: note.charCodeAt(0),
104 bubbles: true,
105 cancelable: true
106 }));
107
108 // Random hold duration
109 await new Promise(r => setTimeout(r, 20 + Math.random() * 40));
110
111 document.dispatchEvent(new KeyboardEvent('keyup', {
112 key: note,
113 code: `Key${note.toUpperCase()}`,
114 keyCode: note.charCodeAt(0),
115 bubbles: true,
116 cancelable: true
117 }));
118
119 // Random gap between notes
120 await new Promise(r => setTimeout(r, 10 + Math.random() * 30));
121 }
122 }
123}
124```
125
126**Stop the fuzzer:** Call `window.stopNotepat()` in the console
127
128#### Algorithm Features
129
130- **Octave randomization:** Each phrase picks a random octave (3-8)
131- **Wavetype variation:** 50% chance to change wavetype between phrases
132- **Scale segments:** Plays 3-8 consecutive notes from the scale
133- **Direction:** Randomly goes up or down the scale
134- **Timing variation:**
135 - Note hold: 20-60ms (randomized)
136 - Note gap: 10-40ms (randomized)
137- **Infinite loop:** Runs continuously until stopped
138
139## Potential Pieces to Fuzz
140
141Here are other Aesthetic Computer pieces that could benefit from fuzzing:
142
143### Drawing/Painting Pieces
144- **line:** Random line drawing with varying colors, thicknesses, positions
145- **paint:** Brush strokes with randomized colors, sizes, and positions
146- **wand:** Magic wand tool with random selections and fills
147- **box:** Random rectangles with varying positions, sizes, and colors
148
149### Interactive/Animation Pieces
150- **prompt:** Random command entry and navigation
151- **recorder:** Automated recording with random durations
152- **video:** Random video playback controls
153- **microphone:** Audio input fuzzing (if applicable)
154
155### Text/Typography Pieces
156- **type:** Random text entry with varying fonts and styles
157- **label:** Random label creation with different parameters
158- **freaky-flowers:** Text-to-visual with randomized input
159
160### Mathematical/Generative Pieces
161- **sotce-graph:** Random data visualization parameters
162- **pixels:** Random pixel manipulation patterns
163- **3d-line:** Random 3D path generation
164
165## Fuzzing Strategies
166
167### 1. Parameter Exploration
168Systematically vary all available parameters to map the creative space:
169- Color ranges
170- Size/scale variations
171- Position randomization
172- Timing variations
173
174### 2. Sequence Testing
175Generate random sequences of valid commands:
176- Mouse movements and clicks
177- Keyboard combinations
178- Command sequences in the prompt
179
180### 3. Boundary Testing
181Push pieces to their limits:
182- Maximum values
183- Rapid input
184- Edge cases
185- Invalid combinations
186
187### 4. Emergent Behavior Discovery
188Run long fuzzing sessions to discover:
189- Unexpected interactions
190- Visual glitches (for art!)
191- Performance characteristics
192- Hidden features
193
194## Tools and Setup
195
196### Chrome DevTools Protocol (CDP)
197We use CDP through Claude Code's MCP chrome-devtools server:
198- `mcp__chrome-devtools__evaluate_script`: Execute JavaScript
199- `mcp__chrome-devtools__take_screenshot`: Capture visual state
200- `mcp__chrome-devtools__take_snapshot`: Get DOM state
201- Keyboard/mouse event dispatching
202
203### Timing Considerations
204- Allow time for animations: 30-50ms
205- Network operations: 400-600ms
206- Page loads: 2000-2500ms
207- Note holds: 20-60ms (for musical pieces)
208
209## Future Directions
210
211### Recording and Playback
212- Save interesting fuzzing sequences
213- Replay specific patterns
214- Create "fuzzing compositions" that can be shared
215
216### Machine Learning
217- Train models on successful fuzzing patterns
218- Learn piece-specific parameter distributions
219- Generate increasingly interesting outputs
220
221### Multi-Piece Fuzzing
222- Fuzz the prompt to navigate between pieces
223- Create workflows that span multiple pieces
224- Discover inter-piece interactions
225
226### Performance Testing
227- Stress test pieces with rapid input
228- Monitor memory and CPU usage
229- Identify optimization opportunities
230
231## Contributing
232
233When fuzzing a new piece:
2341. Document the piece's controls and parameters
2352. Create a basic fuzzing algorithm
2363. Note interesting discoveries or emergent behaviors
2374. Add timing recommendations
2385. Include stop mechanisms for infinite loops
239
240## See Also
241
242- [Aesthetic Computer Main Repo](https://github.com/digitpain/aesthetic-computer)
243- [Aesthetic Computer Docs](https://aesthetic.computer/docs)
244- [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/)