nonbinary.computer
OR-1 dataflow CPU sketch
docs: add dataflow graph renderer design plan
Completed brainstorming and design for dfgraph — a web-based
dataflow graph renderer for dfasm programs. Design includes:
- FastAPI backend with progressive pipeline runner + WebSocket
- cytoscape.js frontend with dagre layout
- Logical view (opcode-coloured nodes) and physical view (PE clusters)
- Live reload, error display, SVG/PNG export
- 8 implementation phases
+253
+253
docs/design-plans/2026-02-23-dataflow-renderer.md
+253
docs/design-plans/2026-02-23-dataflow-renderer.md
···
1
+
# Dataflow Graph Renderer Design
2
+
3
+
## Summary
4
+
5
+
This design introduces a web-based dataflow graph renderer for the OR1 assembler's dfasm language. The tool visualises compiled dataflow programs as interactive directed graphs, displaying nodes (representing operations) and edges (representing data dependencies). It provides two complementary views: a **logical view** showing the algorithm's inherent structure with nodes coloured by operation category (arithmetic, routing, memory, etc.), and a **physical view** showing how the program is mapped onto processing elements (PEs) with IRAM offsets and context slots annotated. The renderer runs a progressive assembly pipeline that attempts to advance through all compilation stages (lowering, resolution, placement, allocation) but gracefully handles partial results — errors at any stage still produce a renderable graph with problematic nodes highlighted in red.
6
+
7
+
The implementation uses a two-process architecture: a Python backend (FastAPI + WebSocket) that watches the source `.dfasm` file, re-assembles it on changes, and pushes JSON graph updates to the frontend; and a browser-based frontend (cytoscape.js) that renders the graph interactively with dagre hierarchical layout. Live reload allows developers to edit source code and see the graph update within a second. Export to SVG and PNG enables documentation and sharing. The design reuses existing patterns from the `asm/` package (progressive pipeline execution, error accumulation in `IRGraph.errors`) and integrates cleanly into the Nix-based development environment.
8
+
9
+
## Definition of Done
10
+
11
+
1. A web-based tool (invoked as `python -m dfgraph file.dfasm`) that opens a browser showing the dfasm program as an interactive directed graph.
12
+
2. **Logical view**: circular nodes coloured by opcode category (arithmetic, routing, memory, I/O, config), edges with arrowheads, port annotations (L/R), branch labels (T/F), constant annotations — inspired by but modernised from the reference academic diagram style.
13
+
3. **Physical view**: same graph but nodes grouped into PE cluster boxes, annotated with IRAM offsets and context slots, edges annotated with resolved Addr info (offset, port, target PE).
14
+
4. **Live reload**: watches the source `.dfasm` file and pushes updates via WebSocket on change, re-assembling and re-rendering automatically.
15
+
5. **Error display**: when assembly fails, render the partial graph (whatever nodes/edges are valid) with errors highlighted in red and an error overlay/panel.
16
+
6. **Export**: SVG and PNG export buttons in the UI.
17
+
7. **Nix-friendly**: all dependencies manageable in the existing Nix flake dev environment.
18
+
19
+
## Acceptance Criteria
20
+
21
+
### dataflow-renderer.AC1: Logical view renders dataflow graph
22
+
- **dataflow-renderer.AC1.1 Success:** Valid dfasm source renders as a directed graph with nodes and edges
23
+
- **dataflow-renderer.AC1.2 Success:** Nodes are circular with opcode mnemonic labels centred inside
24
+
- **dataflow-renderer.AC1.3 Success:** Nodes are coloured by opcode category (arithmetic=blue, logic=green, comparison=amber, routing=purple, memory=orange, io=teal, config=grey)
25
+
- **dataflow-renderer.AC1.4 Success:** Edges show port annotations (L/R) at target end and branch labels (T/F) at source end for routing ops
26
+
- **dataflow-renderer.AC1.5 Success:** Function regions render as dashed bounding boxes around their child nodes
27
+
28
+
### dataflow-renderer.AC2: Progressive pipeline produces partial graphs
29
+
- **dataflow-renderer.AC2.1 Success:** Clean source produces allocate-stage graph with zero errors
30
+
- **dataflow-renderer.AC2.2 Success:** Source with errors at any stage still returns a graph with valid nodes
31
+
32
+
### dataflow-renderer.AC3: Physical view shows PE-grouped layout
33
+
- **dataflow-renderer.AC3.1 Success:** Nodes grouped into PE cluster boxes labelled by PE ID
34
+
- **dataflow-renderer.AC3.2 Success:** Nodes within clusters annotated with IRAM offset and context slot
35
+
- **dataflow-renderer.AC3.3 Success:** Cross-PE edges visually distinct from intra-PE edges
36
+
- **dataflow-renderer.AC3.4 Failure:** Physical view unavailable when pipeline hasn't reached allocate stage
37
+
38
+
### dataflow-renderer.AC4: Live reload on file change
39
+
- **dataflow-renderer.AC4.1 Success:** Saving the dfasm file triggers re-render within 1 second
40
+
- **dataflow-renderer.AC4.2 Success:** Initial load renders the graph without manual refresh
41
+
42
+
### dataflow-renderer.AC5: Error display
43
+
- **dataflow-renderer.AC5.1 Success:** Nodes/edges with errors render with red highlight
44
+
- **dataflow-renderer.AC5.2 Success:** Error panel shows error line, category, and message
45
+
- **dataflow-renderer.AC5.3 Success:** Pipeline errors include suggestions when available
46
+
- **dataflow-renderer.AC5.4 Success:** Fixing errors and saving clears error highlights
47
+
- **dataflow-renderer.AC5.5 Failure:** Lark parse error (no graph possible) shows error-only view
48
+
49
+
### dataflow-renderer.AC6: Export
50
+
- **dataflow-renderer.AC6.1 Success:** SVG export produces valid SVG with all visible graph elements
51
+
- **dataflow-renderer.AC6.2 Success:** PNG export produces raster image at readable resolution
52
+
53
+
### dataflow-renderer.AC7: Nix integration
54
+
- **dataflow-renderer.AC7.1 Success:** All Python and JS dependencies resolve in the Nix flake dev shell
55
+
56
+
## Glossary
57
+
58
+
- **dfasm**: The dataflow assembly language used to program the OR1 architecture; programs are written as graphs of operations connected by data edges.
59
+
- **IRGraph**: Intermediate Representation Graph — the compiler's internal representation of a dfasm program, containing nodes (operations), edges (data dependencies), and accumulated errors; produced by the assembler pipeline.
60
+
- **Processing Element (PE)**: A compute unit in the OR1 architecture that executes dataflow operations; each PE has local IRAM and a matching store for operand pairing.
61
+
- **IRAM**: Instruction RAM — local instruction memory within each PE where compiled operations are stored at specific offsets.
62
+
- **Context slot**: A local storage location within a PE's matching store used to buffer operands for multi-input operations; allocated during compilation.
63
+
- **Addr**: An address type in the OR1 ISA representing a destination for computed results, consisting of an offset, port, and target PE identifier.
64
+
- **Port**: Input channel selector for dyadic operations; either `L` (left) or `R` (right), determining operand ordering.
65
+
- **Opcode category**: Classification of operations by function (arithmetic, logic, comparison, routing, memory, I/O, config) used to assign colours in the logical view.
66
+
- **Progressive pipeline**: A compilation strategy that runs assembler passes (lower → resolve → place → allocate) individually, capturing the deepest successful stage even when later passes fail, enabling partial graph visualisation.
67
+
- **cytoscape.js**: A JavaScript graph visualisation library used for rendering and interacting with the dataflow graph in the browser.
68
+
- **dagre**: A directed graph layout algorithm (via the cytoscape-dagre extension) that arranges nodes hierarchically (top-to-bottom) respecting edge directions.
69
+
- **Compound node**: A cytoscape.js feature where nodes can contain child nodes, used for rendering function regions (logical view) and PE clusters (physical view) as bounding boxes.
70
+
- **FastAPI**: A modern Python web framework used to serve the frontend static files and handle WebSocket connections.
71
+
- **Watchdog**: A Python library for monitoring filesystem events; used to detect when the `.dfasm` source file is modified and trigger re-assembly.
72
+
- **esbuild**: A fast JavaScript bundler that compiles the TypeScript frontend source into a single `bundle.js` file served to the browser.
73
+
74
+
## Architecture
75
+
76
+
Two-process system: a Python backend serves graph data over WebSocket, and a browser frontend renders it with cytoscape.js.
77
+
78
+
**Backend** (`dfgraph/` Python package): FastAPI application with a WebSocket endpoint. On startup, reads the target `.dfasm` file, runs the assembler pipeline progressively (see below), and converts the resulting `IRGraph` to a JSON graph structure. Watchdog monitors the source file for changes; on change, re-assembles and pushes the updated graph JSON over WebSocket. A debounce window of 300ms prevents redundant re-assemblies from rapid editor saves.
79
+
80
+
**Frontend** (`dfgraph/frontend/`): Single-page application using cytoscape.js with the cytoscape-dagre layout extension. Receives graph JSON over WebSocket and renders two switchable views (logical and physical). Bundled with esbuild from TypeScript source. Served as a static asset by the FastAPI backend.
81
+
82
+
**CLI entry point**: `python -m dfgraph path/to/file.dfasm [--port 8420]` starts the server, opens the default browser, and watches the file.
83
+
84
+
### Progressive Pipeline Runner
85
+
86
+
The existing `assemble()` function raises `ValueError` on first error. The renderer instead calls individual passes (`lower`, `resolve`, `place`, `allocate`) sequentially, capturing the `IRGraph` at each stage along with accumulated `graph.errors`. This produces the deepest successful pipeline stage:
87
+
88
+
- **Logical view** requires the graph through `resolve` (validated edges).
89
+
- **Physical view** requires `allocate` (PE assignments, IRAM offsets, resolved Addr destinations).
90
+
- If a pass fails, the renderer falls back to the previous stage's graph plus error annotations.
91
+
92
+
This approach works because every pass accumulates errors in `IRGraph.errors` and returns a graph with valid nodes intact — the pass-through error pattern is already established in `asm/`.
93
+
94
+
### WebSocket Protocol
95
+
96
+
Backend pushes `graph_update` messages containing the full JSON graph on every file change. View switching (logical ↔ physical) is purely a frontend concern — the JSON contains all data for both views. The protocol is extensible: a future `execution_step` message type could support emulator integration without structural changes.
97
+
98
+
### Visual Design
99
+
100
+
**Logical view** (default): dagre hierarchical layout (top-to-bottom). Circular nodes with opcode mnemonic labels. Nodes coloured by opcode category:
101
+
102
+
| Category | Opcodes | Colour |
103
+
|----------|---------|--------|
104
+
| Arithmetic | add, sub, inc, dec, shiftl, shiftr, ashiftr | Blue |
105
+
| Logic | and, or, xor, not | Green |
106
+
| Comparison | eq, lt, lte, gt, gte | Amber |
107
+
| Routing | breq, brgt, brge, brof, sweq, swgt, swge, swof, gate, sel, merge, pass, dup | Purple |
108
+
| Memory | read, write, clear, alloc, free, rd_inc, rd_dec, cmp_sw | Orange |
109
+
| I/O | ior, iow, iorw | Teal |
110
+
| Config | const, free_ctx, load_inst, route_set | Grey |
111
+
112
+
Edge annotations: port labels (L/R) at target end, branch labels (T/F) at source end for routing ops, constant values as small labels near const nodes. Function regions (`$func`) rendered as dashed compound bounding boxes. Error nodes/edges get red border and dashed stroke.
113
+
114
+
**Physical view**: nodes reorganised into PE cluster compound nodes (rounded rectangles labelled "PE 0", "PE 1", etc.). Within each cluster, nodes arranged by IRAM offset. Each node annotated with `[iram: N, ctx: M]`. Cross-PE edges rendered prominently (thicker); intra-PE edges lighter. Edge Addr info shown on hover. SM connections shown as edges to an SM cluster box.
115
+
116
+
**Shared UI**: top bar with view toggle and export buttons. Collapsible error panel at bottom with line numbers and error messages. Cytoscape built-in zoom/pan.
117
+
118
+
## Existing Patterns
119
+
120
+
The `dfgraph/` package follows the same structure as `asm/` and `emu/`: a top-level directory with `__init__.py` exporting the public API, individual modules for distinct responsibilities, and no `__main__.py` pattern in existing packages (though `dfgraph/` adds one for CLI invocation).
121
+
122
+
The progressive pipeline runner mirrors the pattern already used in `tests/pipeline.py` and test helpers, which call `lower()`, `resolve()`, etc. individually rather than through `assemble()`.
123
+
124
+
The error accumulation pattern (`IRGraph.errors` list with `AssemblyError` objects containing `SourceLoc`, `ErrorCategory`, and suggestions) is reused directly — `graph_json.py` serialises these errors into the JSON payload for frontend display.
125
+
126
+
No existing web-serving or frontend patterns exist in the codebase. This is new infrastructure.
127
+
128
+
## Implementation Phases
129
+
130
+
<!-- START_PHASE_1 -->
131
+
### Phase 1: Project Scaffolding
132
+
133
+
**Goal:** Create the `dfgraph/` package structure and install dependencies.
134
+
135
+
**Components:**
136
+
- `dfgraph/__init__.py` — empty public API placeholder
137
+
- `dfgraph/__main__.py` — CLI argument parsing (argparse: file path, `--port`)
138
+
- `dfgraph/frontend/package.json` — cytoscape, cytoscape-dagre, esbuild dev dependency
139
+
- `dfgraph/frontend/src/main.ts` — minimal entry point (empty cytoscape init)
140
+
- `dfgraph/frontend/index.html` — minimal page that loads `dist/bundle.js`
141
+
- Python deps: `fastapi`, `uvicorn`, `watchdog` added to project (pip/uv)
142
+
143
+
**Dependencies:** None (first phase)
144
+
145
+
**Done when:** `python -m dfgraph --help` prints usage, `cd dfgraph/frontend && npx esbuild src/main.ts --bundle --outfile=dist/bundle.js` succeeds, FastAPI imports resolve.
146
+
<!-- END_PHASE_1 -->
147
+
148
+
<!-- START_PHASE_2 -->
149
+
### Phase 2: Progressive Pipeline Runner
150
+
151
+
**Goal:** Build the pipeline module that runs assembler passes individually with error capture.
152
+
153
+
**Components:**
154
+
- `dfgraph/pipeline.py` — `run_progressive(source: str)` function that calls `lower()`, `resolve()`, `place()`, `allocate()` in sequence, capturing the deepest successful `IRGraph` and accumulated errors. Returns a result object with the graph, pipeline stage reached, and error list.
155
+
- `dfgraph/categories.py` — opcode-to-category mapping using `ArithOp`, `LogicOp`, `RoutingOp`, `MemOp`, `CfgOp` type dispatch.
156
+
157
+
**Dependencies:** Phase 1 (package exists)
158
+
159
+
**Done when:** Tests verify: (1) clean source produces `allocate`-stage graph with no errors, (2) source with name errors produces `lower`-stage graph with error list, (3) source with placement errors produces `resolve`-stage graph with error list, (4) every opcode in `MNEMONIC_TO_OP` has a category assignment. Covers `dataflow-renderer.AC2.1`, `dataflow-renderer.AC2.2`, `dataflow-renderer.AC5.1`, `dataflow-renderer.AC5.2`, `dataflow-renderer.AC5.3`.
160
+
<!-- END_PHASE_2 -->
161
+
162
+
<!-- START_PHASE_3 -->
163
+
### Phase 3: IR-to-JSON Conversion
164
+
165
+
**Goal:** Convert `IRGraph` at any pipeline stage into JSON for the frontend.
166
+
167
+
**Components:**
168
+
- `dfgraph/graph_json.py` — converts `IRGraph` to a typed dict/dataclass structure: nodes (with id, opcode, category, const, pe, iram_offset, ctx, error flag, region membership), edges (with source, target, port, source_port, addr info, error flag), regions (tag, kind, node ids), errors (line, column, category, message, suggestions), metadata (stage reached, pe_count, sm_count).
169
+
170
+
**Dependencies:** Phase 2 (pipeline runner provides the graph)
171
+
172
+
**Done when:** Tests verify JSON output for (1) a fully allocated graph has all node/edge fields populated, (2) a partially resolved graph omits PE/IRAM fields, (3) error nodes are flagged correctly in JSON, (4) function regions list their child node ids. Covers `dataflow-renderer.AC1.1`.
173
+
<!-- END_PHASE_3 -->
174
+
175
+
<!-- START_PHASE_4 -->
176
+
### Phase 4: Backend Server with WebSocket
177
+
178
+
**Goal:** FastAPI app that serves the frontend and pushes graph updates over WebSocket.
179
+
180
+
**Components:**
181
+
- `dfgraph/server.py` — FastAPI app with: static file serving for `frontend/dist/` and `frontend/index.html`, WebSocket endpoint at `/ws` that sends `graph_update` JSON on connect and on file change, watchdog `Observer` monitoring the target `.dfasm` file with 300ms debounce.
182
+
- `dfgraph/__main__.py` — expanded to start uvicorn and open browser via `webbrowser.open()`.
183
+
184
+
**Dependencies:** Phase 3 (JSON conversion for WebSocket payload)
185
+
186
+
**Done when:** Tests verify: (1) WebSocket client receives valid graph JSON on connect, (2) modifying the watched file triggers a new `graph_update` within 1 second, (3) HTTP GET `/` serves the HTML page. Covers `dataflow-renderer.AC4.1`, `dataflow-renderer.AC4.2`.
187
+
<!-- END_PHASE_4 -->
188
+
189
+
<!-- START_PHASE_5 -->
190
+
### Phase 5: Frontend — Logical View
191
+
192
+
**Goal:** Cytoscape.js rendering of the logical dataflow graph.
193
+
194
+
**Components:**
195
+
- `dfgraph/frontend/src/main.ts` — WebSocket client, cytoscape initialisation, graph data binding on `graph_update` messages.
196
+
- `dfgraph/frontend/src/style.ts` — cytoscape stylesheet: circular node shape, category-based fill colours, edge arrow styles, port label positioning, branch (T/F) and constant annotations, error styling (red border, dashed stroke).
197
+
- `dfgraph/frontend/src/layout.ts` — dagre layout configuration: `rankDir: 'TB'`, node/rank separation tuning for small graphs. Compound node support for function regions (dashed bounding boxes).
198
+
199
+
**Dependencies:** Phase 4 (backend serves data)
200
+
201
+
**Done when:** Loading a valid `.dfasm` file displays a top-to-bottom graph with coloured circular nodes, labelled edges, and function region boxes. Manual verification against reference diagram style. Covers `dataflow-renderer.AC1.1`, `dataflow-renderer.AC1.2`, `dataflow-renderer.AC1.3`, `dataflow-renderer.AC1.4`, `dataflow-renderer.AC1.5`.
202
+
<!-- END_PHASE_5 -->
203
+
204
+
<!-- START_PHASE_6 -->
205
+
### Phase 6: Frontend — Physical View and View Toggle
206
+
207
+
**Goal:** Add the PE-clustered physical view and a toggle between views.
208
+
209
+
**Components:**
210
+
- `dfgraph/frontend/src/layout.ts` — extended with physical layout: PE compound parent nodes, IRAM-offset ordering within clusters, cross-PE edge styling.
211
+
- `dfgraph/frontend/src/main.ts` — view toggle button in top bar, re-layout on switch.
212
+
- `dfgraph/frontend/src/style.ts` — PE cluster styling (rounded rect, label), IRAM/ctx annotation labels, cross-PE vs intra-PE edge differentiation, Addr hover tooltip.
213
+
214
+
**Dependencies:** Phase 5 (logical view working)
215
+
216
+
**Done when:** Toggling to physical view shows nodes grouped by PE with IRAM offset annotations, cross-PE edges are visually distinct. Physical view is only available when pipeline reaches `allocate` stage. Manual verification. Covers `dataflow-renderer.AC3.1`, `dataflow-renderer.AC3.2`, `dataflow-renderer.AC3.3`, `dataflow-renderer.AC3.4`.
217
+
<!-- END_PHASE_6 -->
218
+
219
+
<!-- START_PHASE_7 -->
220
+
### Phase 7: Error Display
221
+
222
+
**Goal:** Render partial graphs with error highlighting and an error panel.
223
+
224
+
**Components:**
225
+
- `dfgraph/frontend/src/style.ts` — error node/edge styles (red border, dashed stroke).
226
+
- `dfgraph/frontend/src/main.ts` — collapsible error panel at bottom of page, populated from `errors` array in graph JSON. Each error shows line number, category, message, and suggestions. Clicking an error highlights the corresponding node (if identifiable).
227
+
228
+
**Dependencies:** Phase 5 (rendering works), Phase 2 (errors in JSON)
229
+
230
+
**Done when:** A `.dfasm` file with errors shows the partial graph with red-highlighted nodes and an error panel listing errors with line numbers. Modifying the file to fix errors clears the highlights. Manual verification. Covers `dataflow-renderer.AC5.1`, `dataflow-renderer.AC5.4`, `dataflow-renderer.AC5.5`.
231
+
<!-- END_PHASE_7 -->
232
+
233
+
<!-- START_PHASE_8 -->
234
+
### Phase 8: Export (SVG + PNG)
235
+
236
+
**Goal:** Add export buttons for SVG and PNG.
237
+
238
+
**Components:**
239
+
- `dfgraph/frontend/src/export.ts` — SVG export via `cy.svg()`, PNG export via `cy.png()`. Triggered by toolbar buttons. Downloads the file with a name derived from the source filename.
240
+
- `dfgraph/frontend/src/main.ts` — export buttons in the top bar.
241
+
242
+
**Dependencies:** Phase 5 (cytoscape rendering)
243
+
244
+
**Done when:** SVG export produces a valid SVG file with all nodes, edges, and labels. PNG export produces a raster image at reasonable resolution. Manual verification. Covers `dataflow-renderer.AC6.1`, `dataflow-renderer.AC6.2`.
245
+
<!-- END_PHASE_8 -->
246
+
247
+
## Additional Considerations
248
+
249
+
**Future emulator integration:** The WebSocket protocol supports adding an `execution_step` message type for visualising token flow through the dataflow graph during emulation. The architecture (backend pushes events, frontend renders) requires no structural changes — only a new message handler and highlighting logic. This is explicitly out of scope for this design.
250
+
251
+
**Nix flake changes:** `fastapi`, `uvicorn`, and `watchdog` need to be added to the Python environment in `flake.nix`. Node.js and npm are already available. The esbuild bundle step is a one-shot command, not a persistent dev server.
252
+
253
+
<!-- freshness: 2026-02-23 -->