![Aether OS logo](logo-aetheros.jpg) # Aether Substrate > [!WARNING] > **Experimental software — use at your own discretion.** > > Aether Substrate is in early development. It runs LLM agents that execute code in containers, brokers LLM provider credentials, and can expose services to the internet via tunnels. Breaking changes are likely. > > The software is provided **AS IS, with no warranty**. The authors and contributors accept **no liability** for hardware damage, data loss, security incidents, API spend, or any other harm arising from use of this project. See [docs/DISCLAIMER.md](./docs/DISCLAIMER.md) for the full disclaimer and [LICENSE](./LICENSE) for legal terms. Containerized Linux environment that powers [Aether OS](https://aetheros.computer). Allows you to run atproto tools like Tap and Goat, use local LLMs to work with containers and a dedicated workspace, and power Aether OS system services for doing more with your local AT Protocol data. ## What It Powers in Aether OS - **ATProto firehose + repo tracking**: Tap indexer with SQLite storage, runtime repo management, and WebSocket event streaming. - **GOAT CLI access**: resolve handles and DIDs, fetch records, list collections, run curated Goat commands through an authenticated gateway. - **Semantic search**: optional EmbeddingGemma vector indexing with Tap sync, backfills, similar-record lookup, and vector export. - **Persistent agent workspace**: long-lived container with file I/O, foreground/background exec, guarded installs, SSE events, and terminal access. - **Managed sandbox containers**: allowlisted images with CPU/memory caps, toggleable internet, repo cloning, exec, background jobs, and HTTP app proxying. - **LLM gateway**: server-side provider credentials, pluggable providers, and OpenAI-compatible streaming chat with tool use. - **Web access utilities**: plain scrape, search, and agentic browse for when a fetch is not enough. - **Local-first networking**: binds to `127.0.0.1` by default, local HTTPS via Caddy, optional Cloudflare Tunnel or Tailscale for remote access. - **Security guardrails**: authenticated REST/WebSocket, signed proxy tokens, origin checks, rate limits, size limits, SSRF checks, and image allowlists. ## Requirements | Requirement | Details | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Container runtime** | Any OCI runtime with a Docker-compatible CLI and a Compose plugin. See [Supported runtimes](#supported-runtimes) below. | | **Compose** | Compose v2 (`docker compose` / `podman compose`). The standalone `docker-compose` v1 binary is not supported. | | **Disk** | ~4 GB for the image + model cache. Vector search adds ~150 MB of ONNX model downloads on first run. | | **Memory** | The core stack typically sits around ~600 MB RAM without workspace containers. Plan for about 1 GB for Substrate + Tap + Caddy, then add headroom for workspace containers, managed containers, embeddings, or model workloads. `docker-compose.yml` sets a 4 GB upper limit for headroom by default; that is not the minimum required. | | **OS** | macOS, Linux, or Windows with WSL2. The source build works on ARM64 and x86_64 hosts. | | **Node.js (CLI only)** | Node 20+ is required to run the `aether-substrate` CLI. Only needed if you want the interactive operator workflow. Running the stack directly via `docker compose` / `podman compose` has no Node dependency. | ## Installation Use one of these paths: 1. **Substrate CLI**: recommended for most people. It handles `.env`, runtime detection, build/start, health checks, and optional tunnel setup. 2. **Build it yourself**: use this if you are changing the container, API, or compose files directly. ### Path 1: Substrate CLI (recommended) Best for: **getting a working local Substrate quickly with the least manual setup.** The `aether-substrate` CLI wraps build, startup, config editing, health checks, and remote-access setup. It auto-detects Docker or Podman on your PATH. ```bash # Clone this repo if you do not already have it git clone git@tangled.org:aetheros.computer/aether-substrate cd aether-substrate # Build and install the CLI from the local checkout cd cli npm install npm run build npm link cd .. # Interactive bootstrap: env, feature flags, build, startup, optional tunnels aether-substrate init # Useful follow-up commands aether-substrate doctor aether-substrate status aether-substrate logs aether-substrate config --enable containers --enable workspace --restart aether-substrate tunnel cloudflare --hostname substrate.example.com aether-substrate tunnel tailscale ``` Notes: - `aether-substrate init --runtime podman` forces Podman instead of Docker autodetection. - `aether-substrate up` is the non-interactive path once your `.env` is already configured. - The CLI resolves the `substrate/` directory automatically, so it does not matter where you run it from after linking it. - Once the stack is up, `docker compose ps` or Docker Desktop should show at least `aether-substrate` and `aether-caddy`. If workspace support is enabled, you will also see `aetheros-user-workspace`. ### Path 2: Build It Yourself From Source Best for: **custom images, local development, debugging, and changing compose or Docker behavior directly.** ```bash git clone git@tangled.org:aetheros.computer/aether-substrate cd aether-substrate cp .env.example .env # Edit .env and set SUBSTRATE_API_KEY (generate one: openssl rand -hex 32) docker compose up -d --build # or: podman compose up -d --build curl http://127.0.0.1:3100/health docker compose ps docker compose logs -f substrate ``` This path builds from the local [`Dockerfile`](Dockerfile) and the [`substrate-api/`](substrate-api/) source tree. Once the stack is up, `docker compose ps` or Docker Desktop should show at least `aether-substrate` and `aether-caddy`. If workspace support is enabled, you will also see `aetheros-user-workspace`. ## Architecture ``` ┌─────────────────────────────────────────────────┐ │ Host Machine │ │ │ │ ┌──────────┐ ┌────────────────────────────┐ │ │ │ Caddy │───▶│ Substrate Container │ │ │ │ :8443 │ │ │ │ │ │ (HTTPS) │ │ ┌──────────────────────┐ │ │ │ └──────────┘ │ │ Hono API :3100 │ │ │ │ │ │ REST + WebSocket │ │ │ │ │ └──────────────────────┘ │ │ │ │ ┌──────┐ ┌─────┐ │ │ │ │ │ Tap │ │ PTY │ │ │ │ │ │:2480 │ │ │ │ │ │ │ └──────┘ └─────┘ │ │ │ │ ┌──────────────────────┐ │ │ │ │ │ /data (SQLite vol) │ │ │ │ │ └──────────────────────┘ │ │ │ └────────────────────────────┘ │ └─────────────────────────────────────────────────┘ ``` ## Services | Service | Port | Description | | ---------------- | ---- | ----------------------------------- | | Substrate API | 3100 | Hono REST/WebSocket gateway | | Tap | 2480 | ATProto firehose indexer (Go) | | Embedding server | 3101 | Internal ONNX embedding server (Go) | | Caddy | 8443 | HTTPS reverse proxy (internal TLS) | ## API Endpoints ### Public - `GET /health`: Container and service health ### Authenticated (require `x-substrate-key` header) #### System - `GET /services`: Detailed service status - `GET /fs/read`: Read an allowed file as base64 - `POST /exec`: Run legacy allowlisted commands when `SUBSTRATE_ENABLE_EXEC=true` #### Repo Management - `GET /repos`: List tracked repos - `GET /collections`: List indexed collections - `GET /records`: Query indexed records - `POST /repos/add`: Add DIDs to track - `POST /repos/remove`: Remove tracked DIDs - `GET /repos/:did`: Get repo info #### Goat CLI - `GET /goat/resolve/:identifier`: Resolve handle/DID - `GET /goat/get?uri=`: Get AT Protocol record - `GET /goat/ls/:did`: List records - `POST /goat/exec`: Execute Goat command #### Containers - `GET /containers`: List managed containers - `POST /containers`: Create container - `GET /containers/images/allowed`: Get image allowlist - `GET /containers/:id/logs`: Fetch logs - `POST /containers/clone`: Create container and clone repo - `POST /containers/:id/clone`: Clone repo into existing container - `GET /containers/:id/files`: List files - `GET /containers/:id/files/read`: Read file - `POST /containers/:id/files/write`: Write file - `POST /containers/:id/exec`: Execute command in `shell` or `argv` mode - `GET /containers/:id/jobs`: List background jobs - `GET /containers/:id/jobs/:jobId`: Inspect one background job - `POST /containers/:id/jobs/:jobId/cancel`: Cancel a background job - `DELETE /containers/:id`: Remove container #### Workspace - `POST /workspace/ensure`: Create or fetch the persistent workspace container - When workspace is enabled, Substrate also bootstraps this container during startup. - `GET /workspace/files`: List workspace files - `GET /workspace/files/read`: Read a workspace file - `POST /workspace/files/write`: Write a workspace file - `POST /workspace/exec`: Execute a workspace command in `shell` or `argv` mode - `POST /workspace/install`: Install packages with strict validation - `GET /workspace/jobs`: List workspace background jobs - `GET /workspace/jobs/:jobId`: Inspect a workspace background job - `POST /workspace/jobs/:jobId/cancel`: Cancel a workspace background job #### WebSocket - `WS /ws?type=events`: Stream Tap firehose events - `WS /ws?type=terminal`: Substrate host PTY terminal session - `WS /ws?type=container-terminal&workspace=1`: Workspace terminal - `WS /ws?type=container-terminal&containerId=X`: Managed container terminal WebSocket auth uses a first message of `{ "type": "auth", "key": "..." }`. Raw API keys in WebSocket query strings are intentionally rejected. ## Configuration ### Environment Variables | Variable | Default | Description | | ------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `SUBSTRATE_API_KEY` | _(required)_ | Shared auth key. Generate with `openssl rand -hex 32`. | | `CONTAINER_RUNTIME` | `docker` (auto-detected) | Container CLI the `aether-substrate` tool shells out to. Supported: `docker`, `podman`, `nerdctl`, or any docker-compatible binary on your PATH. | | `DOCKER_GID` | `20` on macOS, Linux-dependent elsewhere | Container socket group ID. For rootless Podman, set to your user GID (`id -g`). | | `PORT` | `3100` | API server port | | `TAP_DATABASE_URL` | `sqlite:///data/tap.db` | Tap SQLite database path | | `TAP_URL` | `http://localhost:2480` | Tap base URL used by the API | | `TAP_RELAY_URL` | `relay1.us-east.bsky.network` | ATProto relay to connect to | | `TAP_DISABLE_ACKS` | `true` | Fire-and-forget mode | | `TAP_LOG_LEVEL` | `info` | Tap log verbosity | | `TAP_FULL_NETWORK` | `false` | Index full network (warning: heavy) | | `TAP_COLLECTION_FILTERS` | `computer.aetheros.*,app.bsky.*` | Comma-separated collection patterns | | `SUBSTRATE_ALLOWED_ORIGINS` | _(empty)_ | Extra allowed HTTP/WebSocket origins | | `SUBSTRATE_ENABLE_EXEC` | `false` | Enable legacy system exec endpoint | | `SUBSTRATE_ENABLE_CONTAINERS` | `false` | Enable container management (Docker/Podman socket required inside the container) | | `SUBSTRATE_MAX_CONTAINERS` | `5` | Max managed containers | | `SUBSTRATE_CONTAINER_MAX_CPU` | `1.0` | CPU limit per container | | `SUBSTRATE_CONTAINER_MAX_MEMORY_MB` | `512` | Memory limit per container (MB) | | `SUBSTRATE_CONTAINER_IMAGE_ALLOWLIST` | built-in allowlist | Images allowed for managed containers | | `SUBSTRATE_ENABLE_WORKSPACE` | `false` | Enable workspace container and pre-create it during Substrate startup | | `SUBSTRATE_WORKSPACE_IMAGE` | `node:22-alpine` | Workspace container image | | `SUBSTRATE_ENABLE_TERMINAL_WS` | `false` | Enable Substrate host terminal WebSocket | | `SUBSTRATE_HOST` | `localhost` | Hostname for Caddy HTTPS proxy | | `SUBSTRATE_HTTPS_PORT` | `8443` | HTTPS port for Caddy | ### .env.example ```bash # Required. Generate with: openssl rand -hex 32 SUBSTRATE_API_KEY= # Container runtime used by the CLI. docker (default), podman, nerdctl, or a custom binary # CONTAINER_RUNTIME=docker # Container socket group ID # macOS: 20 (staff) | Linux docker: `getent group docker | cut -d: -f3` | Rootless Podman: `id -g` DOCKER_GID=20 # Feature flags SUBSTRATE_ENABLE_CONTAINERS=true SUBSTRATE_ENABLE_WORKSPACE=true SUBSTRATE_ENABLE_TERMINAL_WS=false ``` ## Networking & Remote Access By default, all ports bind to `127.0.0.1`, so the Substrate is only accessible from the local machine. For remote or persistent HTTPS access, use one of these tunnel options: | Method | Best For | Guide | | --------------------- | ------------------------------------------------ | ------------------------------------------ | | **Cloudflare Tunnel** | Always-on HTTPS, custom domains, zero open ports | [docs/cloudflared.md](docs/cloudflared.md) | | **Tailscale** | Private mesh VPN, team access, MagicDNS | [docs/tailscale.md](docs/tailscale.md) | | **Caddy (built-in)** | Local HTTPS, LAN access | Already included in docker-compose | See [docs/networking.md](docs/networking.md) for an overview of all networking options and when to use each. ## Data Persistence Data is stored in a Docker volume `substrate-data` mounted at `/data`: - `tap.db`: SQLite database with tracked repos, records, and events - `models/`: ONNX model cache for the embedding server ### Project Structure ``` substrate/ ├── Dockerfile # Multi-stage build (Go + Node.js) ├── Caddyfile # HTTPS reverse proxy config ├── docker-compose.yml # Full dev composition ├── docker-compose.cloudflared.yml # Cloudflare tunnel overlay ├── docs/ │ ├── networking.md # Networking overview │ ├── cloudflared.md # Cloudflare Tunnel guide │ └── tailscale.md # Tailscale guide ├── cli/ # `aether-substrate` operator CLI (Node) ├── tools/ │ ├── export-standalone-repo.sh # Fresh standalone repo export helper │ └── embedding-server/ # Go ONNX embedding service └── substrate-api/ ├── src/ │ ├── index.ts # Entry point (HTTP + WebSocket) │ ├── app.ts # Hono app, middleware, routes │ ├── config.ts # Environment config │ ├── routes/ # API route handlers │ │ ├── containers.ts │ │ ├── embeddings.ts │ │ ├── events.ts │ │ ├── goat.ts │ │ ├── llm.ts │ │ ├── system.ts │ │ ├── tap.ts │ │ ├── vm.ts │ │ ├── web.ts │ │ └── workspace.ts │ ├── services/ # Business logic │ │ ├── container-manager.ts │ │ ├── db.ts │ │ ├── embeddings/ │ │ ├── execution-core.ts │ │ ├── goat.ts │ │ ├── llm.ts │ │ ├── lsp-bridge.ts │ │ ├── scraper.ts │ │ ├── service-manager.ts │ │ ├── tap.ts │ │ ├── terminal.ts │ │ ├── vm-registry.ts │ │ └── workspace-watcher.ts │ ├── types/ │ └── utils/ ├── package.json └── tsconfig.json ``` ## Supported runtimes | Runtime | Status | Notes | | -------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Docker Desktop | Supported | Reference implementation. | | OrbStack | Supported | Ships a drop-in `docker` CLI; nothing else to configure. | | Rancher Desktop | Supported | Use the `dockerd` (moby) backend. | | Colima | Supported | `colima start` gives you a working `docker` CLI. | | Podman (4.6+) | Supported | Set `CONTAINER_RUNTIME=podman`. Requires either the built-in `podman compose` subcommand or [`podman-compose`](https://github.com/containers/podman-compose). See [Podman notes](#podman-notes). | | nerdctl / containerd | Experimental | Set `CONTAINER_RUNTIME=nerdctl`. Compose plugin from [nerdctl](https://github.com/containerd/nerdctl) must be present. | Any other binary that implements the `docker` CLI surface can be passed via `CONTAINER_RUNTIME=` or `aether-substrate init --runtime `. ### Podman notes - Rootless Podman does not have a `docker` group, so set `DOCKER_GID` to your user GID (`id -g`). The CLI does this automatically during `init`. - Enable the Podman API socket before starting the stack so Substrate's container manager can talk to it: ```bash systemctl --user enable --now podman.socket export DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock ``` Then pass `DOCKER_HOST` into compose, or add a `volumes:` mapping in `docker-compose.yml` from `/run/user/$UID/podman/podman.sock` → `/var/run/docker.sock`. - `SUBSTRATE_ENABLE_CONTAINERS=true` needs the Podman socket accessible inside the container. If you only want the API and Tap, leave that flag off and none of the socket wiring is needed. ## License Aether Substrate is licensed under the [Apache License 2.0](./LICENSE). You are free to use, modify, and redistribute the software, including for commercial purposes, provided you preserve attribution and the license notice. See [docs/DISCLAIMER.md](./docs/DISCLAIMER.md) for the experimental-status and liability disclaimer that accompanies this license.