Tangled knot on NixOS on Proxmox VE

Self-hosted [Tangled](https://tangled.org) **knot** — a git server on the AT Protocol (the network behind Bluesky) — running on NixOS in an unprivileged Proxmox LXC container. Public identity `did:web:knot.example.com`, TLS certificates via ACME, home IP hidden behind a Cloudflare proxy.

## Architecture


```mermaid
flowchart TB
    Browser["Browser / tangled.org"]
    Relays["atproto relays"]
    Admin["You, on the LAN"]
    CF["Cloudflare<br/>proxied DNS · TLS edge · hides home IP"]
    Out["Let's Encrypt · relays · DNS"]

    subgraph LXC["Proxmox LXC (NixOS)"]
        NGINX["nginx<br/>80 / 443"]
        SSHD["sshd<br/>22"]
        KNOT["knot-server<br/>5555 · 2222<br/>repos · identity · git"]
    end

    Browser ==>|HTTPS + WebSockets| CF
    CF ==>|TCP 80 / 443| NGINX
    NGINX ==>|loopback| KNOT

    Relays -.->|event stream| CF
    Admin -.->|SSH| SSHD
    Admin -.->|git push| KNOT
    KNOT -.->|announcements · certs · DNS lookups| Out
```

| Port | Service | Who can reach it |
|---|---|---|
| `22` | sshd (server admin) | LAN |
| `2222` | knot git-ssh (git pushes) | LAN / tailnet |
| `80`, `443` | nginx (web, TLS) | public (via Cloudflare) |
| `5555` | knot HTTP | loopback only (behind nginx) |
| UDP `443` | QUIC/HTTP-3 | browser ⇄ Cloudflare only; never reaches the server |
| UDP `53` | DNS | outbound |

## Setup

- Get latest NixOS build for the minimal Proxmox LxC by following the[Proxmox LXC wiki page](https://nixos.wiki/wiki/Proxmox_Linux_Container))
    - It's so minimal it won't have ethernet or wifi set so you won't be able to run `nix update-channels` 
    - when you first get in you'll need to run `source /etc/set-environment` to be able to use the shell

- Set CPU to 2 cores, 8 GB RAM and 20 GB disk. Knot is written in Rust and has no pre-built binary cache, so ~1,450 packages compile from source. Both values can be resized live in Proxmox; shrink back to 2 GB afterward.
- A public hostname in a DNS zone you control
- A Cloudflare API token scoped to `Zone.DNS:Edit` for certificate issuance
- An atproto account: `curl -s 'https://public.api.bsky.app/xrpc/com.atproto.identity.resolveHandle?handle=YOUR.handle'`

## Secrets

```bash
mkdir -p /etc/knot && chmod 700 /etc/knot
echo "KNOT_MASTER_KEY=$(head -c 32 /dev/urandom | base64)" > /etc/knot/knot.env
chmod 400 /etc/knot/knot.env
```

- **Losing this key destroys the knot's identity.** On first start the knot generates a signing key and locks it with the master key; the `did:web` document is derived from it. Back up `/etc/knot/knot.env` together with `/var/lib/knot`.
- Never store these files in `/etc/nixos/` — that directory is copied into the world-readable nix store on every build. Keep them root-readable only (`chmod 400`); systemd reads them as root before dropping privileges.

## Configuration

Three files in `/etc/nixos/`. 

### flake.nix

```nix
{
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    tangled.url = "git+https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is";  # Tangled's NixOS module
  };
  outputs = { self, nixpkgs, tangled, ... }: {
    nixosConfigurations.knot = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [ 
      tangled.nixosModules.knot-rs 
      ./configuration.nix 
      ./knot.nix ];
    };
  };
}
```

### configuration.nix — the host (container, network, TLS, proxy)

```nix
{ modulesPath, lib, pkgs, ... }:
{
  imports = [ (modulesPath + "/virtualisation/proxmox-lxc.nix") ];  # container settings
  nix.settings.sandbox = false;                 # required inside an unprivileged LXC
  system.stateVersion = "26.05";                # set once, never change

  networking = {
    hostName = "knot";                          # note the capital N in hostName
    useDHCP = false;
    interfaces.eth0.ipv4.addresses = [{ address = "192.168.1.10"; prefixLength = 24; }];
    defaultGateway = { address = "192.168.1.1"; interface = "eth0"; };  # the { address; interface; } form is required
    nameservers = [ "1.1.1.1" ];
    firewall.allowedTCPPorts = [ 80 443 ];      # replaced by the lockdown section below
  };

  services.openssh = { enable = true; settings.PasswordAuthentication = false; };
  users.users.root.openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAA... your-public-key" ];

  security.acme = {
    acceptTerms = true;
    defaults.email = "admin@example.com";
    certs."example.com" = {
      domain = "example.com";
      extraDomainNames = [ "knot.example.com" ];  # one certificate covers both names
      group = "nginx";
      dnsProvider = "cloudflare";               # DNS-01: proves domain control via DNS, no open port needed
      dnsResolver = "1.1.1.1:53";
      environmentFile = "/var/lib/acme/cloudflare.env";  # contains CLOUDFLARE_DNS_API_TOKEN=…
    };
  };

  services.nginx = {
    enable = true;
    recommendedTlsSettings = true;
    recommendedProxySettings = true;            # adds X-Forwarded-For (knot uses it for rate limiting)
    virtualHosts."knot.example.com" = {
      forceSSL = true;
      useACMEHost = "example.com";
      locations."/" = {
        proxyPass = "http://127.0.0.1:5555";
        proxyWebsockets = true;                 # required: the event stream is a WebSocket
      };
      locations."/events" = {
        proxyPass = "http://127.0.0.1:5555";
        proxyWebsockets = true;
        extraConfig = "proxy_read_timeout 86400s;";
      };
    };
  };
}
```

### knot.nix 

```nix
{ config, pkgs, ... }:
{
  services.tangled.knot-rs = {
    enable = true;
    environmentFile = "/etc/knot/knot.env";
    settings = {
      server = {
        hostname = "knot.example.com";          # becomes the did:web identity; changing it changes identity
        admins = [ "did:plc:YOUR_ADMIN_DID" ];  # first entry is the advertised owner
      };
      atproto.plc_directory = "https://plc.directory";   # required
      contact.email = "admin@example.com";               # required
      lfs.store_path = "/var/lib/knot/lfs";              # required
      xrpc = {
        trusted_proxy_header = "x-forwarded-for";
        trusted_proxies = [ "127.0.0.1" "::1" ];
      };
    };
  };
}
```

## Create a repo

:warn: Everything on a knot is publicly browsable. There is no private repo option.

1. **Create** at `tangled.org/repo/new` — Owner: your identity, Host: your knot, and tick *Use permalink*.
2. **Register an SSH key** in your tangled.org settings, then verify access: `ssh -T git@<CT-IP> -p 2222` prints `Welcome to this knot!`. That banner is normal output, not an error.
3. **Push** using the *permalink* URL. The knot gives each repository its own DID (returned by the create call as `"did"`):

   ```bash
   git remote add knot "ssh://git@192.168.1.10:2222/did:plc:REPO_DID"
   git push knot main
   ```

   - Never add a `.git` suffix to this URL — it will 404.
   - Both ends must use the same git object format. Knot-created repositories default to **SHA-256**; if your local repository is SHA-1 you get `fatal: the receiving end does not support this repository's hash algorithm`. Create local repositories with `git init --object-format=sha256`, or set `git config --global init.defaultObjectFormat sha256`.
   - The `git` in `git@…` is a protocol convention (like GitHub's), not a user account on the server.
4. **A new repository does not appear on tangled.org until it receives its first push.** Relays only carry live events; content pushed before they start listening is never indexed. If a repository page stays 404, push something and wait a few minutes.
5. **Your handle must resolve to your DID**, or logins fail with `AuthMissing`. Either add a DNS TXT record `_atproto.example.com = did=did:plc:YOUR_ADMIN_DID`, or serve the DID at `https://example.com/.well-known/atproto-did`.


## Gotchas

| Symptom                                                               | Cause                                                                                               | Fix                                                                                       |
| --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Assertions about root filesystem / GRUB                               | NixOS doesn't know it's a container                                                                 | `boot.isContainer = true;` (the proxmox-lxc profile does this)                            |
| `…option was accessed but has no value defined`                       | Required module options, revealed one per build                                                     | `grep -n -B1 -A8 mkOption …/nix/modules/knot-rs.nix`                                      |
| Empty `PATH` after `pct enter`                                        | That command skips the login step, so the shell's environment is never set up                       | Use SSH, or `pct exec <vmid> -- bash -l`                                                  |
| eth0 shows `unmanaged`, no IP                                         | The kernel creates network interfaces powered off and unconfigured; a policy daemon must claim them | Declare networking in NixOS; until the first build, configure manually with `ip addr add` |
| A config file's contents seem ignored                                 | It's not in the flake's `modules` list                                                              | Import it; also check options aren't defined twice with different values                  |
| SSH lockout; knot git-ssh dead                                        | System sshd was moved to 2222 and restricted to a `git` user                                        | Keep sshd on 22 for admin; knot owns 2222                                                 |
| `defaultGateway.interface is not optional`                            | systemd-networkd requires routes to name an interface                                               | `defaultGateway = { address = …; interface = "eth0"; };`                                  |
| `the receiving end does not support this repository's hash algorithm` | SHA-1 local repository vs SHA-256 knot repository                                                   | `git init --object-format=sha256`                                                         |
| Repo created but tangled.org shows 404                                | No live events have been emitted yet                                                                | Push content                                                                              |
| Cloudflare **Error 1016**                                             | A leftover CNAME record on the name                                                                 | Delete it; create the A record                                                            |
| SSH `Permission denied (publickey)`                                   | A key generated *on* the server was put in `authorizedKeys`                                         | It must hold the **client's** public key (on the client: `ssh-add -L`)                    |


## Maintenance

- **Updates**: run `nix flake update && nixos-rebuild switch` on a schedule. (`system.autoUpgrade` alone never updates the pinned inputs.) Note that updating nixpkgs triggers a full Rust rebuild.
- **Backups**: `/var/lib/knot` and `/etc/knot/knot.env` together — both are required to keep the identity.
- **If a build runs out of memory** (swap full, console unresponsive), fix it from the Proxmox host: `pct set <vmid> --memory 8192 --swap 4096` and `pct resize <vmid> rootfs +10G` (both live, no reboot). To abort a stuck build: `pct exec <vmid> -- systemctl restart nix-daemon`.
- **Debugging**: `nginx -t -c <config path>` validates the nginx config; `nginx -T -c <config path>` prints the full generated config; `journalctl -u knot-rs -f` tails the knot; `curl -s "https://…/xrpc/org.tangled.knot.describeKnot"` shows what the knot reports about itself.i