johnandersen777.bsky.social/agent-dev at main

2025-07-12 Engineering Logs#

Spindle bring up

Goals: MCP + OAuth2.1 bsky + Spindle + Tangled.sh + Issue Assignment to AI

doctl compute droplet create \
  --api-url https://alice.chadig.com \
  --region sfo3 \
  --size s-4vcpu-8gb \
  --image ubuntu-24-04-x64 \
  --tag-names \
    "oidc-sub:space-readwrite:spindle-0001,oidc-sub:database:13d94fe3-7e6c-4c24-bd3a-686c337c1a00" \
  --ssh-keys 46961532 \
  spindle-0001
export DROPLET_IPV4=$(doctl compute droplet list | grep spindle-0001 | awk '{print $3}')
ssh root@$DROPLET_IPV4

Enable passwordless sudo
Install docker
- https://docs.docker.com/engine/install/ubuntu/

echo 'export EDITOR=vim' | tee -a /etc/environment
sudo sed -i 's/^%sudo\s\+ALL=(ALL:ALL)\s\+ALL$/%sudo ALL=(ALL:ALL) NOPASSWD: ALL/' /etc/sudoers

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

useradd -s $(which bash) -m -G sudo,docker johnandersen777
su johnandersen777

cd
tmux

mkdir ~/.ssh
sudo cat /root/.ssh/authorized_keys | tee ~/.ssh/authorized_keys
sudo apt-get update && sudo apt-get upgrade -y && sudo apt-get install -y golang
go install golang.org/dl/go1.24.5@latest
~/go/bin/go1.24.5 download
sudo rm /usr/bin/go
sudo mv ~/go/bin/go1.24.5 /usr/bin/go
sudo chown root:root /usr/bin/go
go version
stat /var/run/secrets/digitalocean.com/serviceaccount/token

git clone https://tangled.sh/@tangled.sh/core
cd core/
go mod download
go build -o ./cmd/spindle/spindle ./cmd/spindle/main.go
export SPINDLE_SERVER_HOSTNAME="spindle-0001.tangled.chadig.com"
export SPINDLE_SERVER_OWNER="did:plc:5svqtrhheairglgiiyvutzik"
export SPINDLE_SERVER_LISTEN_ADDR=127.0.0.1:6555
./cmd/spindle/spindle

Setup DNS
Register Spindle
- https://tangled.sh/spindles
  - failed to fetch owner: failed to fetch /owner
Create repo
https://tangled.sh/@tangled.sh/core/blob/master/docs/spindle/pipeline.md

sudo apt-get install -y caddy
export SPINDLE_SERVER_HOSTNAME="spindle-0001.tangled.chadig.com"
export SPINDLE_SERVER_LISTEN_ADDR=127.0.0.1:6555
sudo systemctl enable --now caddy
sudo tee /etc/caddy/Caddyfile <<EOF
${SPINDLE_SERVER_HOSTNAME} {
    reverse_proxy /events/* ${SPINDLE_SERVER_LISTEN_ADDR} {
        header_up Connection "upgrade"
        header_up Upgrade {http.request.header.Upgrade}
    }

    reverse_proxy /logs/* ${SPINDLE_SERVER_LISTEN_ADDR} {
        header_up Connection "upgrade"
        header_up Upgrade {http.request.header.Upgrade}
    }

    reverse_proxy ${SPINDLE_SERVER_LISTEN_ADDR}
}
EOF
sudo systemctl restart caddy

git clone git@tangled.sh:johnandersen777.bsky.social/agent-dev
cd agent-dev
mkdir -p .tangled/workflows
cat > .tangled/workflows/fix-todos.yaml <<'EOF'
when:
  - event: ["push", "pull_request"]
    branch: ["main", "develop"]
  - event: ["manual"]

dependencies:
  ## from nixpkgs
  nixpkgs:
    - python

steps:
  - name: "Install dependencies"
    command: |
      python -m pip install uv

environment:
  TEST_ENV_VAR: "feedface"

## current repository is cloned and checked out at the target ref
## by default.
clone:
  skip: false
  depth: 50
  submodules: true
EOF
git add .
git branch -M main
git commit -sm 'ci: init'
git push -u origin main

https://tangled.sh/@johnandersen777.bsky.social/agent-dev/pipelines
- Waiting for spindle ...
  - Spindle showed no new log lines

johnandersen777@spindle-0001:~/core$ ./cmd/spindle/spindle
time=2025-07-13T01:43:27.663Z level=INFO msg="adding did to in-memory filter" did=did:plc:5svqtrhheairglgiiyvutzik
time=2025-07-13T01:43:27.663Z level=INFO msg="owner set" did=did:plc:5svqtrhheairglgiiyvutzik
time=2025-07-13T01:43:27.664Z level=INFO msg="starting spindle server" address=127.0.0.1:6555
time=2025-07-13T01:43:27.665Z level=INFO msg="done waiting for did"
time=2025-07-13T01:43:27.665Z level=INFO msg="starting knot event consumer"
time=2025-07-13T01:43:27.666Z level=INFO msg="starting consumer" config="{Sources:map[] ProcessFunc:0xb66360 RetryInterval:15m0s MaxRetryInterval:1h0m0s ConnectionTimeout:10s WorkerCount:5 QueueSize:100 Logger:0xc0000341c0 Dev:false CursorStore:0xc000010108}"
time=2025-07-13T01:43:27.667Z level=INFO msg="found last time_us" time_us=1752370997303575
time=2025-07-13T01:43:27.667Z level=INFO msg="connecting to websocket" component=jetstream-client url="wss://jetstream1.us-west.bsky.network/subscribe?cursor=1752370997303575&wantedCollections=sh.tangled.spindle.member&wantedCollections=sh.tangled.repo"
time=2025-07-13T01:43:27.748Z level=INFO msg="starting websocket read loop" component=jetstream-client

For others to view logs: http://spindle-0001.tangled.chadig.com:8080

sudo snap install ttyd --classic
ttyd -p 8080 -- tmux a

Then some websocket read failures

time=2025-07-13T02:09:00.549Z level=ERROR msg="failed to read message from websocket" component=jetstream-client error="websocket: close 1006 (abnormal closure): unexpected EOF"
time=2025-07-13T02:09:00.549Z level=ERROR msg="error reading jetstream" error="read loop failed: failed to read message from websocket: websocket: close 1006 (abnormal closure): unexpected EOF"
time=2025-07-13T02:09:00.549Z level=INFO msg="found last time_us" time_us=1752372507667748
time=2025-07-13T02:09:00.549Z level=INFO msg="connecting to websocket" component=jetstream-client url="wss://jetstream1.us-west.bsky.network/subscribe?cursor=1752372507667748&wantedCollections=sh.tangled.spindle.member&wantedCollections=sh.tangled.repo"
time=2025-07-13T02:09:00.604Z level=ERROR msg="error reading jetstream" error="websocket: bad handshake"
time=2025-07-13T02:09:00.604Z level=INFO msg="found last time_us" time_us=1752372507667748
time=2025-07-13T02:09:00.604Z level=INFO msg="connecting to websocket" component=jetstream-client url="wss://jetstream1.us-west.bsky.network/subscribe?cursor=1752372507667748&wantedCollections=sh.tangled.spindle.member&wantedCollections=sh.tangled.repo"
time=2025-07-13T02:09:00.659Z level=ERROR msg="error reading jetstream" error="websocket: bad handshake"

Set spindle via repo settings
- https://tangled.sh/@johnandersen777.bsky.social/agent-dev/settings
- Added self as member to Spindle just in case
  - https://tangled.sh/spindles/spindle-0001.tangled.chadig.com
Pushed more commits
- Observed 500 error
- Started discord thread per https://bsky.app/profile/oppi.li/post/3ltiascy3ps2o
  - https://discord.com/channels/1361963801993285692/1393783888530243647

time=2025-07-13T02:20:42.171Z level=INFO msg="adding did to in-memory filter" did=did:plc:5svqtrhheairglgiiyvutzik
time=2025-07-13T02:20:42.171Z level=INFO msg="owner set" did=did:plc:5svqtrhheairglgiiyvutzik
time=2025-07-13T02:20:42.172Z level=INFO msg="adding source start" knot=knot1.tangled.sh
time=2025-07-13T02:20:42.172Z level=INFO msg="done waiting for did"
time=2025-07-13T02:20:42.172Z level=INFO msg="starting spindle server" address=127.0.0.1:6555
time=2025-07-13T02:20:42.172Z level=INFO msg="starting knot event consumer"
time=2025-07-13T02:20:42.173Z level=INFO msg="found last time_us" time_us=1752373198588925
time=2025-07-13T02:20:42.173Z level=INFO msg="connecting to websocket" component=jetstream-client url="wss://jetstream1.us-west.bsky.network/subscribe?cursor=1752373198588925&wantedCollections=sh.tangled.spindle.member&wantedCollections=sh.tangled.repo"
time=2025-07-13T02:20:42.173Z level=INFO msg="starting consumer" config="{Sources:map[{Knot:knot1.tangled.sh}:{}] ProcessFunc:0xb66360 RetryInterval:15m0s MaxRetryInterval:1h0m0s ConnectionTimeout:10s WorkerCount:5 QueueSize:100 Logger:0xc00038bcb0 Dev:false CursorStore:0xc00013f080}"
time=2025-07-13T02:20:42.174Z level=INFO msg=connecting url="wss://knot1.tangled.sh/events?cursor=1752372973338564442"
time=2025-07-13T02:20:42.238Z level=INFO msg="starting websocket read loop" component=jetstream-client
time=2025-07-13T02:20:42.558Z level=INFO msg=connected source={Knot:knot1.tangled.sh}
time=2025-07-13T02:21:17.377Z level=INFO msg="pipeline enqueued successfully" id=3ltsstfefqd22
time=2025-07-13T02:21:17.377Z level=INFO msg="starting all workflows in parallel" component=spindle pipeline="{Knot:knot1.tangled.sh Rkey:3ltsstfefqd22}"
time=2025-07-13T02:21:17.378Z level=INFO msg="setting up workflow" component=spindle workflow=knot1.tangled.sh-3ltsstfefqd22-fix-todos.yaml
time=2025-07-13T02:21:20.424Z level=ERROR msg="pipeline failed!" component=spindle workflowId=knot1.tangled.sh-3ltsstfefqd22-fix-todos.yaml error="Error response from daemon: received unexpected HTTP status: 500 Internal Server Error"

Tried rebooting server, same error

Clone this repository

Clone this repository

2025-07-12 Engineering Logs#