Deploying askimut (Coolify VPS) #

Askimut runs as two Compose services built from one image (Dockerfile):

• web is the SolidStart server (node .output/server/index.mjs), port 3000. On startup it applies pending Drizzle SQL migrations to Turso before listening (node .output/server/migrate.mjs in the entrypoint), then publishes lexicons when authority credentials are configured (node .output/server/publish-lexicons.mjs).
• ingester is a long-lived firehose consumer (node .output/server/ingester.mjs). It consumes Bluesky Jetstream filtered to at.askimut.* and backfills each repo's historical records via com.atproto.repo.listRecords. Jetstream delivers identity/account events for the whole network, so the ingester applies them only for DIDs already in the users table (the web app creates users at OAuth signup).

Both share only the database (Turso, remote libSQL). Coolify's built-in Traefik proxy terminates TLS, so there is no Caddy/nginx in the compose file.

1. Database (Turso) #

1. Create a Turso database and an auth token.

2. Set TURSO_DATABASE_URL and TURSO_AUTH_TOKEN in Coolify. Migrations run automatically on each deploy when the web container starts (the entrypoint runs migrate.mjs before the server). Migration files live in drizzle/ and are committed (including drizzle/meta/).

   Do not add a separate one-shot migrate Compose service on Coolify. An exited migrate container can mark the whole stack unhealthy and return 502 Bad Gateway for your domain even when migration succeeded (coolify#6591).

When you change the schema locally:

1. Run pnpm db:generate to create a new migration file.
2. Test locally: pnpm db:migrate (same runtime migrator as production).
3. Commit the new files under drizzle/ and redeploy. Coolify applies them.

Manual fallback (debugging or pre-first-deploy):

TURSO_DATABASE_URL=libsql://askimut-<org>.turso.io \
TURSO_AUTH_TOKEN=<token> \
pnpm db:migrate

2. Coolify resource #

1. New Resource → Docker Compose → Git repository.
2. Repo URL: the tangled.org remote. Add the Coolify-generated deploy key to the tangled.org repo. Branch: main. Compose file: docker-compose.yml.
3. Coolify builds the image on deploy (same context for both services → built once).
4. Compose file path must match the repo exactly: docker-compose.yaml.

3. Environment variables (Coolify UI) #

Var	Value	Scope	Notes
PUBLIC_URL	https://askimut.at	web	must be https://, which drives the OAuth client_id and Secure cookie. http:// only works on localhost.
TURSO_DATABASE_URL	libsql://askimut-<org>.turso.io	both
TURSO_AUTH_TOKEN	(secret)	both	mark secret
AT_PROTOCOL_ENABLED	true	web
JETSTREAM_URL	wss://jetstream2.us-east.bsky.network/subscribe	ingester	default in compose
HOST / PORT	0.0.0.0 / 3000	web	set in compose
LEXICON_AUTHORITY_IDENTIFIER	Bluesky handle of the _lexicon.askimut.at publisher	web	optional. The entrypoint skips publish if unset; OAuth needs it once DNS is live
LEXICON_AUTHORITY_PASSWORD	app password for that handle	web	mark secret

Do NOT set APP_URL, DEV_PDS_URL, DEV_PLC_URL, JETSTREAM_URL (dev-only; DEV_PDS_URL would enable allowHttp and a wrong handleResolver). There is no OAUTH_PRIVATE_KEY because the OAuth client is public (DPoP).

4. Domain + TLS #

Set the domain on the web service, port 3000. Coolify's Traefik issues TLS. PUBLIC_URL must match the HTTPS origin (same scheme/host, no trailing slash). If Coolify gives you an http://…sslip.io preview URL, enable SSL on the domain first, then set PUBLIC_URL=https://…. OAuth will fail with "URL must use https:" otherwise.

The ingester needs no domain. Optionally enable a Coolify deploy-webhook URL for forge-triggered redeploys.

5. Deploy + verification #

After deploy:

• curl https://askimut.example.com/api/live → 200 {"status":"ok"}
• curl https://askimut.example.com/api/health → 200 {"status":"ok"}
• curl https://askimut.example.com/client-metadata.json → JSON, client_id matches the URL and scope lists granular permissions (not transition:generic)
• OAuth login with a real bsky.social handle succeeds
• Publish a question → ingester logs a Jetstream commit; the DB row flips reindexed
• Visible Unconfirmed badge clears once the Ingester logs the commit and the tab revalidates (~30s)
• Idle confirmed tab: no /api/events connection; DB_READ_DEBUG=true shows zero recurring revalidation DB reads
• Healthy idle Ingester: after connect, logs [jetstream] connected (and [jetstream:posts] connect wantedDids=[…] when AMA discovery is enabled) and then silence until a commit arrives. It persists the cursor every five seconds with no log line.
• Jetstream reconnect: sever transport → exactly one [jetstream] connecting: with cursor rewind
• No PDS fallback: with the Ingester stopped, a visible Unconfirmed record stays Unconfirmed (no server-side PDS read); it clears after reconnect (cursor rewind) or restart (backfill re-arms)
• Process-global 60s known-user/discovery refresh still occurs (expected background reads)
• / redirect works (pre-existing middleware.ts sets Location without a 3xx status, so confirm the behavior in prod)

6. Lexicon publication #

OAuth requests include:at.askimut.appPermissions. All lexicons listed in lexicons.json must exist as com.atproto.lexicon.schema records on the authority DID before login and record writes succeed on the open network.

Authority DNS (one-time): _lexicon.askimut.at TXT → did=<publisher-did> (the DID of the account below).

On each deploy: when LEXICON_AUTHORITY_IDENTIFIER and LEXICON_AUTHORITY_PASSWORD are set on the web service, the entrypoint runs publish-lexicons.mjs after migrations (same code path as pnpm publish:lexicons). putRecord is idempotent, so it is safe on every startup.

If credentials are unset, the entrypoint skips publish with a log line; granular OAuth will fail until DNS + credentials are configured.

When lexicon JSON changes:

1. Commit updated files under lexicons/ and lexicons.json if needed.
2. Redeploy. Publication runs automatically when credentials are set.
3. Allow ~30 minutes for authorization-server permission-set cache refresh after permission-set changes.
4. Ask existing users to sign out and sign in again if writes fail with insufficient scope.

Manual fallback (debugging):

LEXICON_AUTHORITY_IDENTIFIER=<authority-handle> \
LEXICON_AUTHORITY_PASSWORD=<app-password> \
pnpm publish:lexicons

Or inside the image:

docker compose run --rm -e ASKIMUT_SERVICE=publish-lexicons \
  -e LEXICON_AUTHORITY_IDENTIFIER=... -e LEXICON_AUTHORITY_PASSWORD=... web

Local dev network: pnpm dev:network bootstraps a lex-authority.test account, publishes the permission set, and configures the PDS (lexiconDidAuthority). After changing the lexicon JSON, restart dev-network or run pnpm reset:all if resolution fails.

7. Pre-push discipline (no CI) #

There is no CI (repo is on tangled.org, not GitHub). Run locally before pushing:

pnpm test && pnpm typecheck && pnpm lint && pnpm build

Optional .git/hooks/pre-push (make it executable):

#!/bin/sh
pnpm test && pnpm typecheck && pnpm lint && pnpm build

Local end-to-end (optional) #

With Turso credentials in a local .env:

docker compose up --build
curl http://localhost:3000/api/health   # via the mapped port if you add one

docker-compose.yml has no ports: mapping (Coolify's proxy handles routing). Add a temporary ports: ["3000:3000"] to the web service for local testing only.

8. Debugging failed deploys (Coolify) #

Containers may disappear from docker ps -a after a failed deploy (Coolify prunes stopped containers). Use these instead:

Empty Coolify logs + 502 #

If both the web and ingester log tabs are empty after redeploy, containers likely never reached a running state (build failed, or web never passed its healthcheck so ingester never starts). See depends_on: service_healthy in compose.

Check in order:

1. Deployments → latest → build log (persists even when containers are gone)
2. startup.log on the host (see below). The entrypoint logs env validation and migration/publish failures before the app process starts
3. Run docker ps -a --filter name=askimut and look for Restarting, Exited, or web (unhealthy)

The Docker healthcheck hits /api/live (process up only). /api/health still checks Turso for manual verification but is not used for routing health.

Startup log file (persists on the host) #

Both services append to ./logs/startup.log via a bind mount. On the VPS the path is under Coolify's application directory. Create logs/ on first deploy if missing. The entrypoint creates it inside the container, but the host bind mount may need mkdir logs once:

# Find your app UUID folder
sudo ls /data/coolify/applications/

# Create host log dir once if needed, then read after a failed deploy
sudo mkdir -p /data/coolify/applications/<uuid>/logs
sudo tail -100 /data/coolify/applications/<uuid>/logs/startup.log

The entrypoint logs env validation (OK: / ERROR: for each required var) and the process exit code before the container dies.

Coolify UI #

Deployments → latest deploy → build log (persists even when containers are gone). The Logs tab only works while a container is running.

Catch a crash live #

Redeploy, then immediately on SSH:

sudo docker events &
sudo docker ps -a
sudo docker logs <web-container-name> --tail 200

Docker log retention #

Compose sets json-file logging (10 MB × 5 files). Stopped container logs remain under /var/lib/docker/containers/<id>/ until pruned:

sudo find /var/lib/docker/containers -name '*-json.log' -mmin -30 2>/dev/null