# Deploying askimut (Coolify VPS)

Askimut runs as two Compose services built from one image (`Dockerfile`):

- **web** is the SolidStart server (`node .output/server/index.mjs`), port 3000.
  On startup it applies pending Drizzle SQL migrations to Turso before listening
  (`node .output/server/migrate.mjs` in the entrypoint), then publishes lexicons
  when authority credentials are configured (`node .output/server/publish-lexicons.mjs`).
- **ingester** is a long-lived firehose consumer (`node .output/server/ingester.mjs`).
  It consumes Bluesky Jetstream filtered to
  `at.askimut.*` and backfills each repo's historical records via
  `com.atproto.repo.listRecords`. Jetstream delivers identity/account events for
  the whole network, so the ingester applies them only for DIDs already in the
  `users` table (the web app creates users at OAuth signup).

Both share only the database (Turso, remote libSQL). Coolify's built-in Traefik
proxy terminates TLS, so there is no Caddy/nginx in the compose file.

## 1. Database (Turso)

1. Create a Turso database and an auth token.
2. Set `TURSO_DATABASE_URL` and `TURSO_AUTH_TOKEN` in Coolify. Migrations run
   automatically on each deploy when the web container starts (the entrypoint
   runs `migrate.mjs` before the server). Migration files live in `drizzle/`
   and are committed (including `drizzle/meta/`).

   Do **not** add a separate one-shot `migrate` Compose service on Coolify. An
   exited migrate container can mark the whole stack unhealthy and return 502
   Bad Gateway for your domain even when migration succeeded
   ([coolify#6591](https://github.com/coollabsio/coolify/issues/6591)).

When you change the schema locally:

1. Run `pnpm db:generate` to create a new migration file.
2. Test locally: `pnpm db:migrate` (same runtime migrator as production).
3. Commit the new files under `drizzle/` and redeploy. Coolify applies them.

Manual fallback (debugging or pre-first-deploy):

```bash
TURSO_DATABASE_URL=libsql://askimut-<org>.turso.io \
TURSO_AUTH_TOKEN=<token> \
pnpm db:migrate
```

## 2. Coolify resource

1. New Resource → **Docker Compose** → Git repository.
2. Repo URL: the tangled.org remote. Add the Coolify-generated deploy key to
   the tangled.org repo. Branch: `main`. Compose file: `docker-compose.yml`.
3. Coolify builds the image on deploy (same context for both services → built once).
4. Compose file path must match the repo exactly: `docker-compose.yaml`.

## 3. Environment variables (Coolify UI)

| Var                   | Value                                             | Scope    | Notes                                                                                                 |
| --------------------- | ------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------- |
| `PUBLIC_URL`          | `https://askimut.at`                              | web      | **must be `https://`**, which drives the OAuth `client_id` and Secure cookie. `http://` only works on localhost. |
| `TURSO_DATABASE_URL`  | `libsql://askimut-<org>.turso.io`                 | both     |                                                                                                       |
| `TURSO_AUTH_TOKEN`    | (secret)                                          | both     | mark secret                                                                                           |
| `AT_PROTOCOL_ENABLED` | `true`                                            | web      |                                                                                                       |
| `JETSTREAM_URL`       | `wss://jetstream2.us-east.bsky.network/subscribe` | ingester | default in compose                                                                                    |
| `HOST` / `PORT`       | `0.0.0.0` / `3000`                                | web      | set in compose                                                                                        |
| `LEXICON_AUTHORITY_IDENTIFIER` | Bluesky handle of the `_lexicon.askimut.at` publisher | web | optional. The entrypoint skips publish if unset; OAuth needs it once DNS is live |
| `LEXICON_AUTHORITY_PASSWORD`   | app password for that handle                        | web | mark secret                                                                                       |

**Do NOT set** `APP_URL`, `DEV_PDS_URL`, `DEV_PLC_URL`, `JETSTREAM_URL` (dev-only;
`DEV_PDS_URL` would enable `allowHttp` and a wrong handleResolver). There is no
`OAUTH_PRIVATE_KEY` because the OAuth client is public (DPoP).

## 4. Domain + TLS

Set the domain on the web service, port 3000. Coolify's Traefik issues TLS.
**`PUBLIC_URL` must match the HTTPS origin** (same scheme/host, no trailing slash).
If Coolify gives you an `http://…sslip.io` preview URL, enable SSL on the domain
first, then set `PUBLIC_URL=https://…`. OAuth will fail with "URL must use https:"
otherwise.

The ingester needs no domain. Optionally enable a Coolify deploy-webhook URL for
forge-triggered redeploys.

## 5. Deploy + verification

After deploy:

- [ ] `curl https://askimut.example.com/api/live` → `200 {"status":"ok"}`
- [ ] `curl https://askimut.example.com/api/health` → `200 {"status":"ok"}`
- [ ] `curl https://askimut.example.com/client-metadata.json` → JSON, `client_id`
      matches the URL and `scope` lists granular permissions (not
      `transition:generic`)
- [ ] OAuth login with a real bsky.social handle succeeds
- [ ] Publish a question → ingester logs a Jetstream commit; the DB row flips
      `reindexed`
- [ ] Visible Unconfirmed badge clears once the Ingester logs the commit and the tab revalidates (~30s)
- [ ] Idle confirmed tab: no `/api/events` connection; `DB_READ_DEBUG=true` shows zero recurring revalidation DB reads
- [ ] **Healthy idle Ingester:** after connect, logs `[jetstream] connected` (and
      `[jetstream:posts] connect wantedDids=[…]` when AMA discovery is enabled) and
      then silence until a commit arrives. It persists the cursor every five seconds with no log line.
- [ ] **Jetstream reconnect:** sever transport → exactly one `[jetstream] connecting:` with cursor rewind
- [ ] **No PDS fallback:** with the Ingester stopped, a visible Unconfirmed record stays Unconfirmed (no server-side PDS read); it clears after reconnect (cursor rewind) or restart (backfill re-arms)
- [ ] Process-global 60s known-user/discovery refresh still occurs (expected background reads)
- [ ] `/` redirect works (pre-existing `middleware.ts` sets `Location` without a
      3xx status, so confirm the behavior in prod)

## 6. Lexicon publication

OAuth requests `include:at.askimut.appPermissions`. All lexicons listed in
`lexicons.json` must exist as `com.atproto.lexicon.schema` records on the
authority DID before login and record writes succeed on the open network.

**Authority DNS (one-time):** `_lexicon.askimut.at` TXT → `did=<publisher-did>`
(the DID of the account below).

**On each deploy:** when `LEXICON_AUTHORITY_IDENTIFIER` and
`LEXICON_AUTHORITY_PASSWORD` are set on the web service, the entrypoint runs
`publish-lexicons.mjs` after migrations (same code path as `pnpm publish:lexicons`).
`putRecord` is idempotent, so it is safe on every startup.

If credentials are unset, the entrypoint skips publish with a log line; granular OAuth will
fail until DNS + credentials are configured.

**When lexicon JSON changes:**

1. Commit updated files under `lexicons/` and `lexicons.json` if needed.
2. Redeploy. Publication runs automatically when credentials are set.
3. Allow ~30 minutes for authorization-server permission-set cache refresh after
   permission-set changes.
4. Ask existing users to sign out and sign in again if writes fail with
   insufficient scope.

Manual fallback (debugging):

```bash
LEXICON_AUTHORITY_IDENTIFIER=<authority-handle> \
LEXICON_AUTHORITY_PASSWORD=<app-password> \
pnpm publish:lexicons
```

Or inside the image:

```bash
docker compose run --rm -e ASKIMUT_SERVICE=publish-lexicons \
  -e LEXICON_AUTHORITY_IDENTIFIER=... -e LEXICON_AUTHORITY_PASSWORD=... web
```

**Local dev network:** `pnpm dev:network` bootstraps a `lex-authority.test`
account, publishes the permission set, and configures the PDS
(`lexiconDidAuthority`). After changing the lexicon JSON, restart dev-network or
run `pnpm reset:all` if resolution fails.

## 7. Pre-push discipline (no CI)

There is no CI (repo is on tangled.org, not GitHub). Run locally before pushing:

```bash
pnpm test && pnpm typecheck && pnpm lint && pnpm build
```

Optional `.git/hooks/pre-push` (make it executable):

```bash
#!/bin/sh
pnpm test && pnpm typecheck && pnpm lint && pnpm build
```

## Local end-to-end (optional)

With Turso credentials in a local `.env`:

```bash
docker compose up --build
curl http://localhost:3000/api/health   # via the mapped port if you add one
```

`docker-compose.yml` has no `ports:` mapping (Coolify's proxy handles routing).
Add a temporary `ports: ["3000:3000"]` to the web service for local testing
only.

## 8. Debugging failed deploys (Coolify)

Containers may disappear from `docker ps -a` after a failed deploy (Coolify prunes
stopped containers). Use these instead:

### Empty Coolify logs + 502

If both the web and ingester log tabs are empty after redeploy, containers likely
never reached a running state (build failed, or web never passed its healthcheck so
ingester never starts). See `depends_on: service_healthy` in compose.

Check in order:

1. **Deployments → latest → build log** (persists even when containers are gone)
2. **`startup.log`** on the host (see below). The entrypoint logs env validation
   and migration/publish failures before the app process starts
3. Run **`docker ps -a --filter name=askimut`** and look for `Restarting`,
   `Exited`, or web `(unhealthy)`

The Docker healthcheck hits `/api/live` (process up only). `/api/health` still
checks Turso for manual verification but is not used for routing health.

### Startup log file (persists on the host)

Both services append to `./logs/startup.log` via a bind mount. On the VPS the
path is under Coolify's application directory. Create `logs/` on first deploy if
missing. The entrypoint creates it inside the container, but the host bind
mount may need `mkdir logs` once:

```bash
# Find your app UUID folder
sudo ls /data/coolify/applications/

# Create host log dir once if needed, then read after a failed deploy
sudo mkdir -p /data/coolify/applications/<uuid>/logs
sudo tail -100 /data/coolify/applications/<uuid>/logs/startup.log
```

The entrypoint logs env validation (`OK:` / `ERROR:` for each required var) and
the process exit code before the container dies.

### Coolify UI

**Deployments → latest deploy → build log** (persists even when containers are
gone). The **Logs** tab only works while a container is running.

### Catch a crash live

Redeploy, then immediately on SSH:

```bash
sudo docker events &
sudo docker ps -a
sudo docker logs <web-container-name> --tail 200
```

### Docker log retention

Compose sets `json-file` logging (10 MB × 5 files). Stopped container logs remain
under `/var/lib/docker/containers/<id>/` until pruned:

```bash
sudo find /var/lib/docker/containers -name '*-json.log' -mmin -30 2>/dev/null
```