hotfix: add configurable timeout to queue listener database connection (#280)

CRITICAL FIX for production outage at 2025-11-18 05:04 UTC.

## root cause
asyncpg.connect() in queue listener had no timeout. when Neon database
was slow/unresponsive, the connection attempt would hang indefinitely,
exhausting SQLAlchemy's 5-connection pool and causing all API requests
to timeout with 500 errors.

## analysis
queried logfire for healthy connection latencies:
- typical: 2-4ms
- p50: 25ms
- p95: 352ms
- p99: 549ms

## fix
- add QUEUE_CONNECT_TIMEOUT setting (default: 3.0s)
- use asyncio.wait_for() with configurable timeout on asyncpg.connect()
- timeout provides 5x headroom over p99 while failing fast
- add specific TimeoutError handling with clear logging
- connection failures now fail fast instead of hanging

## configuration
adjust timeout via environment variable if needed:
```
QUEUE_CONNECT_TIMEOUT=5.0 # increase if seeing timeout errors
```

## impact
prevents queue listener from blocking all database connections when
Neon is slow. the listener will retry after reconnect_delay (5s)
instead of hanging indefinitely and breaking the entire app.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <[e-mail]>

src/backend/_internal/queue.py

···
             db_url = db_url.replace("postgresql+psycopg://", "postgresql://")
 
         try:
-            self.conn = await asyncpg.connect(db_url)
+            # add timeout to prevent hanging indefinitely on slow/unresponsive database
+            # default 3s timeout provides 5x headroom over p99 latency (~550ms) while failing fast
+            timeout = settings.database.queue_connect_timeout
+            self.conn = await asyncio.wait_for(asyncpg.connect(db_url), timeout=timeout)
             if not self.conn:
                 raise Exception("failed to connect to database")
             await self.conn.add_listener("queue_changes", self._handle_notification)
             logger.info("queue service connected to database and listening")
+        except TimeoutError:
+            logger.error(
+                f"database connection timed out after {settings.database.queue_connect_timeout}s for queue listening"
+            )
+            raise
         except Exception:
             logger.exception("failed to connect to database for queue listening")
             raise

src/backend/config.py

···
         validation_alias="DATABASE_URL",
         description="PostgreSQL connection string",
     )
+    queue_connect_timeout: float = Field(
+        default=3.0,
+        validation_alias="QUEUE_CONNECT_TIMEOUT",
+        description="Timeout in seconds for queue listener database connections",
+    )
 
 
 class StorageSettings(RelaySettingsSection):