docs: document docket worker settings and 2025-12-30 incident (#670)

adds warning about modifying Docket/Worker constructor args after the
heartbeat_interval=30s change broke all task execution. documents:
- which settings not to change
- the incident timeline
- testing requirements if settings must be changed

also filed issue on pydocket: https://github.com/chrisguidry/docket/issues/267

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

authored by zzstoatzz.io Claude Opus 4.5 and committed by GitHub 9ce66955 f90e2e5a

Changed files
+19
backend
src
backend
_internal
docs
+2
backend/src/backend/_internal/background.py
··· 55 55 extra={"docket_name": settings.docket.name, "url": settings.docket.url}, 56 56 ) 57 57 58 + # WARNING: do not modify Docket() or Worker() constructor args without 59 + # reading docs/backend/background-tasks.md - see 2025-12-30 incident 58 60 async with Docket( 59 61 name=settings.docket.name, 60 62 url=settings.docket.url,
+17
docs/backend/background-tasks.md
··· 39 39 DOCKET_WORKER_CONCURRENCY=10 # concurrent task limit 40 40 ``` 41 41 42 + ### ⚠️ worker settings - do not modify 43 + 44 + the worker is initialized in `backend/_internal/background.py` with pydocket's defaults. **do not change these settings without extensive testing:** 45 + 46 + | setting | default | why it matters | 47 + |---------|---------|----------------| 48 + | `heartbeat_interval` | 2s | changing this broke all task execution (2025-12-30 incident) | 49 + | `minimum_check_interval` | 1s | affects how quickly tasks are picked up | 50 + | `scheduling_resolution` | 1s | affects scheduled task precision | 51 + 52 + **2025-12-30 incident**: setting `heartbeat_interval=30s` caused all scheduled tasks (likes, comments, exports) to silently fail while perpetual tasks continued running. root cause unclear - correlation was definitive but mechanism wasn't found in pydocket source. reverted in PR #669. 53 + 54 + if you need to tune worker settings: 55 + 1. test extensively in staging with real task volume 56 + 2. verify ALL task types execute (not just perpetual tasks) 57 + 3. check logfire for task execution spans 58 + 42 59 when `DOCKET_URL` is not set, docket is disabled and tasks fall back to `asyncio.create_task()` (fire-and-forget). 43 60 44 61 ### local development