docs: audit and update documentation (#547)

* docs: audit and update documentation

- promote connection-pool-exhaustion runbook from sandbox to docs/runbooks/
- update CLAUDE.md files with current module descriptions
- _internal: add background tasks, moderation, jobs
- api: add albums, playlists, exports, moderation, stats
- docs: add runbooks, testing, moderation, lexicons
- refresh docs/README.md with streamlined navigation
- update STATUS.md:
- add Dec 9 docket/concurrent exports section
- update immediate priorities (moderation cleanup, cost dashboard)
- de-emphasize transcoder (moved to backlog)
- add docket to technical architecture

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[e-mail]>

* docs: update README with current state

- add docket, logfire to tech stack
- add moderation/transcoder services section
- update local dev to include dev-services (redis)
- add useful commands section with actual justfile targets
- update features (playlists, scrobbling, timed comments, support links)
- add data ownership section
- update project structure (_internal services)
- consolidate links + documentation into single section
- add mirrors section (github, tangled)
- remove stale atproto fork reference

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[e-mail]>

---------

Co-authored-by: Claude <[e-mail]>

README.md

···
 - **framework**: [FastAPI](https://fastapi.tiangolo.com)
 - **database**: [Neon PostgreSQL](https://neon.com)
 - **storage**: [Cloudflare R2](https://developers.cloudflare.com/r2/)
+- **background tasks**: [docket](https://github.com/zzstoatzz/docket) (Redis-backed)
 - **hosting**: [Fly.io](https://fly.io)
-- **auth**: [atproto OAuth 2.1](https://atproto.com/specs/oauth) ([fork with OAuth implementation](https://github.com/zzstoatzz/atproto))
+- **observability**: [Pydantic Logfire](https://logfire.pydantic.dev)
+- **auth**: [atproto OAuth 2.1](https://atproto.com/specs/oauth)
 
 ### frontend
-- **framework**: [SvelteKit](https://kit.svelte.dev)
+- **framework**: [SvelteKit](https://kit.svelte.dev) with Svelte 5 runes
 - **runtime**: [Bun](https://bun.sh)
 - **hosting**: [Cloudflare Pages](https://pages.cloudflare.com)
 - **styling**: vanilla CSS (lowercase aesthetic)
 
+### services
+- **moderation**: Rust ATProto labeler for copyright/sensitive content
+- **transcoder**: Rust audio conversion service (ffmpeg)
+
 </details>
 
 <details>
···
 
 ### prerequisites
 
-- [uv](https://docs.astral.sh/uv/) for Python package management
-- [bun](https://bun.sh/) for frontend development
-- [just](https://github.com/casey/just) for task running (recommended)
+- [uv](https://docs.astral.sh/uv/) for Python
+- [bun](https://bun.sh/) for frontend
+- [just](https://github.com/casey/just) for task running
+- [docker](https://www.docker.com/) for dev services (redis)
 
 ### quick start
 
-using [just](https://github.com/casey/just):
+```bash
+# install dependencies
+uv sync
+cd frontend && bun install && cd ..
 
-```bash
-# install dependencies (uv handles backend venv automatically)
-uv sync # For root-level deps, if any, and initializes uv
-just frontend install
+# start dev services (redis for background tasks)
+just dev-services
 
 # run backend (hot reloads at http://localhost:8001)
 just backend run
 
 # run frontend (hot reloads at http://localhost:5173)
 just frontend dev
+```
+
+### useful commands
 
-# run transcoder (hot reloads at http://localhost:8082)
-just transcoder run
+```bash
+# run tests
+just backend test
+
+# run linting
+just backend lint
+just frontend check
+
+# database migrations
+just backend migrate "migration message"
+just backend migrate-up
+
+# stop dev services
+just dev-services-down
 ```
 
 </details>
···
 <summary>features</summary>
 
 ### listening
-- audio playback with persistent queue across tabs/devices
-- like tracks with counts visible to all listeners
+- audio playback with persistent queue across tabs
+- like tracks, add to playlists
 - browse artist profiles and discographies
-- share tracks and albums with nice link previews
+- share tracks, albums, and playlists with link previews
+- unified search with Cmd/Ctrl+K
+- teal.fm scrobbling
 
 ### creating
-- well-scoped OAuth authentication via ATProto (bluesky accounts)
-- upload tracks with title, artwork, and featured artists
-- organize tracks into albums with cover art
-- edit metadata and replace artwork anytime
-- track play counts and like analytics
-- publish ATProto track and like records to your PDS
+- OAuth authentication via ATProto (bluesky accounts)
+- upload tracks with title, artwork, tags, and featured artists
+- organize tracks into albums and playlists
+- drag-and-drop reordering
+- timed comments with clickable timestamps
+- artist support links (ko-fi, patreon, etc.)
+
+### data ownership
+- tracks, likes, playlists synced to your PDS as ATProto records
+- portable identity - your data travels with you
+- public by default - any client can read your music records
 
 </details>
-
 
 <details>
 <summary>project structure</summary>
 
 ```
 plyr.fm/
-├── backend/              # FastAPI app & Python tooling
+├── backend/              # FastAPI app
 │   ├── src/backend/      # application code
 │   │   ├── api/          # public endpoints
-│   │   ├── _internal/    # internal services
+│   │   ├── _internal/    # services (auth, atproto, background tasks)
 │   │   ├── models/       # database schemas
-│   │   └── storage/      # storage adapters
+│   │   └── storage/      # R2 adapter
 │   ├── tests/            # pytest suite
-│   └── alembic/          # database migrations
+│   └── alembic/          # migrations
 ├── frontend/             # SvelteKit app
 │   ├── src/lib/          # components & state
 │   └── src/routes/       # pages
+├── moderation/           # Rust labeler service
 ├── transcoder/           # Rust audio service
 ├── docs/                 # documentation
 └── justfile              # task runner
···
 <summary>costs</summary>
 
 ~$35-40/month:
-- fly.io backend (production): ~$5/month (shared-cpu-1x, 256MB RAM)
-- fly.io backend (staging): ~$5/month (shared-cpu-1x, 256MB RAM)
-- fly.io transcoder: ~$0-5/month (auto-scales to zero when idle)
-- neon postgres: $5/month (starter plan)
-- audd audio fingerprinting: ~$10/month (enterprise API for copyright detection)
-- cloudflare pages: free (frontend hosting)
-- cloudflare r2: ~$0.16/month (6 buckets across dev/staging/prod)
+- fly.io backend (prod + staging): ~$10/month
+- fly.io transcoder: ~$0-5/month (auto-scales to zero)
+- neon postgres: $5/month
+- audd audio fingerprinting: ~$10/month
+- cloudflare (pages + r2): ~$0.16/month
 
 </details>
 
 ## links
 
 - **production**: https://plyr.fm
+- **staging**: https://stg.plyr.fm
 - **API docs**: https://api.plyr.fm/docs
 - **python SDK / MCP server**: [plyrfm](https://github.com/zzstoatzz/plyr-python-client) ([PyPI](https://pypi.org/project/plyrfm/))
-- **repository**: https://github.com/zzstoatzz/plyr.fm
+- **documentation**: [docs/README.md](docs/README.md)
+- **status**: [STATUS.md](STATUS.md)
 
-## documentation
-
-- [deployment guide](docs/deployment/environments.md)
-- [configuration](docs/backend/configuration.md)
-- [full documentation](docs/README.md)
+### mirrors
+- **github**: https://github.com/zzstoatzz/plyr.fm
+- **tangled**: https://tangled.sh/@zzstoatzz.io/plyr.fm

STATUS.md

···
 
 ### December 2025
 
+#### docket background tasks & concurrent exports (PRs #534-546, Dec 9)
+
+**docket integration** (PRs #534, #536, #539):
+- migrated background tasks from inline asyncio to docket (Redis-backed task queue)
+- copyright scanning, media export, ATProto sync, and teal scrobbling now run via docket
+- graceful fallback to asyncio for local development without Redis
+- parallel test execution with xdist template databases (#540)
+
+**concurrent export downloads** (PR #545):
+- exports now download tracks in parallel (up to 4 concurrent) instead of sequentially
+- significantly faster for users with many tracks or large files
+- zip creation remains sequential (zipfile constraint)
+
+**ATProto refactor** (PR #534):
+- reorganized ATProto record code into `_internal/atproto/records/` by lexicon namespace
+- extracted `client.py` for low-level PDS operations
+- cleaner separation between plyr.fm and teal.fm lexicons
+
+**documentation & observability**:
+- AudD API cost tracking dashboard (#546)
+- promoted runbooks from sandbox to `docs/runbooks/`
+- updated CLAUDE.md files across the codebase
+
+---
+
 #### artist support links & inline playlist editing (PRs #520-532, Dec 8)
 
 **artist support link** (PR #532):
···
 
 ## immediate priorities
 
-### high priority features
-1. **audio transcoding pipeline integration** (issue #153)
-   - ✅ standalone transcoder service deployed at https://plyr-transcoder.fly.dev/
-   - ⏳ next: integrate into plyr.fm upload pipeline
-
 ### known issues
 - playback auto-start on refresh (#225)
-- no AIFF/AIF transcoding support (#153)
 - iOS PWA audio may hang on first play after backgrounding
 
-### new features
-- issue #146: content-addressable storage (hash-based deduplication)
-- issue #155: add track metadata (genres, tags, descriptions)
+### immediate focus
+- **moderation cleanup**: consolidate copyright detection, reduce AudD API costs, streamline labeler integration (issues #541-544)
+- **public cost dashboard**: interactive page showing platform running costs (fly.io, neon, audd, r2) - transparency + prompts for community support
+
+### feature ideas
 - issue #334: add 'share to bluesky' option for tracks
 - issue #373: lyrics field and Genius-style annotations
-- issue #393: moderation - represent confirmed takedown state in labeler
+
+### backlog
+- audio transcoding pipeline integration (#153) - transcoder service deployed, integration deferred
 
 ## technical state
 
···
 - framework: FastAPI with uvicorn
 - database: Neon PostgreSQL (serverless)
 - storage: Cloudflare R2 (S3-compatible)
+- background tasks: docket (Redis-backed)
 - hosting: Fly.io (2x shared-cpu VMs)
 - observability: Pydantic Logfire
 - auth: ATProto OAuth 2.1
···
 - ✅ unified search with Cmd/Ctrl+K
 - ✅ teal.fm scrobbling
 - ✅ copyright moderation with ATProto labeler
+- ✅ docket background tasks (copyright scan, export, atproto sync, scrobble)
+- ✅ media export with concurrent downloads
 
 **albums**
 - ✅ album CRUD with cover art
···
 
 ## documentation
 
-- [deployment overview](docs/deployment/overview.md)
-- [configuration guide](docs/configuration.md)
-- [queue design](docs/queue-design.md)
-- [logfire querying](docs/logfire-querying.md)
-- [moderation & labeler](docs/moderation/atproto-labeler.md)
-- [unified search](docs/frontend/search.md)
-- [keyboard shortcuts](docs/frontend/keyboard-shortcuts.md)
-- [lexicons overview](docs/lexicons/overview.md)
+- [docs/README.md](docs/README.md) - documentation index
+- [runbooks](docs/runbooks/) - production incident procedures
+- [background tasks](docs/backend/background-tasks.md) - docket task system
+- [logfire querying](docs/tools/logfire.md) - observability queries
+- [moderation & labeler](docs/moderation/atproto-labeler.md) - copyright, sensitive content
+- [lexicons overview](docs/lexicons/overview.md) - ATProto record schemas
 
 ---
 
-this is a living document. last updated 2025-12-08.
+this is a living document. last updated 2025-12-09.

backend/src/backend/_internal/CLAUDE.md

···
   - `records/fm_plyr/`: plyr.fm lexicons (track, like, comment, list, profile)
   - `records/fm_teal/`: teal.fm lexicons (play, status)
   - `sync.py`: high-level sync orchestration (profile, albums, liked list)
+- **background**: docket-based background task system
+  - `background.py`: docket client initialization and configuration
+  - `background_tasks.py`: task functions (copyright scan, export, atproto sync, teal scrobble)
 - **queue**: fisher-yates shuffle with retry, postgres LISTEN/NOTIFY for cache invalidation
 - **uploads**: streaming chunked uploads to R2/filesystem, duplicate detection via file_id
+- **moderation**: copyright scanning via AudD, sensitive image flagging
+- **jobs**: job tracking for long-running operations (exports)
 
 gotchas:
 - ATProto records organized under `_internal/atproto/records/` by lexicon namespace
 - file_id is sha256 hash truncated to 16 chars
-- queue cache is TTL-based (5min), hydration includes duplicate track_ids
+- queue cache is TTL-based (5min), hydration includes duplicate track_ids
+- background tasks use docket (Redis-backed) with asyncio fallback for local dev

backend/src/backend/api/CLAUDE.md

···
 - session management in `_internal/auth.py`
 
 resources:
-- **tracks**: upload, edit, delete, like/unlike, play count tracking
-- **artists**: profiles synced from ATProto identities
+- **tracks**: upload, edit, delete, like/unlike, play count tracking, timed comments
+- **albums**: CRUD with cover art, track ordering, ATProto list records
+- **playlists**: CRUD with drag-and-drop reordering, ATProto list records
+- **artists**: profiles synced from ATProto identities, support links
 - **audio**: streaming via 307 redirects to R2 CDN
 - **queue**: server-authoritative with optimistic client updates
-- **preferences**: user settings (accent color, auto-play)
+- **preferences**: user settings (accent color, auto-play, teal scrobbling, sensitive artwork)
+- **exports**: media export with SSE progress tracking, concurrent downloads
+- **moderation**: sensitive image management, copyright label checking
+- **stats**: platform statistics (track count, play count, total duration)

docs/CLAUDE.md

···
 organized knowledge base - check here before researching.
 
 structure:
-- **frontend/** - svelte 5 state patterns, ui components
-- **backend/** - config system, features, transcoder service
+- **frontend/** - svelte 5 state patterns, ui components, navigation
+- **backend/** - config system, background tasks, database, transcoder
 - **deployment/** - environments, migrations, fly.io
-- **tools/** - logfire queries, neon mcp, pdsx cli
+- **tools/** - logfire queries, neon mcp, pdsx cli, plyr.fm mcp
 - **local-development/** - setup guide for new contributors
+- **moderation/** - copyright detection, sensitive content, ATProto labeler
+- **lexicons/** - ATProto record schemas (track, like, comment, list, profile)
+- **runbooks/** - operational procedures for production incidents
+- **testing/** - pytest patterns, parallel execution, template databases
 
 when you solve a problem or make a design choice, document it here with as much detail as needed

docs/README.md

···
 # plyr.fm documentation
 
-this directory contains all documentation for the plyr.fm project.
+organized knowledge base for plyr.fm development.
 
-## documentation index
+## quick navigation
 
-### authentication & security
-- **[authentication.md](./authentication.md)** - secure cookie-based authentication, HttpOnly cookies, XSS protection, environment architecture, migration from localStorage
-
-### frontend
-- **[state-management.md](./frontend/state-management.md)** - global state management with Svelte 5 runes (toast notifications, tracks cache, upload manager, queue management, liked tracks, preferences, localStorage persistence)
-- **[toast-notifications.md](./frontend/toast-notifications.md)** - user feedback system for async operations with smooth transitions and auto-dismiss
-- **[queue.md](./frontend/queue.md)** - music queue management with server sync
-- **[keyboard-shortcuts.md](./frontend/keyboard-shortcuts.md)** - global keyboard shortcuts with context-aware filtering (Q for queue toggle, patterns for adding new shortcuts)
+### operations
+- **[runbooks/](./runbooks/)** - production incident procedures
+  - [connection-pool-exhaustion](./runbooks/connection-pool-exhaustion.md) - 500s, stuck connections
 
 ### backend
-- **[configuration.md](./backend/configuration.md)** - backend configuration and environment setup
-- **[liked-tracks.md](./backend/liked-tracks.md)** - ATProto-backed track likes with error handling and consistency guarantees
-- **[streaming-uploads.md](./backend/streaming-uploads.md)** - SSE-based progress tracking for file uploads with fire-and-forget pattern
-- **[transcoder.md](./backend/transcoder.md)** - rust-based HTTP service for audio format conversion (ffmpeg integration, authentication, fly.io deployment)
+- **[background-tasks.md](./backend/background-tasks.md)** - docket-based task system (copyright scan, export, scrobble)
+- **[configuration.md](./backend/configuration.md)** - environment setup and settings
+- **[database/](./backend/database/)** - connection pooling, neon-specific patterns
+- **[streaming-uploads.md](./backend/streaming-uploads.md)** - SSE progress tracking
+- **[transcoder.md](./backend/transcoder.md)** - rust audio conversion service
+
+### frontend
+- **[state-management.md](./frontend/state-management.md)** - svelte 5 runes patterns
+- **[keyboard-shortcuts.md](./frontend/keyboard-shortcuts.md)** - global shortcuts
+- **[navigation.md](./frontend/navigation.md)** - SvelteKit routing patterns
+- **[search.md](./frontend/search.md)** - unified search with Cmd+K
 
 ### deployment
-- **[environments.md](./deployment/environments.md)** - staging vs production environments, automated deployment via GitHub Actions, CORS, secrets management
-- **[database-migrations.md](./deployment/database-migrations.md)** - automated migration workflow via fly.io release commands, alembic usage, safety procedures
+- **[environments.md](./deployment/environments.md)** - staging vs production
+- **[database-migrations.md](./deployment/database-migrations.md)** - alembic workflow
 
 ### tools
-- **[logfire.md](./tools/logfire.md)** - SQL query patterns for Logfire DataFusion database, finding exceptions, analyzing performance bottlenecks
-- **[neon.md](./tools/neon.md)** - Neon Postgres database management and best practices
-- **[pdsx.md](./tools/pdsx.md)** - ATProto PDS explorer and debugging tools
+- **[logfire.md](./tools/logfire.md)** - SQL query patterns for observability
+- **[neon.md](./tools/neon.md)** - postgres database management
+- **[pdsx.md](./tools/pdsx.md)** - ATProto PDS explorer
+
+### atproto
+- **[lexicons/](./lexicons/)** - record schemas (track, like, comment, list, profile)
+- **[authentication.md](./authentication.md)** - OAuth 2.1 flow
+
+### moderation
+- **[moderation/](./moderation/)** - copyright detection, sensitive content, labeler
+
+### testing
+- **[testing/](./testing/)** - pytest patterns, parallel execution
 
 ### local development
-- **[setup.md](./local-development/setup.md)** - complete local development setup guide
+- **[local-development/setup.md](./local-development/setup.md)** - getting started
 
-## ATProto integration
+## architecture overview
 
 plyr.fm uses a hybrid storage model:
-- audio files stored in cloudflare R2 (scalable, CDN-backed)
-- metadata stored as ATProto records on user's PDS (decentralized, user-owned)
-- local database indexes for fast queries
+- **audio files**: cloudflare R2 (CDN-backed, zero egress)
+- **metadata**: ATProto records on user's PDS (decentralized, user-owned)
+- **indexes**: neon postgres for fast queries
 
 key namespaces:
-- `fm.plyr.track` - track metadata (title, artist, album, features, image, audio file reference)
-- `fm.plyr.like` - user likes on tracks (subject references track URI)
+- `fm.plyr.track` - track metadata
+- `fm.plyr.like` - user likes
+- `fm.plyr.comment` - timed comments
+- `fm.plyr.list` - playlists and albums
+- `fm.plyr.actor.profile` - artist profiles
 
 ## quick start
 
-### current state
-
-plyr.fm is fully functional with:
-- ✅ OAuth 2.1 authentication (ATProto)
-- ✅ secure cookie-based sessions (HttpOnly, XSS protection)
-- ✅ R2 storage for audio files (cloudflare CDN)
-- ✅ track upload with streaming (prevents OOM)
-- ✅ ATProto record creation (fm.plyr.track namespace)
-- ✅ music player with queue management
-- ✅ liked tracks (fm.plyr.like namespace)
-- ✅ artist pages and track discovery
-- ✅ share buttons across track, album, and artist detail pages for quick copy-to-clipboard links
-- ✅ modular audio player with dedicated subcomponents for metadata, transport, progress, and volume controls
-- ✅ image uploads for track artwork
-- ✅ audio transcoding service (rust + ffmpeg)
-- ✅ server-sent events for upload progress
-- ✅ toast notifications
-- ✅ user preferences (accent color, auto-play)
-- ✅ keyboard shortcuts (Q for queue toggle)
-
-### local development
-
-see **[local-development/setup.md](./local-development/setup.md)** for complete setup instructions.
-
-quick start:
 ```bash
 # backend
-uv run uvicorn backend.main:app --reload --host 0.0.0.0 --port 8001
+just backend run
 
 # frontend
-cd frontend && bun run dev
+just frontend dev
 
-# transcoder (optional)
-cd transcoder && just run
+# run tests
+just backend test
 ```
 
-### deployment
-
-see **[deployment/environments.md](./deployment/environments.md)** for details on:
-- staging vs production environments
-- automated deployment via GitHub Actions
-- environment variables and secrets
-
-see **[deployment/database-migrations.md](./deployment/database-migrations.md)** for:
-- migration workflow and safety procedures
-- alembic usage and testing
-
-## architecture decisions
-
-### why R2 instead of PDS blobs?
-
-PDS blobs are designed for smaller files like images. audio files are:
-- larger (5-50MB per track)
-- require streaming
-- benefit from CDN distribution
-
-R2 provides:
-- scalable storage
-- free egress to cloudflare CDN
-- simple HTTP URLs
-- cost-effective (~$0.015/GB/month)
-
-### why fm.plyr namespace?
-
-plyr.fm uses `fm.plyr.*` as the ATProto namespace:
-- `fm.plyr.track` for track metadata
-- `fm.plyr.like` for user likes
-
-this is a domain-specific lexicon that allows:
-- clear ownership and governance
-- faster iteration without formal approval
-- alignment with the plyr.fm brand
-
-### why hybrid storage?
-
-storing metadata on ATProto provides:
-- user data sovereignty (users own their catalog)
-- decentralization (no single point of failure)
-- portability (users can move to another client)
-
-storing audio on R2 provides:
-- performance (fast streaming via CDN)
-- scalability (handles growth)
-- cost efficiency (cheaper than PDS blobs)
-
-### why separate transcoder service?
-
-the transcoder runs as a separate rust service because:
-- ffmpeg operations are CPU-intensive and can block event loop
-- rust provides better performance for media processing
-- isolation prevents transcoding from affecting API latency
-- can scale independently from main backend
-
-## testing
-
-plyr.fm uses pytest for backend testing:
-
-```bash
-# run all tests
-just test
-
-# run specific test file
-just test tests/api/test_track_likes.py
-
-# run with verbose output
-just test -v
-```
-
-test categories:
-- API endpoints (`tests/api/`)
-- storage backends (`tests/storage/`)
-- ATProto integration (`tests/test_atproto.py`)
-- audio format validation (`tests/test_audio_formats.py`)
-
-see [`tests/CLAUDE.md`](../tests/CLAUDE.md) for testing guidelines.
-
-## troubleshooting
-
-### R2 upload fails
-
-```
-error: failed to upload to R2
-```
-
-**check**:
-- R2 credentials in `.env`
-- bucket exists and is accessible
-- account ID is correct
-
-### ATProto record creation fails
-
-```
-error: failed to create atproto record
-```
-
-**check**:
-- OAuth session is valid (not expired)
-- user has write permissions
-- PDS is accessible
-- record format is valid
-
-### audio won't play
-
-```
-404: audio file not found
-```
-
-**check**:
-- `STORAGE_BACKEND` matches actual storage
-- R2 bucket has public read access
-- file_id matches database record
-
-## monitoring
-
-### key metrics to track
-
-1. **upload success rate**
-   - total uploads attempted
-   - successful R2 uploads
-   - successful record creations
-
-2. **storage costs**
-   - total R2 storage (GB)
-   - monthly operations count
-   - estimated cost
-
-3. **playback metrics**
-   - tracks played
-   - average stream duration
-   - errors/failures
-
-### logging
-
-add structured logging for debugging:
-
-```python
-import structlog
-
-logger = structlog.get_logger()
-
-logger.info(
-    "track_uploaded",
-    track_id=track.id,
-    r2_url=r2_url,
-    atproto_uri=atproto_uri,
-)
-```
-
-## security considerations
-
-### audio file access
-
-**current**: R2 URLs are public (anyone with URL can access)
-
-**acceptable for MVP** because:
-- music is meant to be shared
-- no sensitive content
-- URL guessing is impractical (content-based hashes)
-
-**future enhancement**: signed URLs with expiration
-
-### record ownership
-
-**enforced by ATProto**: only user with valid OAuth session can create records in their repo
-
-**enforced by backend**: tracks are associated with `artist_did` and only owner can delete
-
-### rate limiting
-
-**recommended**: limit uploads to prevent abuse
-- 10 uploads per hour per user
-- 100MB total per hour per user
-
-## cost estimates
-
-current monthly costs (~$15-20/month):
-- fly.io backend: $5-10/month (shared-cpu-1x, 256MB RAM)
-- fly.io transcoder: $5-10/month (shared-cpu-1x, 256MB RAM)
-- neon postgres: free tier (0.5GB storage, 3GB data transfer)
-- cloudflare R2: ~$0.16/month (6 buckets: audio-dev, audio-stg, audio-prod, images-dev, images-stg, images-prod)
-- cloudflare pages: free (frontend hosting)
-
-R2 storage scaling (audio + images):
-- 1,000 tracks: ~$0.16/month
-- 10,000 tracks: ~$1.58/month
-- 100,000 tracks: ~$15.81/month
-
-## references
-
-### ATProto documentation
-
-- [repository spec](https://atproto.com/specs/repository)
-- [lexicon spec](https://atproto.com/specs/lexicon)
-- [data model](https://atproto.com/specs/data-model)
-- [OAuth 2.1](https://atproto.com/specs/oauth)
-
-### cloudflare documentation
-
-- [R2 overview](https://developers.cloudflare.com/r2/)
-- [R2 pricing](https://developers.cloudflare.com/r2/pricing/)
-- [S3 compatibility](https://developers.cloudflare.com/r2/api/s3/)
-
-### plyr.fm project files
-
-- project instructions: `CLAUDE.md`
-- main readme: `README.md`
-- justfile: `justfile` (task runner)
-- backend: `src/backend/`
-- frontend: `frontend/`
-- transcoder: `transcoder/`
+see [local-development/setup.md](./local-development/setup.md) for complete setup.
 
 ## contributing
 
-when working on plyr.fm:
-
-1. **test empirically first** - run code and prove it works
-2. **reference existing docs** - check docs directory before researching
-3. **keep it simple** - MVP over perfection
-4. **use lowercase** - respect plyr.fm's aesthetic
-5. **no sprawl** - avoid creating multiple versions of files
-6. **document decisions** - update docs as you work
-
-## questions?
-
-if anything is unclear:
-- check the relevant phase document
-- review example projects in sandbox
-- consult ATProto official docs
-- look at your atproto fork implementation
+1. check docs before researching externally
+2. document decisions as you make them
+3. keep it simple - MVP over perfection
+4. use lowercase aesthetic

docs/runbooks/README.md

···
+# runbooks
+
+operational procedures for production incidents.
+
+## available runbooks
+
+- [connection-pool-exhaustion](connection-pool-exhaustion.md) - 500s everywhere, queue listener down, stuck connections
+
+## when to use
+
+runbooks are for known failure modes with established remediation steps. if you encounter a new type of incident:
+
+1. stabilize first (restart machines if needed)
+2. investigate using [logfire](../tools/logfire.md)
+3. document the incident and create a new runbook
+
+## general troubleshooting
+
+```bash
+# check machine status
+fly status -a relay-api
+
+# view recent logs
+fly logs -a relay-api
+
+# restart machines
+fly machines list -a relay-api
+fly machines restart <machine-id> -a relay-api
+```

docs/runbooks/connection-pool-exhaustion.md

···
+# connection pool exhaustion
+
+## symptoms
+
+- 500 errors across multiple endpoints
+- 30-second request timeouts
+- logfire shows: `QueuePool limit of size 10 overflow 5 reached, connection timed out`
+- queue listener logs: `queue listener connection lost, attempting reconnect`
+- database connection errors mentioning multiple Neon IP addresses timing out
+
+## observed behavior (2025-12-08 incident)
+
+evidence from logfire spans:
+
+| time (UTC) | event | duration |
+|------------|-------|----------|
+| 06:32:40 | queue service connected | - |
+| 06:32:50-06:33:29 | SQLAlchemy connects succeeding | 3-6ms |
+| 06:33:36 | queue heartbeat times out | 5s timeout |
+| 06:33:36-06:36:04 | ~2.5 min gap with no spans | - |
+| 06:36:04 | GET /albums starts | hangs 24 min |
+| 06:36:06 | GET /moderation starts | **succeeds in 14ms** |
+| 06:36:06 | GET /auth/me starts | hangs 18 min |
+| 06:36:31 | multiple requests | **succeed in 3-15ms** |
+
+key observation: **some connections succeed in 3ms while others hang for 20+ minutes simultaneously**. the stuck connections show psycopg retrying across 12 different Neon IP addresses.
+
+## what we know
+
+1. the queue listener heartbeat (`SELECT 1`) times out after 5 seconds
+2. psycopg retries connection attempts across multiple IPs when one fails
+3. each IP retry has its own timeout, so total time = timeout × number of IPs
+4. some connections succeed immediately while others get stuck
+5. restarting the fly machines clears the stuck connections
+
+## what we don't know
+
+- why some connections succeed while others fail simultaneously
+- whether this is a Neon proxy issue, DNS issue, or application issue
+- why psycopg doesn't give up after a reasonable total timeout
+
+## remediation
+
+restart the fly machines to clear stuck connections:
+
+```bash
+# list machines
+fly machines list -a relay-api
+
+# restart both machines
+fly machines restart <machine-id-1> <machine-id-2> -a relay-api
+```
+
+## verification
+
+check logfire for healthy spans after restart:
+
+```sql
+SELECT
+  span_name,
+  message,
+  start_timestamp,
+  duration * 1000 as duration_ms,
+  otel_status_code
+FROM records
+WHERE deployment_environment = 'production'
+  AND start_timestamp > NOW() - INTERVAL '5 minutes'
+ORDER BY start_timestamp DESC
+LIMIT 30
+```
+
+you should see:
+- `queue service connected to database and listening`
+- database queries completing in <50ms
+- no ERROR status codes
+
+## incident history
+
+- **2025-11-17**: first occurrence, queue listener hung indefinitely (fixed by adding timeout)
+- **2025-12-02**: cold start variant, 10 errors (fixed by increasing pool size)
+- **2025-12-08**: 37 errors in one hour, some connections stuck 20+ min while others worked
+
+## future investigation
+
+- consider adding a total connection timeout that caps retries across all IPs
+- investigate whether disabling IPv6 reduces retry time
+- add monitoring/alerting for queue listener disconnects
+- consider circuit breaker pattern to fail fast when connections are failing
+
+## related docs
+
+- [connection pooling config](../backend/database/connection-pooling.md)
+- [logfire querying guide](../tools/logfire.md)