quickdid / docs / production-deployment.md

# QuickDID Production Deployment Guide

This guide provides comprehensive instructions for deploying QuickDID in a production environment using Docker. QuickDID supports multiple caching strategies: Redis (distributed), SQLite (single-instance), or in-memory caching.

## Table of Contents

- [Prerequisites](#prerequisites)
- [Environment Configuration](#environment-configuration)
- [Docker Deployment](#docker-deployment)
- [Docker Compose Setup](#docker-compose-setup)
- [Health Monitoring](#health-monitoring)
- [Security Considerations](#security-considerations)
- [Troubleshooting](#troubleshooting)

## Prerequisites

- Docker 20.10.0 or higher
- Docker Compose 2.0.0 or higher (optional, for multi-container setup)
- Redis 6.0 or higher (optional, for persistent caching and queue management)
- SQLite 3.35 or higher (optional, alternative to Redis for single-instance caching)
- Valid SSL certificates for HTTPS (recommended for production)
- Domain name configured with appropriate DNS records

## Environment Configuration

Create a `.env` file in your deployment directory with the following configuration:

```bash
# ============================================================================
# QuickDID Production Environment Configuration
# ============================================================================

# ----------------------------------------------------------------------------
# REQUIRED CONFIGURATION
# ----------------------------------------------------------------------------

# External hostname for service endpoints
# This should be your public domain name with port if non-standard
# Examples: 
#   - quickdid.example.com
#   - quickdid.example.com:8080
#   - localhost:3007 (for testing only)
HTTP_EXTERNAL=quickdid.example.com

# ----------------------------------------------------------------------------
# NETWORK CONFIGURATION
# ----------------------------------------------------------------------------

# HTTP server port (default: 8080)
# This is the port the service will bind to inside the container
# Map this to your desired external port in docker-compose.yml
HTTP_PORT=8080

# PLC directory hostname (default: plc.directory)
# Change this if using a custom PLC directory or testing environment
PLC_HOSTNAME=plc.directory

# ----------------------------------------------------------------------------
# CACHING CONFIGURATION
# ----------------------------------------------------------------------------

# Redis connection URL for caching (recommended for production)
# Format: redis://[username:password@]host:port/database
# Examples:
#   - redis://localhost:6379/0 (local Redis, no auth)
#   - redis://user:[e-mail]:6379/0 (remote with auth)
#   - redis://redis:6379/0 (Docker network)
#   - rediss://secure-redis.example.com:6380/0 (TLS)
# Benefits: Persistent cache, distributed caching, better performance
REDIS_URL=redis://redis:6379/0

# SQLite database URL for caching (alternative to Redis for single-instance deployments)
# Format: sqlite:path/to/database.db
# Examples:
#   - sqlite:./quickdid.db (file-based database)
#   - sqlite::memory: (in-memory database for testing)
#   - sqlite:/var/lib/quickdid/cache.db (absolute path)
# Benefits: Persistent cache, single-file storage, no external dependencies
# Note: Cache priority is Redis > SQLite > Memory (first available is used)
# SQLITE_URL=sqlite:./quickdid.db

# TTL for in-memory cache in seconds (default: 600 = 10 minutes)
# Range: 60-3600 recommended
# Lower = fresher data, more DNS/HTTP lookups
# Higher = better performance, potentially stale data
CACHE_TTL_MEMORY=600

# TTL for Redis cache in seconds (default: 7776000 = 90 days)
# Range: 3600-31536000 (1 hour to 1 year)
# Recommendations:
#   - 86400 (1 day) for frequently changing data
#   - 604800 (1 week) for balanced performance
#   - 7776000 (90 days) for stable data
CACHE_TTL_REDIS=86400

# TTL for SQLite cache in seconds (default: 7776000 = 90 days)
# Range: 3600-31536000 (1 hour to 1 year)
# Same recommendations as Redis TTL
# Only used when SQLITE_URL is configured
CACHE_TTL_SQLITE=86400

# ----------------------------------------------------------------------------
# QUEUE CONFIGURATION
# ----------------------------------------------------------------------------

# Queue adapter type: 'mpsc', 'redis', 'sqlite', 'noop', or 'none' (default: mpsc)
# - 'mpsc': In-memory queue for single-instance deployments
# - 'redis': Distributed queue for multi-instance or HA deployments
# - 'sqlite': Persistent queue for single-instance deployments
# - 'noop': Disable queue processing (testing only)
# - 'none': Alias for 'noop'
QUEUE_ADAPTER=redis

# Redis URL for queue adapter (uses REDIS_URL if not set)
# Set this if you want to use a separate Redis instance for queuing
# QUEUE_REDIS_URL=redis://queue-redis:6379/1

# Redis key prefix for queues (default: queue:handleresolver:)
# Useful when sharing Redis instance with other services
QUEUE_REDIS_PREFIX=queue:quickdid:prod:

# Redis blocking timeout for queue operations in seconds (default: 5)
# Range: 1-60 recommended
# Lower = more responsive to shutdown, more polling
# Higher = less polling overhead, slower shutdown
QUEUE_REDIS_TIMEOUT=5

# Enable deduplication for Redis queue to prevent duplicate handles (default: false)
# When enabled, uses Redis SET with TTL to track handles being processed
# Prevents the same handle from being queued multiple times within the TTL window
QUEUE_REDIS_DEDUP_ENABLED=false

# TTL for Redis queue deduplication keys in seconds (default: 60)
# Range: 10-300 recommended
# Determines how long to prevent duplicate handle resolution requests
QUEUE_REDIS_DEDUP_TTL=60

# Worker ID for Redis queue (defaults to "worker1")
# Set this for predictable worker identification in multi-instance deployments
# Examples: worker-001, prod-us-east-1, $(hostname)
QUEUE_WORKER_ID=prod-worker-1

# Buffer size for MPSC queue (default: 1000)
# Range: 100-100000
# Increase for high-traffic deployments using MPSC adapter
QUEUE_BUFFER_SIZE=5000

# Maximum queue size for SQLite adapter work shedding (default: 10000)
# Range: 100-1000000 (recommended)
# When exceeded, oldest entries are deleted to maintain this limit
# Set to 0 to disable work shedding (unlimited queue size)
# Benefits: Prevents unbounded disk usage, maintains recent work items
QUEUE_SQLITE_MAX_SIZE=10000

# ----------------------------------------------------------------------------
# HTTP CLIENT CONFIGURATION
# ----------------------------------------------------------------------------

# HTTP User-Agent header
# Identifies your service to other AT Protocol services
# Default: Auto-generated with current version from Cargo.toml
# Format: quickdid/{version} (+https://github.com/smokesignal.events/quickdid)
USER_AGENT=quickdid/1.0.0-rc.5 (+https://quickdid.example.com)

# Custom DNS nameservers (comma-separated)
# Use for custom DNS resolution or to bypass local DNS
# Examples:
#   - 8.8.8.8,8.8.4.4 (Google DNS)
#   - 1.1.1.1,1.0.0.1 (Cloudflare DNS)
# DNS_NAMESERVERS=1.1.1.1,1.0.0.1

# Additional CA certificates (comma-separated file paths)
# Use when connecting to services with custom CA certificates
# CERTIFICATE_BUNDLES=/certs/custom-ca.pem,/certs/internal-ca.pem

# ----------------------------------------------------------------------------
# LOGGING AND MONITORING
# ----------------------------------------------------------------------------

# Logging level (debug, info, warn, error)
# Use 'info' for production, 'debug' for troubleshooting
RUST_LOG=info

# Structured logging format (optional)
# Set to 'json' for machine-readable logs
# RUST_LOG_FORMAT=json

# ----------------------------------------------------------------------------
# RATE LIMITING CONFIGURATION
# ----------------------------------------------------------------------------

# Maximum concurrent handle resolutions (default: 0 = disabled)
# When > 0, enables semaphore-based rate limiting
# Range: 0-10000 (0 = disabled)
# Protects upstream DNS/HTTP services from being overwhelmed
RESOLVER_MAX_CONCURRENT=0

# Timeout for acquiring rate limit permit in milliseconds (default: 0 = no timeout)
# When > 0, requests will timeout if they can't acquire a permit within this time
# Range: 0-60000 (max 60 seconds)
# Prevents requests from waiting indefinitely when rate limiter is at capacity
RESOLVER_MAX_CONCURRENT_TIMEOUT_MS=0

# ----------------------------------------------------------------------------
# HTTP CACHING CONFIGURATION
# ----------------------------------------------------------------------------

# ETAG seed for cache invalidation (default: application version)
# Used to generate ETAG checksums for HTTP responses
# Changing this value invalidates all client-cached responses
# Examples:
#   - prod-2024-01-15 (deployment-specific)
#   - v1.0.0-1705344000 (version with timestamp)
#   - config-update-2024-01-15 (after configuration changes)
# Default uses the application version from Cargo.toml
# ETAG_SEED=prod-2024-01-15

# Maximum age for HTTP Cache-Control header in seconds (default: 86400 = 24 hours)
# Set to 0 to disable Cache-Control header
# Controls how long clients and intermediate caches can cache responses
CACHE_MAX_AGE=86400

# Stale-if-error directive for Cache-Control in seconds (default: 172800 = 48 hours)
# Allows stale content to be served if backend errors occur
# Provides resilience during service outages
CACHE_STALE_IF_ERROR=172800

# Stale-while-revalidate directive for Cache-Control in seconds (default: 86400 = 24 hours)
# Allows stale content to be served while fetching fresh content in background
# Improves perceived performance for users
CACHE_STALE_WHILE_REVALIDATE=86400

# Max-stale directive for Cache-Control in seconds (default: 172800 = 48 hours)
# Maximum time client will accept stale responses
# Provides upper bound on cached content age
CACHE_MAX_STALE=172800

# Min-fresh directive for Cache-Control in seconds (default: 3600 = 1 hour)
# Minimum time response must remain fresh
# Clients won't accept responses expiring within this time
CACHE_MIN_FRESH=3600

# ----------------------------------------------------------------------------
# METRICS CONFIGURATION
# ----------------------------------------------------------------------------

# Metrics adapter type: 'noop' or 'statsd' (default: noop)
# - 'noop': No metrics collection (default)
# - 'statsd': Send metrics to StatsD server
METRICS_ADAPTER=statsd

# StatsD host and port (required when METRICS_ADAPTER=statsd)
# Format: hostname:port
# Examples:
#   - localhost:8125 (local StatsD)
#   - statsd.example.com:8125 (remote StatsD)
METRICS_STATSD_HOST=localhost:8125

# Bind address for StatsD UDP socket (default: [::]:0)
# Controls which local address to bind for sending UDP packets
# Examples:
#   - [::]:0 (IPv6 any address, random port - default)
#   - 0.0.0.0:0 (IPv4 any address, random port)
#   - 192.168.1.100:0 (specific interface)
METRICS_STATSD_BIND=[::]:0

# Prefix for all metrics (default: quickdid)
# Used to namespace metrics in your monitoring system
# Examples:
#   - quickdid (default)
#   - prod.quickdid
#   - us-east-1.quickdid
METRICS_PREFIX=quickdid

# Tags for all metrics (comma-separated key:value pairs)
# Added to all metrics for filtering and grouping
# Examples:
#   - env:production,service:quickdid
#   - env:staging,region:us-east-1,version:1.0.0
METRICS_TAGS=env:production,service:quickdid

# ----------------------------------------------------------------------------
# PROACTIVE REFRESH CONFIGURATION
# ----------------------------------------------------------------------------

# Enable proactive cache refresh (default: false)
# When enabled, cache entries nearing expiration are automatically refreshed
# in the background to prevent cache misses for frequently accessed handles
PROACTIVE_REFRESH_ENABLED=false

# Threshold for proactive refresh as percentage of TTL (default: 0.8)
# Range: 0.0-1.0 (0% to 100% of TTL)
# Example: 0.8 means refresh when 80% of TTL has elapsed
# Lower values = more aggressive refreshing, higher load
# Higher values = less aggressive refreshing, more cache misses
PROACTIVE_REFRESH_THRESHOLD=0.8

# ----------------------------------------------------------------------------
# JETSTREAM CONSUMER CONFIGURATION
# ----------------------------------------------------------------------------

# Enable Jetstream consumer for real-time cache updates (default: false)
# When enabled, connects to AT Protocol firehose for live updates
# Processes Account events (deleted/deactivated) and Identity events (handle changes)
# Automatically reconnects with exponential backoff on connection failures
JETSTREAM_ENABLED=false

# Jetstream WebSocket hostname (default: jetstream.atproto.tools)
# The firehose service to connect to for real-time AT Protocol events
# Examples:
#   - jetstream.atproto.tools (production firehose)
#   - jetstream-staging.atproto.tools (staging environment)
#   - localhost:6008 (local development)
JETSTREAM_HOSTNAME=jetstream.atproto.tools

# ----------------------------------------------------------------------------
# STATIC FILES CONFIGURATION
# ----------------------------------------------------------------------------

# Directory path for serving static files (default: www)
# This directory should contain:
#   - index.html (landing page)
#   - .well-known/atproto-did (service DID identifier)
#   - .well-known/did.json (DID document)
# In Docker, this defaults to /app/www
# You can mount custom files via Docker volumes
STATIC_FILES_DIR=/app/www

# ----------------------------------------------------------------------------
# PERFORMANCE TUNING
# ----------------------------------------------------------------------------

# Tokio runtime worker threads (defaults to CPU count)
# Adjust based on your container's CPU allocation
# TOKIO_WORKER_THREADS=4

# Maximum concurrent connections (optional)
# Helps prevent resource exhaustion
# MAX_CONNECTIONS=10000

# ----------------------------------------------------------------------------
# DOCKER-SPECIFIC CONFIGURATION
# ----------------------------------------------------------------------------

# Container restart policy (for docker-compose)
# Options: no, always, on-failure, unless-stopped
RESTART_POLICY=unless-stopped

# Resource limits (for docker-compose)
# Adjust based on your available resources
MEMORY_LIMIT=512M
CPU_LIMIT=1.0
```

## Docker Deployment

### Building the Docker Image

Create a `Dockerfile` in your project root:

```dockerfile
# Build stage
FROM rust:1.75-slim AS builder

# Install build dependencies
RUN apt-get update && apt-get install -y \
    pkg-config \
    libssl-dev \
    && rm -rf /var/lib/apt/lists/*

# Create app directory
WORKDIR /app

# Copy source files
COPY Cargo.toml Cargo.lock ./
COPY src ./src

# Build the application
RUN cargo build --release

# Runtime stage
FROM debian:bookworm-slim

# Install runtime dependencies
RUN apt-get update && apt-get install -y \
    ca-certificates \
    libssl3 \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Create non-root user
RUN useradd -m -u 1000 quickdid

# Copy binary from builder
COPY --from=builder /app/target/release/quickdid /usr/local/bin/quickdid

# Set ownership and permissions
RUN chown quickdid:quickdid /usr/local/bin/quickdid

# Switch to non-root user
USER quickdid

# Expose default port
EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

# Run the application
ENTRYPOINT ["quickdid"]
```

Build the image:

```bash
docker build -t quickdid:latest .
```

### Running a Single Instance

```bash
# Run with environment file
docker run -d \
  --name quickdid \
  --env-file .env \
  -p 8080:8080 \
  --restart unless-stopped \
  quickdid:latest
```

## Docker Compose Setup

### Redis-based Production Setup with Jetstream

Create a `docker-compose.yml` file for a complete production setup with Redis and optional Jetstream consumer:

```yaml
version: '3.8'

services:
  quickdid:
    image: quickdid:latest
    container_name: quickdid
    env_file: .env
    ports:
      - "8080:8080"
    depends_on:
      redis:
        condition: service_healthy
    networks:
      - quickdid-network
    restart: ${RESTART_POLICY:-unless-stopped}
    deploy:
      resources:
        limits:
          memory: ${MEMORY_LIMIT:-512M}
          cpus: ${CPU_LIMIT:-1.0}
        reservations:
          memory: 256M
          cpus: '0.5'
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 10s
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  redis:
    image: redis:7-alpine
    container_name: quickdid-redis
    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
    volumes:
      - redis-data:/data
    networks:
      - quickdid-network
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 3
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  # Optional: Nginx reverse proxy with SSL
  nginx:
    image: nginx:alpine
    container_name: quickdid-nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./certs:/etc/nginx/certs:ro
      - ./acme-challenge:/var/www/acme:ro
    depends_on:
      - quickdid
    networks:
      - quickdid-network
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

networks:
  quickdid-network:
    driver: bridge

volumes:
  redis-data:
    driver: local
```

### SQLite-based Single-Instance Setup with Jetstream

For single-instance deployments without Redis, create a simpler `docker-compose.sqlite.yml` with optional Jetstream consumer:

```yaml
version: '3.8'

services:
  quickdid:
    image: quickdid:latest
    container_name: quickdid-sqlite
    environment:
      HTTP_EXTERNAL: quickdid.example.com
      HTTP_PORT: 8080
      SQLITE_URL: sqlite:/data/quickdid.db
      CACHE_TTL_MEMORY: 600
      CACHE_TTL_SQLITE: 86400
      QUEUE_ADAPTER: sqlite
      QUEUE_BUFFER_SIZE: 5000
      QUEUE_SQLITE_MAX_SIZE: 10000
      # Optional: Enable Jetstream for real-time cache updates
      # JETSTREAM_ENABLED: true
      # JETSTREAM_HOSTNAME: jetstream.atproto.tools
      RUST_LOG: info
    ports:
      - "8080:8080"
    volumes:
      - quickdid-data:/data
    networks:
      - quickdid-network
    restart: ${RESTART_POLICY:-unless-stopped}
    deploy:
      resources:
        limits:
          memory: ${MEMORY_LIMIT:-256M}
          cpus: ${CPU_LIMIT:-0.5}
        reservations:
          memory: 128M
          cpus: '0.25'
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 10s
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  # Optional: Nginx reverse proxy with SSL
  nginx:
    image: nginx:alpine
    container_name: quickdid-nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./certs:/etc/nginx/certs:ro
      - ./acme-challenge:/var/www/acme:ro
    depends_on:
      - quickdid
    networks:
      - quickdid-network
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

networks:
  quickdid-network:
    driver: bridge

volumes:
  quickdid-data:
    driver: local
```

### Nginx Configuration (nginx.conf)

```nginx
events {
    worker_connections 1024;
}

http {
    upstream quickdid {
        server quickdid:8080;
    }

    server {
        listen 80;
        server_name quickdid.example.com;

        # ACME challenge for Let's Encrypt
        location /.well-known/acme-challenge/ {
            root /var/www/acme;
        }

        # Redirect HTTP to HTTPS
        location / {
            return 301 https://$server_name$request_uri;
        }
    }

    server {
        listen 443 ssl http2;
        server_name quickdid.example.com;

        ssl_certificate /etc/nginx/certs/fullchain.pem;
        ssl_certificate_key /etc/nginx/certs/privkey.pem;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers HIGH:!aNULL:!MD5;

        location / {
            proxy_pass http://quickdid;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            
            # WebSocket support (if needed)
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            # Timeouts
            proxy_connect_timeout 60s;
            proxy_send_timeout 60s;
            proxy_read_timeout 60s;
        }

        # Health check endpoint
        location /health {
            proxy_pass http://quickdid/health;
            access_log off;
        }
    }
}
```

### Starting the Stack

```bash
# Start Redis-based stack
docker-compose up -d

# Start SQLite-based stack
docker-compose -f docker-compose.sqlite.yml up -d

# View logs
docker-compose logs -f
# or for SQLite setup
docker-compose -f docker-compose.sqlite.yml logs -f

# Check service status
docker-compose ps

# Stop all services
docker-compose down
# or for SQLite setup
docker-compose -f docker-compose.sqlite.yml down
```

## Health Monitoring

QuickDID provides health check endpoints for monitoring:

### Basic Health Check

```bash
curl http://quickdid.example.com/health
```

Expected response:
```json
{
  "status": "healthy",
  "version": "1.0.0",
  "uptime_seconds": 3600
}
```

### Monitoring with Prometheus (Optional)

Add to your `docker-compose.yml`:

```yaml
  prometheus:
    image: prom/prometheus:latest
    container_name: quickdid-prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    ports:
      - "9090:9090"
    networks:
      - quickdid-network
    restart: unless-stopped

volumes:
  prometheus-data:
    driver: local
```

## Security Considerations

### 1. Service Key Protection

- **Never commit** sensitive configuration to version control
- Store keys in a secure secret management system (e.g., HashiCorp Vault, AWS Secrets Manager)
- Rotate keys regularly
- Use different keys for different environments

### 2. Network Security

- Use HTTPS in production with valid SSL certificates
- Implement rate limiting at the reverse proxy level
- Use firewall rules to restrict access to Redis
- Enable Redis authentication in production

### 3. Container Security

- Run containers as non-root user (already configured in Dockerfile)
- Keep base images updated
- Scan images for vulnerabilities regularly
- Use read-only filesystems where possible

### 4. Redis Security

```bash
# Add to Redis configuration for production
requirepass your_strong_password_here
maxclients 10000
timeout 300
```

### 5. Environment Variables

- Use Docker secrets or external secret management
- Avoid logging sensitive environment variables
- Implement proper access controls

## Troubleshooting

### Common Issues and Solutions

#### 1. Container Won't Start

```bash
# Check logs
docker logs quickdid

# Verify environment variables
docker exec quickdid env | grep -E "HTTP_EXTERNAL|HTTP_PORT"

# Test Redis connectivity
docker exec quickdid redis-cli -h redis ping
```

#### 2. Handle Resolution Failures

```bash
# Enable debug logging
docker exec quickdid sh -c "export RUST_LOG=debug"

# Check DNS resolution
docker exec quickdid nslookup plc.directory

# Verify Redis cache (if using Redis)
docker exec -it quickdid-redis redis-cli
> KEYS handle:*
> TTL handle:example_key

# Check SQLite cache (if using SQLite)
docker exec quickdid sqlite3 /data/quickdid.db ".tables"
docker exec quickdid sqlite3 /data/quickdid.db "SELECT COUNT(*) FROM handle_resolution_cache;"
```

#### 3. Performance Issues

```bash
# Monitor Redis memory usage (if using Redis)
docker exec quickdid-redis redis-cli INFO memory

# Check SQLite database size (if using SQLite)
docker exec quickdid ls -lh /data/quickdid.db
docker exec quickdid sqlite3 /data/quickdid.db "PRAGMA page_count; PRAGMA page_size;"

# Check container resource usage
docker stats quickdid

# Analyze slow queries (with debug logging)
docker logs quickdid | grep "resolution took"
```

#### 4. Health Check Failures

```bash
# Manual health check
docker exec quickdid curl -v http://localhost:8080/health

# Check service binding
docker exec quickdid netstat -tlnp | grep 8080
```

### Debugging Commands

```bash
# Interactive shell in container
docker exec -it quickdid /bin/bash

# Test handle resolution
curl "http://localhost:8080/xrpc/com.atproto.identity.resolveHandle?handle=example.bsky.social"

# Check Redis keys (if using Redis)
docker exec quickdid-redis redis-cli --scan --pattern "handle:*" | head -20

# Check SQLite cache entries (if using SQLite)
docker exec quickdid sqlite3 /data/quickdid.db "SELECT COUNT(*) as total_entries, MIN(updated) as oldest, MAX(updated) as newest FROM handle_resolution_cache;"

# Check SQLite queue entries (if using SQLite queue adapter)
docker exec quickdid sqlite3 /data/quickdid.db "SELECT COUNT(*) as queue_entries, MIN(queued_at) as oldest, MAX(queued_at) as newest FROM handle_resolution_queue;"

# Monitor real-time logs
docker-compose logs -f quickdid | grep -E "ERROR|WARN"
```

## Maintenance

### Backup and Restore

#### Redis Backup
```bash
# Backup Redis data
docker exec quickdid-redis redis-cli BGSAVE
docker cp quickdid-redis:/data/dump.rdb ./backups/redis-$(date +%Y%m%d).rdb

# Restore Redis data
docker cp ./backups/redis-backup.rdb quickdid-redis:/data/dump.rdb
docker restart quickdid-redis
```

#### SQLite Backup
```bash
# Backup SQLite database
docker exec quickdid sqlite3 /data/quickdid.db ".backup /tmp/backup.db"
docker cp quickdid:/tmp/backup.db ./backups/sqlite-$(date +%Y%m%d).db

# Alternative: Copy database file directly (service must be stopped)
docker-compose -f docker-compose.sqlite.yml stop quickdid
docker cp quickdid:/data/quickdid.db ./backups/sqlite-$(date +%Y%m%d).db
docker-compose -f docker-compose.sqlite.yml start quickdid

# Restore SQLite database
docker-compose -f docker-compose.sqlite.yml stop quickdid
docker cp ./backups/sqlite-backup.db quickdid:/data/quickdid.db
docker-compose -f docker-compose.sqlite.yml start quickdid
```

### Updates and Rollbacks

```bash
# Update to new version
docker pull quickdid:new-version
docker-compose down
docker-compose up -d

# Rollback if needed
docker-compose down
docker tag quickdid:previous quickdid:latest
docker-compose up -d
```

### Log Rotation

Configure Docker's built-in log rotation in `/etc/docker/daemon.json`:

```json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
```

## Performance Optimization

### Caching Strategy Selection

**Cache Priority**: QuickDID uses the first available cache in this order:
1. **Redis** (distributed, best for multi-instance)
2. **SQLite** (persistent, best for single-instance)
3. **Memory** (fast, but lost on restart)

**Real-time Updates with Jetstream**: When `JETSTREAM_ENABLED=true`, QuickDID:
- Connects to AT Protocol firehose for live cache updates
- Processes Account events to purge deleted/deactivated accounts
- Processes Identity events to update handle-to-DID mappings
- Automatically reconnects with exponential backoff on failures
- Tracks metrics for successful and failed event processing

**Recommendations by Deployment Type**:
- **Single instance, persistent**: Use SQLite for both caching and queuing (`SQLITE_URL=sqlite:./quickdid.db`, `QUEUE_ADAPTER=sqlite`)
- **Multi-instance, HA**: Use Redis for both caching and queuing (`REDIS_URL=redis://redis:6379/0`, `QUEUE_ADAPTER=redis`)
- **Real-time sync**: Enable Jetstream consumer (`JETSTREAM_ENABLED=true`) for live cache updates
- **Testing/development**: Use memory-only caching with MPSC queuing (`QUEUE_ADAPTER=mpsc`)
- **Hybrid**: Configure both Redis and SQLite for redundancy

### Queue Strategy Selection

**Queue Adapter Options**:
1. **Redis** (`QUEUE_ADAPTER=redis`) - Distributed queuing, best for multi-instance deployments
2. **SQLite** (`QUEUE_ADAPTER=sqlite`) - Persistent queuing, best for single-instance deployments
3. **MPSC** (`QUEUE_ADAPTER=mpsc`) - In-memory queuing, lightweight for single-instance without persistence needs
4. **No-op** (`QUEUE_ADAPTER=noop`) - Disable queuing entirely (testing only)

### Redis Optimization

```redis
# Add to redis.conf or pass as command arguments
maxmemory 2gb
maxmemory-policy allkeys-lru
save ""  # Disable persistence for cache-only usage
tcp-keepalive 300
timeout 0
```

### System Tuning

```bash
# Add to host system's /etc/sysctl.conf
net.core.somaxconn = 1024
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10000 65000
fs.file-max = 100000
```

## Configuration Validation

QuickDID validates all configuration at startup. The following rules are enforced:

### Required Fields

- **HTTP_EXTERNAL**: Must be provided
- **HTTP_EXTERNAL**: Must be provided

### Value Constraints

1. **TTL Values** (`CACHE_TTL_MEMORY`, `CACHE_TTL_REDIS`, `CACHE_TTL_SQLITE`):
   - Must be positive integers (> 0)
   - Recommended minimum: 60 seconds

2. **Timeout Values** (`QUEUE_REDIS_TIMEOUT`):
   - Must be positive integers (> 0)
   - Recommended range: 1-60 seconds

3. **Queue Adapter** (`QUEUE_ADAPTER`):
   - Must be one of: `mpsc`, `redis`, `sqlite`, `noop`, `none`
   - Case-sensitive

4. **Rate Limiting** (`RESOLVER_MAX_CONCURRENT`):
   - Must be between 0 and 10000
   - 0 = disabled (default)
   - When > 0, limits concurrent handle resolutions

5. **Rate Limiting Timeout** (`RESOLVER_MAX_CONCURRENT_TIMEOUT_MS`):
   - Must be between 0 and 60000 (milliseconds)
   - 0 = no timeout (default)
   - Maximum: 60000ms (60 seconds)

### Validation Errors

If validation fails, QuickDID will exit with one of these error codes:

- `error-quickdid-config-1`: Missing required environment variable
- `error-quickdid-config-2`: Invalid configuration value
- `error-quickdid-config-3`: Invalid TTL value (must be positive)
- `error-quickdid-config-4`: Invalid timeout value (must be positive)

### Testing Configuration

```bash
# Validate configuration without starting service
HTTP_EXTERNAL=test quickdid --help

# Test with specific values (will fail validation)
CACHE_TTL_MEMORY=0 quickdid --help

# Debug configuration parsing
RUST_LOG=debug HTTP_EXTERNAL=test quickdid
```

## Support and Resources

- **Documentation**: [QuickDID GitHub Repository](https://github.com/smokesignal.events/quickdid)
- **Configuration Reference**: See [configuration-reference.md](./configuration-reference.md) for detailed documentation of all options
- **AT Protocol Specs**: [atproto.com](https://atproto.com)
- **Issues**: Report bugs via GitHub Issues
- **Community**: Join the AT Protocol Discord server

## License

QuickDID is licensed under the MIT License. See LICENSE file for details.