documentation: Updating documentation with recent config and featuers

CLAUDE.md

···
 # QuickDID - Development Guide for Claude
 
 ## Overview
-QuickDID is a high-performance AT Protocol identity resolution service written in Rust. It provides handle-to-DID resolution with multi-layer caching (Redis, SQLite, in-memory), queue processing, metrics support, and proactive cache refreshing.
+QuickDID is a high-performance AT Protocol identity resolution service written in Rust. It provides bidirectional handle-to-DID and DID-to-handle resolution with multi-layer caching (Redis, SQLite, in-memory), queue processing, metrics support, proactive cache refreshing, and real-time cache updates via Jetstream consumer.
 
 ## Configuration
 
···
 1. **Handle Resolution** (`src/handle_resolver/`)
    - `BaseHandleResolver`: Core resolution using DNS and HTTP
    - `RateLimitedHandleResolver`: Semaphore-based rate limiting with optional timeout
-   - `CachingHandleResolver`: In-memory caching layer
-   - `RedisHandleResolver`: Redis-backed persistent caching
-   - `SqliteHandleResolver`: SQLite-backed persistent caching
+   - `CachingHandleResolver`: In-memory caching layer with bidirectional support
+   - `RedisHandleResolver`: Redis-backed persistent caching with bidirectional lookups
+   - `SqliteHandleResolver`: SQLite-backed persistent caching with bidirectional support
    - `ProactiveRefreshResolver`: Automatically refreshes cache entries before expiration
+   - All resolvers implement `HandleResolver` trait with:
+     - `resolve`: Handle-to-DID resolution
+     - `purge`: Remove entries by handle or DID
+     - `set`: Manually update handle-to-DID mappings
    - Uses binary serialization via `HandleResolutionResult` for space efficiency
    - Resolution stack: Cache → ProactiveRefresh (optional) → RateLimited (optional) → Base → DNS/HTTP
    - Includes resolution timing measurements for metrics
···
    - Tracks counters, gauges, and timings
    - Configurable tags for environment/service identification
    - No-op adapter for development environments
+   - Metrics for Jetstream event processing
+
+6. **Jetstream Consumer** (`src/jetstream_handler.rs`)
+   - Consumes AT Protocol firehose events via WebSocket
+   - Processes Account events (purges deleted/deactivated accounts)
+   - Processes Identity events (updates handle-to-DID mappings)
+   - Automatic reconnection with exponential backoff
+   - Comprehensive metrics for event processing
+   - Spawned as cancellable task using task manager
 
 ## Key Technical Details
 
···
 - Other DID methods stored with full identifier
 
 ### Redis Integration
-- **Caching**: Uses MetroHash64 for key generation, stores binary data
+- **Bidirectional Caching**: 
+  - Stores both handle→DID and DID→handle mappings
+  - Uses MetroHash64 for key generation
+  - Binary data storage for efficiency
+  - Automatic synchronization of both directions
 - **Queuing**: Reliable queue with processing/dead letter queues
 - **Key Prefixes**: Configurable via `QUEUE_REDIS_PREFIX` environment variable
 
···
    - Acquire semaphore permit (with optional timeout)
    - If timeout configured and exceeded, return error
 3. Perform DNS TXT lookup or HTTP well-known query
-4. Cache result with appropriate TTL
+4. Cache result with appropriate TTL in both directions (handle→DID and DID→handle)
 5. Return DID or error
+
+### Cache Management Operations
+- **Purge**: Removes entries by either handle or DID
+  - Uses `atproto_identity::resolve::parse_input` for identifier detection
+  - Removes both handle→DID and DID→handle mappings
+  - Chains through all resolver layers
+- **Set**: Manually updates handle-to-DID mappings
+  - Updates both directions in cache
+  - Normalizes handles to lowercase
+  - Chains through all resolver layers
 
 ## Environment Variables
 
···
 - `PROACTIVE_REFRESH_ENABLED`: Enable proactive cache refreshing (default: false)
 - `PROACTIVE_REFRESH_THRESHOLD`: Refresh when TTL remaining is below this threshold (0.0-1.0, default: 0.8)
 
+### Optional - Jetstream Consumer
+- `JETSTREAM_ENABLED`: Enable Jetstream consumer for real-time cache updates (default: false)
+- `JETSTREAM_HOSTNAME`: Jetstream WebSocket hostname (default: jetstream.atproto.tools)
+
 ## Error Handling
 
 All error strings must use this format:
···
 ### Test Coverage Areas
 - Handle resolution with various DID methods
 - Binary serialization/deserialization
-- Redis caching and expiration
+- Redis caching and expiration with bidirectional lookups
 - Queue processing logic
 - HTTP endpoint responses
+- Jetstream event handler processing
+- Purge and set operations across resolver layers
 
 ## Development Patterns
 
···
 
 ### Debugging Resolution Issues
 1. Enable debug logging: `RUST_LOG=debug`
-2. Check Redis cache: `redis-cli GET "handle:<hash>"`
+2. Check Redis cache: 
+   - Handle lookup: `redis-cli GET "handle:<hash>"`
+   - DID lookup: `redis-cli GET "handle:<hash>"` (same key format)
 3. Check SQLite cache: `sqlite3 quickdid.db "SELECT * FROM handle_resolution_cache;"`
 4. Monitor queue processing in logs
 5. Check rate limiting: Look for "Rate limit permit acquisition timed out" errors
 6. Verify DNS/HTTP connectivity to AT Protocol infrastructure
 7. Monitor metrics for resolution timing and cache hit rates
+8. Check Jetstream consumer status:
+   - Look for "Jetstream consumer" log entries
+   - Monitor `jetstream.*` metrics
+   - Check reconnection attempts in logs
 
 ## Dependencies
 - `atproto-identity`: Core AT Protocol identity resolution
+- `atproto-jetstream`: AT Protocol Jetstream event consumer
 - `bincode`: Binary serialization
 - `deadpool-redis`: Redis connection pooling
 - `metrohash`: Fast non-cryptographic hashing

README.md

···
 ## Features
 
 - **Fast Handle Resolution**: Resolves AT Protocol handles to DIDs using DNS TXT records and HTTP well-known endpoints
+- **Bidirectional Caching**: Supports both handle-to-DID and DID-to-handle lookups with automatic cache synchronization
 - **Multi-Layer Caching**: Flexible caching with three tiers:
   - In-memory caching with configurable TTL (default: 600 seconds)
   - Redis-backed persistent caching (default: 90-day TTL)
   - SQLite-backed persistent caching (default: 90-day TTL)
+- **Jetstream Consumer**: Real-time cache updates from AT Protocol firehose:
+  - Processes Account and Identity events
+  - Automatically purges deleted/deactivated accounts
+  - Updates handle-to-DID mappings in real-time
+  - Comprehensive metrics for event processing
+  - Automatic reconnection with backoff
 - **HTTP Caching**: Client-side caching support with:
   - ETag generation with configurable seed for cache invalidation
   - Cache-Control headers with max-age, stale-while-revalidate, and stale-if-error directives
···
 - **Metrics & Monitoring**: 
   - StatsD metrics support for counters, gauges, and timings
   - Resolution timing measurements
+  - Jetstream event processing metrics
   - Configurable tags for environment/service identification
   - Integration guides for Telegraf and TimescaleDB
   - Configurable bind address for StatsD UDP socket (IPv4/IPv6)
···
   - Redis-based deduplication for queue items
   - Prevents duplicate handle resolution work
   - Configurable TTL for deduplication keys
+- **Cache Management APIs**: 
+  - `purge` method for removing entries by handle or DID
+  - `set` method for manually updating handle-to-DID mappings
+  - Chainable operations across resolver layers
 - **AT Protocol Compatible**: Implements XRPC endpoints for seamless integration with AT Protocol infrastructure
 - **Comprehensive Error Handling**: Structured errors with unique identifiers (e.g., `error-quickdid-config-1`), health checks, and graceful shutdown
 - **12-Factor App**: Environment-based configuration following cloud-native best practices
···
 - `PROACTIVE_REFRESH_ENABLED`: Enable proactive cache refreshing (default: false)
 - `PROACTIVE_REFRESH_THRESHOLD`: Refresh when TTL remaining is below this threshold (0.0-1.0, default: 0.8)
 
+#### Jetstream Consumer
+- `JETSTREAM_ENABLED`: Enable Jetstream consumer for real-time cache updates (default: false)
+- `JETSTREAM_HOSTNAME`: Jetstream WebSocket hostname (default: jetstream.atproto.tools)
+
 #### Static Files
 - `STATIC_FILES_DIR`: Directory for serving static files (default: www)
 
···
 
 ### Production Examples
 
-#### Redis-based with Metrics (Multi-instance/HA)
+#### Redis-based with Metrics and Jetstream (Multi-instance/HA)
 ```bash
 HTTP_EXTERNAL=quickdid.example.com \
 HTTP_PORT=3000 \
···
 METRICS_PREFIX=quickdid \
 METRICS_TAGS=env:prod,service:quickdid \
 CACHE_MAX_AGE=86400 \
+JETSTREAM_ENABLED=true \
+JETSTREAM_HOSTNAME=jetstream.atproto.tools \
 RUST_LOG=info \
 ./target/release/quickdid
 ```
···
            ↓                  ↓                 ↓               ↓
         Memory/Redis/    Background         Semaphore      AT Protocol
           SQLite         Refresher         (optional)     Infrastructure
+             ↑                                                  
+       Jetstream Consumer ← Real-time Updates from AT Protocol Firehose
 ```
 
 ### Cache Priority
···
 2. SQLite (if configured) - Best for single-instance with persistence
 3. In-memory (fallback) - Always available
 
+### Real-time Cache Updates
+When Jetstream is enabled, QuickDID maintains cache consistency by:
+- Processing Account events to purge deleted/deactivated accounts
+- Processing Identity events to update handle-to-DID mappings
+- Automatically reconnecting with exponential backoff on connection failures
+- Tracking metrics for successful and failed event processing
+
 ### Deployment Strategies
 
 - **Single-instance**: Use SQLite for both caching and queuing
 - **Multi-instance/HA**: Use Redis for distributed caching and queuing
 - **Development**: Use in-memory caching with MPSC queuing
+- **Real-time sync**: Enable Jetstream consumer for live cache updates
 
 ## API Endpoints
 

docs/configuration-reference.md

···
 - [Queue Configuration](#queue-configuration)
 - [Rate Limiting Configuration](#rate-limiting-configuration)
 - [HTTP Caching Configuration](#http-caching-configuration)
+- [Metrics Configuration](#metrics-configuration)
+- [Proactive Refresh Configuration](#proactive-refresh-configuration)
+- [Jetstream Consumer Configuration](#jetstream-consumer-configuration)
+- [Static Files Configuration](#static-files-configuration)
 - [Configuration Examples](#configuration-examples)
 - [Validation Rules](#validation-rules)
 
···
 - TTL=3600s (1 hour), threshold=0.8: Refresh after 48 minutes
 - TTL=86400s (1 day), threshold=0.8: Refresh after 19.2 hours
 
+## Jetstream Consumer Configuration
+
+### `JETSTREAM_ENABLED`
+
+**Required**: No  
+**Type**: Boolean  
+**Default**: `false`  
+
+Enable Jetstream consumer for real-time cache updates from the AT Protocol firehose. When enabled, QuickDID connects to the Jetstream WebSocket service to receive live updates about account and identity changes.
+
+**How it works**:
+- Subscribes to Account and Identity events from the firehose
+- Processes Account events to purge deleted/deactivated accounts
+- Processes Identity events to update handle-to-DID mappings
+- Automatically reconnects with exponential backoff on connection failures
+- Tracks metrics for successful and failed event processing
+
+**Examples**:
+```bash
+# Enable Jetstream consumer (recommended for production)
+JETSTREAM_ENABLED=true
+
+# Disable Jetstream consumer (default)
+JETSTREAM_ENABLED=false
+```
+
+**Benefits**:
+- Real-time cache synchronization with AT Protocol network
+- Automatic removal of deleted/deactivated accounts
+- Immediate handle change updates
+- Reduces stale data in cache
+
+**Considerations**:
+- Requires stable WebSocket connection
+- Increases network traffic (incoming events)
+- Best for services requiring up-to-date handle mappings
+- Automatically handles reconnection on failures
+
+### `JETSTREAM_HOSTNAME`
+
+**Required**: No  
+**Type**: String  
+**Default**: `jetstream.atproto.tools`  
+
+The hostname of the Jetstream WebSocket service to connect to for real-time AT Protocol events. Only used when `JETSTREAM_ENABLED=true`.
+
+**Examples**:
+```bash
+# Production firehose (default)
+JETSTREAM_HOSTNAME=jetstream.atproto.tools
+
+# Staging environment
+JETSTREAM_HOSTNAME=jetstream-staging.atproto.tools
+
+# Local development firehose
+JETSTREAM_HOSTNAME=localhost:6008
+
+# Custom deployment
+JETSTREAM_HOSTNAME=jetstream.example.com
+```
+
+**Event Processing**:
+- **Account events**: 
+  - `status: deleted` → Purges handle and DID from all caches
+  - `status: deactivated` → Purges handle and DID from all caches
+  - Other statuses → Ignored
+  
+- **Identity events**:
+  - Updates handle-to-DID mapping in cache
+  - Removes old handle mapping if changed
+  - Maintains bidirectional cache consistency
+
+**Metrics Tracked** (when metrics are enabled):
+- `jetstream.events.received`: Total events received
+- `jetstream.events.processed`: Successfully processed events
+- `jetstream.events.failed`: Failed event processing
+- `jetstream.connections.established`: Successful connections
+- `jetstream.connections.failed`: Failed connection attempts
+
+**Reconnection Behavior**:
+- Initial retry delay: 1 second
+- Maximum retry delay: 60 seconds
+- Exponential backoff with jitter
+- Automatic recovery on transient failures
+
+**Recommendations**:
+- **Production**: Use default `jetstream.atproto.tools`
+- **Development**: Consider local firehose for testing
+- **High availability**: Monitor connection metrics
+- **Network issues**: Check WebSocket connectivity
+
 ## Static Files Configuration
 
 ### `STATIC_FILES_DIR`
···
 PROACTIVE_REFRESH_ENABLED=true
 PROACTIVE_REFRESH_THRESHOLD=0.8
 
+# Jetstream Consumer (optional, recommended for real-time sync)
+JETSTREAM_ENABLED=true
+JETSTREAM_HOSTNAME=jetstream.atproto.tools
+
 # HTTP Caching (Cache-Control headers)
 CACHE_MAX_AGE=86400  # 24 hours
 CACHE_STALE_IF_ERROR=172800  # 48 hours
···
 # Rate Limiting (optional, recommended for production)
 RESOLVER_MAX_CONCURRENT=100
 RESOLVER_MAX_CONCURRENT_TIMEOUT_MS=5000  # 5 second timeout
+
+# Jetstream Consumer (optional, recommended for real-time sync)
+JETSTREAM_ENABLED=true
+JETSTREAM_HOSTNAME=jetstream.atproto.tools
 
 # HTTP Caching (Cache-Control headers)
 CACHE_MAX_AGE=86400  # 24 hours
···
 # Proactive Refresh (recommended for HA)
 PROACTIVE_REFRESH_ENABLED=true
 PROACTIVE_REFRESH_THRESHOLD=0.7  # More aggressive for HA
+
+# Jetstream Consumer (recommended for real-time sync in HA)
+JETSTREAM_ENABLED=true
+JETSTREAM_HOSTNAME=jetstream.atproto.tools
 
 # Logging
 RUST_LOG=warn
···
       CACHE_TTL_REDIS: 86400
       QUEUE_ADAPTER: redis
       QUEUE_REDIS_TIMEOUT: 5
+      JETSTREAM_ENABLED: true
+      JETSTREAM_HOSTNAME: jetstream.atproto.tools
       RUST_LOG: info
     ports:
       - "8080:8080"
···
       QUEUE_ADAPTER: sqlite
       QUEUE_BUFFER_SIZE: 5000
       QUEUE_SQLITE_MAX_SIZE: 10000
+      JETSTREAM_ENABLED: true
+      JETSTREAM_HOSTNAME: jetstream.atproto.tools
       RUST_LOG: info
     ports:
       - "8080:8080"
···
 2. **Single-instance deployments**: Use SQLite for persistent caching and queuing
 3. **Development/testing**: Use memory-only caching with MPSC queuing
 4. **Hybrid setups**: Configure both Redis and SQLite for redundancy
-5. **Queue adapter guidelines**:
+5. **Real-time sync**: Enable Jetstream consumer for live cache updates
+6. **Queue adapter guidelines**:
    - Redis: Best for multi-instance deployments with distributed processing
    - SQLite: Best for single-instance deployments needing persistence
    - MPSC: Best for single-instance deployments without persistence needs
-6. **Cache TTL guidelines**:
+7. **Cache TTL guidelines**:
    - Redis: Shorter TTLs (1-7 days) for frequently updated handles
    - SQLite: Longer TTLs (7-90 days) for stable single-instance caching
    - Memory: Short TTLs (5-30 minutes) as fallback
+8. **Jetstream guidelines**:
+   - Production: Enable for real-time cache synchronization
+   - High-traffic: Essential for reducing stale data
+   - Development: Can be disabled for simpler testing
+   - Monitor WebSocket connection health in production
 
 ### Monitoring
 

docs/production-deployment.md

···
 PROACTIVE_REFRESH_THRESHOLD=0.8
 
 # ----------------------------------------------------------------------------
+# JETSTREAM CONSUMER CONFIGURATION
+# ----------------------------------------------------------------------------
+
+# Enable Jetstream consumer for real-time cache updates (default: false)
+# When enabled, connects to AT Protocol firehose for live updates
+# Processes Account events (deleted/deactivated) and Identity events (handle changes)
+# Automatically reconnects with exponential backoff on connection failures
+JETSTREAM_ENABLED=false
+
+# Jetstream WebSocket hostname (default: jetstream.atproto.tools)
+# The firehose service to connect to for real-time AT Protocol events
+# Examples:
+#   - jetstream.atproto.tools (production firehose)
+#   - jetstream-staging.atproto.tools (staging environment)
+#   - localhost:6008 (local development)
+JETSTREAM_HOSTNAME=jetstream.atproto.tools
+
+# ----------------------------------------------------------------------------
 # STATIC FILES CONFIGURATION
 # ----------------------------------------------------------------------------
 
···
 
 ## Docker Compose Setup
 
-### Redis-based Production Setup
+### Redis-based Production Setup with Jetstream
 
-Create a `docker-compose.yml` file for a complete production setup with Redis:
+Create a `docker-compose.yml` file for a complete production setup with Redis and optional Jetstream consumer:
 
 ```yaml
 version: '3.8'
···
     driver: local
 ```
 
-### SQLite-based Single-Instance Setup
+### SQLite-based Single-Instance Setup with Jetstream
 
-For single-instance deployments without Redis, create a simpler `docker-compose.sqlite.yml`:
+For single-instance deployments without Redis, create a simpler `docker-compose.sqlite.yml` with optional Jetstream consumer:
 
 ```yaml
 version: '3.8'
···
       QUEUE_ADAPTER: sqlite
       QUEUE_BUFFER_SIZE: 5000
       QUEUE_SQLITE_MAX_SIZE: 10000
+      # Optional: Enable Jetstream for real-time cache updates
+      # JETSTREAM_ENABLED: true
+      # JETSTREAM_HOSTNAME: jetstream.atproto.tools
       RUST_LOG: info
     ports:
       - "8080:8080"
···
 2. **SQLite** (persistent, best for single-instance)
 3. **Memory** (fast, but lost on restart)
 
+**Real-time Updates with Jetstream**: When `JETSTREAM_ENABLED=true`, QuickDID:
+- Connects to AT Protocol firehose for live cache updates
+- Processes Account events to purge deleted/deactivated accounts
+- Processes Identity events to update handle-to-DID mappings
+- Automatically reconnects with exponential backoff on failures
+- Tracks metrics for successful and failed event processing
+
 **Recommendations by Deployment Type**:
 - **Single instance, persistent**: Use SQLite for both caching and queuing (`SQLITE_URL=sqlite:./quickdid.db`, `QUEUE_ADAPTER=sqlite`)
 - **Multi-instance, HA**: Use Redis for both caching and queuing (`REDIS_URL=redis://redis:6379/0`, `QUEUE_ADAPTER=redis`)
+- **Real-time sync**: Enable Jetstream consumer (`JETSTREAM_ENABLED=true`) for live cache updates
 - **Testing/development**: Use memory-only caching with MPSC queuing (`QUEUE_ADAPTER=mpsc`)
 - **Hybrid**: Configure both Redis and SQLite for redundancy
 

src/bin/quickdid.rs

···
                         compression: false,
                         zstd_dictionary_location: String::new(),
                         jetstream_hostname: jetstream_hostname.clone(),
-                        collections: vec![], // Listen to all collections
+                        // Listen to the "community.lexicon.collection.fake" collection
+                        // so that we keep an active connection open but only for
+                        // account and identity events.
+                        collections: vec!["community.lexicon.collection.fake".to_string()], // Listen to all collections
                         dids: vec![],
                         max_message_size_bytes: None,
                         cursor: None,