mstallmo.com
forked from
smokesignal.events/quickdid
QuickDID is a high-performance AT Protocol identity resolution service written in Rust. It provides handle-to-DID resolution with Redis-backed caching and queue processing.
1# QuickDID
2
3QuickDID is a high-performance AT Protocol identity resolution service written in Rust. It provides blazing-fast handle-to-DID resolution with intelligent caching strategies, supporting in-memory, Redis-backed, and SQLite-backed persistent caching with binary serialization for optimal storage efficiency. The service includes proactive cache refreshing to maintain optimal performance and comprehensive metrics support for production monitoring.
4
5Built following the 12-factor app methodology with minimal dependencies and optimized for production use, QuickDID delivers exceptional performance while maintaining a lean footprint. Configuration is handled exclusively through environment variables, with only `--version` and `--help` command-line arguments supported.
6
7## ⚠️ Production Disclaimer
8
9**This project is a release candidate and has not been fully vetted for production use.** While it includes comprehensive error handling and has been designed with production features in mind, more thorough testing is necessary before deploying in critical environments. Use at your own risk and conduct appropriate testing for your use case.
10
11## Performance
12
13QuickDID is designed for high throughput and low latency:
14
15- **Binary serialization** reduces cache storage by ~40% compared to JSON
16- **Rate limiting** protects upstream services from being overwhelmed
17- **Work shedding** in SQLite queue adapter prevents unbounded growth
18- **Configurable TTLs** allow fine-tuning cache freshness vs. performance
19- **Connection pooling** for Redis minimizes connection overhead
20
21## Features
22
23- **Fast Handle Resolution**: Resolves AT Protocol handles to DIDs using DNS TXT records and HTTP well-known endpoints
24- **Bidirectional Caching**: Supports both handle-to-DID and DID-to-handle lookups with automatic cache synchronization
25- **Multi-Layer Caching**: Flexible caching with three tiers:
26 - In-memory caching with configurable TTL (default: 600 seconds)
27 - Redis-backed persistent caching (default: 90-day TTL)
28 - SQLite-backed persistent caching (default: 90-day TTL)
29- **Jetstream Consumer**: Real-time cache updates from AT Protocol firehose:
30 - Processes Account and Identity events
31 - Automatically purges deleted/deactivated accounts
32 - Updates handle-to-DID mappings in real-time
33 - Comprehensive metrics for event processing
34 - Automatic reconnection with backoff
35- **HTTP Caching**: Client-side caching support with:
36 - ETag generation with configurable seed for cache invalidation
37 - Cache-Control headers with max-age, stale-while-revalidate, and stale-if-error directives
38 - CORS headers for cross-origin requests
39- **Rate Limiting**: Semaphore-based concurrency control with optional timeout to protect upstream services
40- **Binary Serialization**: Compact storage format reduces cache size by ~40% compared to JSON
41- **Queue Processing**: Asynchronous handle resolution with multiple adapters:
42 - MPSC (in-memory, default)
43 - Redis (distributed)
44 - SQLite (persistent with work shedding)
45 - No-op (testing)
46- **Metrics & Monitoring**:
47 - StatsD metrics support for counters, gauges, and timings
48 - Resolution timing measurements
49 - Jetstream event processing metrics
50 - Configurable tags for environment/service identification
51 - Integration guides for Telegraf and TimescaleDB
52 - Configurable bind address for StatsD UDP socket (IPv4/IPv6)
53- **Proactive Cache Refresh**:
54 - Automatically refreshes cache entries before expiration
55 - Configurable refresh threshold
56 - Prevents cache misses for frequently accessed handles
57 - Metrics tracking for refresh operations
58- **Queue Deduplication**:
59 - Redis-based deduplication for queue items
60 - Prevents duplicate handle resolution work
61 - Configurable TTL for deduplication keys
62- **Cache Management APIs**:
63 - `purge` method for removing entries by handle or DID
64 - `set` method for manually updating handle-to-DID mappings
65 - Chainable operations across resolver layers
66- **AT Protocol Compatible**: Implements XRPC endpoints for seamless integration with AT Protocol infrastructure
67- **Comprehensive Error Handling**: Structured errors with unique identifiers (e.g., `error-quickdid-config-1`), health checks, and graceful shutdown
68- **12-Factor App**: Environment-based configuration following cloud-native best practices
69- **Minimal Dependencies**: Optimized dependency tree for faster compilation and reduced attack surface
70
71## Building
72
73### Prerequisites
74
75- Rust 1.70 or later
76- Redis (optional, for persistent caching and distributed queuing)
77- SQLite 3.35+ (optional, for single-instance persistent caching)
78
79### Build Commands
80
81```bash
82# Clone the repository
83git clone https://github.com/yourusername/quickdid.git
84cd quickdid
85
86# Build the project
87cargo build --release
88
89# Run tests
90cargo test
91
92# Run with debug logging
93RUST_LOG=debug cargo run
94```
95
96## Minimum Configuration
97
98QuickDID requires minimal configuration to run. Configuration is validated at startup, and the service will exit with specific error codes if validation fails.
99
100### Required
101
102- `HTTP_EXTERNAL`: External hostname for service endpoints (e.g., `localhost:3007`)
103
104### Example Minimal Setup
105
106```bash
107HTTP_EXTERNAL=localhost:3007 cargo run
108```
109
110### Static Files
111
112QuickDID serves static files from the `www` directory by default. This includes:
113- Landing page (`index.html`)
114- AT Protocol well-known files (`.well-known/atproto-did` and `.well-known/did.json`)
115
116Generate the `.well-known` files for your deployment:
117
118```bash
119HTTP_EXTERNAL=your-domain.com ./generate-wellknown.sh
120```
121
122This will start QuickDID with:
123- HTTP server on port 8080 (default)
124- In-memory caching only (600-second TTL default)
125- MPSC queue adapter for async processing
126- Default worker ID: "worker1"
127- Connection to plc.directory for DID resolution
128- Rate limiting disabled (default)
129
130### Optional Configuration
131
132For production deployments, consider these additional environment variables:
133
134#### Network & Service
135- `HTTP_PORT`: Server port (default: 8080)
136- `PLC_HOSTNAME`: PLC directory hostname (default: plc.directory)
137- `USER_AGENT`: HTTP User-Agent for outgoing requests
138- `DNS_NAMESERVERS`: Custom DNS servers (comma-separated)
139
140#### Caching
141- `REDIS_URL`: Redis connection URL (e.g., `redis://localhost:6379`)
142- `SQLITE_URL`: SQLite database URL (e.g., `sqlite:./quickdid.db`)
143- `CACHE_TTL_MEMORY`: In-memory cache TTL in seconds (default: 600)
144- `CACHE_TTL_REDIS`: Redis cache TTL in seconds (default: 7776000 = 90 days)
145- `CACHE_TTL_SQLITE`: SQLite cache TTL in seconds (default: 7776000 = 90 days)
146
147#### Queue Processing
148- `QUEUE_ADAPTER`: Queue type - 'mpsc', 'redis', 'sqlite', 'noop', or 'none' (default: mpsc)
149- `QUEUE_WORKER_ID`: Worker identifier (default: worker1)
150- `QUEUE_BUFFER_SIZE`: MPSC queue buffer size (default: 1000)
151- `QUEUE_REDIS_PREFIX`: Redis key prefix for queues (default: queue:handleresolver:)
152- `QUEUE_REDIS_TIMEOUT`: Redis blocking timeout in seconds (default: 5)
153- `QUEUE_REDIS_DEDUP_ENABLED`: Enable queue deduplication (default: false)
154- `QUEUE_REDIS_DEDUP_TTL`: TTL for deduplication keys in seconds (default: 60)
155- `QUEUE_SQLITE_MAX_SIZE`: Max SQLite queue size for work shedding (default: 10000)
156
157#### Rate Limiting
158- `RESOLVER_MAX_CONCURRENT`: Maximum concurrent handle resolutions (default: 0 = disabled)
159- `RESOLVER_MAX_CONCURRENT_TIMEOUT_MS`: Timeout for acquiring rate limit permit in ms (default: 0 = no timeout)
160
161#### HTTP Cache Control
162- `CACHE_MAX_AGE`: Max-age for Cache-Control header in seconds (default: 86400)
163- `CACHE_STALE_IF_ERROR`: Stale-if-error directive in seconds (default: 172800)
164- `CACHE_STALE_WHILE_REVALIDATE`: Stale-while-revalidate in seconds (default: 86400)
165- `CACHE_MAX_STALE`: Max-stale directive in seconds (default: 86400)
166- `ETAG_SEED`: Seed value for ETag generation (default: application version)
167
168#### Metrics
169- `METRICS_ADAPTER`: Metrics adapter type - 'noop' or 'statsd' (default: noop)
170- `METRICS_STATSD_HOST`: StatsD host and port (required when METRICS_ADAPTER=statsd)
171- `METRICS_STATSD_BIND`: Bind address for StatsD UDP socket (default: [::]:0 for IPv6, can use 0.0.0.0:0 for IPv4)
172- `METRICS_PREFIX`: Prefix for all metrics (default: quickdid)
173- `METRICS_TAGS`: Comma-separated tags (e.g., env:prod,service:quickdid)
174
175#### Proactive Refresh
176- `PROACTIVE_REFRESH_ENABLED`: Enable proactive cache refreshing (default: false)
177- `PROACTIVE_REFRESH_THRESHOLD`: Refresh when TTL remaining is below this threshold (0.0-1.0, default: 0.8)
178
179#### Jetstream Consumer
180- `JETSTREAM_ENABLED`: Enable Jetstream consumer for real-time cache updates (default: false)
181- `JETSTREAM_HOSTNAME`: Jetstream WebSocket hostname (default: jetstream.atproto.tools)
182
183#### Static Files
184- `STATIC_FILES_DIR`: Directory for serving static files (default: www)
185
186#### Logging
187- `RUST_LOG`: Logging level (e.g., debug, info, warn, error)
188
189### Production Examples
190
191#### Redis-based with Metrics and Jetstream (Multi-instance/HA)
192```bash
193HTTP_EXTERNAL=quickdid.example.com \
194HTTP_PORT=3000 \
195REDIS_URL=redis://localhost:6379 \
196CACHE_TTL_REDIS=86400 \
197QUEUE_ADAPTER=redis \
198QUEUE_WORKER_ID=prod-worker-1 \
199RESOLVER_MAX_CONCURRENT=100 \
200RESOLVER_MAX_CONCURRENT_TIMEOUT_MS=5000 \
201METRICS_ADAPTER=statsd \
202METRICS_STATSD_HOST=localhost:8125 \
203METRICS_PREFIX=quickdid \
204METRICS_TAGS=env:prod,service:quickdid \
205CACHE_MAX_AGE=86400 \
206JETSTREAM_ENABLED=true \
207JETSTREAM_HOSTNAME=jetstream.atproto.tools \
208RUST_LOG=info \
209./target/release/quickdid
210```
211
212#### SQLite-based (Single-instance)
213```bash
214HTTP_EXTERNAL=quickdid.example.com \
215HTTP_PORT=3000 \
216SQLITE_URL=sqlite:./quickdid.db \
217CACHE_TTL_SQLITE=86400 \
218QUEUE_ADAPTER=sqlite \
219QUEUE_SQLITE_MAX_SIZE=10000 \
220RESOLVER_MAX_CONCURRENT=50 \
221RUST_LOG=info \
222./target/release/quickdid
223```
224
225## Architecture
226
227QuickDID uses a layered architecture for optimal performance:
228
229```
230Request → Cache Layer → Proactive Refresh → Rate Limiter → Base Resolver → DNS/HTTP
231 ↓ ↓ ↓ ↓
232 Memory/Redis/ Background Semaphore AT Protocol
233 SQLite Refresher (optional) Infrastructure
234 ↑
235 Jetstream Consumer ← Real-time Updates from AT Protocol Firehose
236```
237
238### Cache Priority
239QuickDID checks caches in this order:
2401. Redis (if configured) - Best for distributed deployments
2412. SQLite (if configured) - Best for single-instance with persistence
2423. In-memory (fallback) - Always available
243
244### Real-time Cache Updates
245When Jetstream is enabled, QuickDID maintains cache consistency by:
246- Processing Account events to purge deleted/deactivated accounts
247- Processing Identity events to update handle-to-DID mappings
248- Automatically reconnecting with exponential backoff on connection failures
249- Tracking metrics for successful and failed event processing
250
251### Deployment Strategies
252
253- **Single-instance**: Use SQLite for both caching and queuing
254- **Multi-instance/HA**: Use Redis for distributed caching and queuing
255- **Development**: Use in-memory caching with MPSC queuing
256- **Real-time sync**: Enable Jetstream consumer for live cache updates
257
258## API Endpoints
259
260- `GET /_health` - Health check endpoint
261- `GET /xrpc/com.atproto.identity.resolveHandle` - Resolve handle to DID
262- `GET /.well-known/atproto-did` - Serve DID document for the service
263- `OPTIONS /*` - CORS preflight support for all endpoints
264
265## Docker Deployment
266
267QuickDID can be deployed using Docker. See the [production deployment guide](docs/production-deployment.md) for detailed Docker and Docker Compose configurations.
268
269### Quick Docker Setup
270
271```bash
272# Build the image
273docker build -t quickdid:latest .
274
275# Run with environment file
276docker run -d \
277 --name quickdid \
278 --env-file .env \
279 -p 8080:8080 \
280 quickdid:latest
281```
282
283## Documentation
284
285- [Configuration Reference](docs/configuration-reference.md) - Complete list of all configuration options
286- [Production Deployment Guide](docs/production-deployment.md) - Docker, monitoring, and production best practices
287- [Metrics Guide](docs/telegraf-timescaledb-metrics-guide.md) - Setting up metrics with Telegraf and TimescaleDB
288- [Development Guide](CLAUDE.md) - Architecture details and development patterns
289
290## Railway Deployment
291
292QuickDID includes Railway deployment resources in the `railway-resources/` directory for easy deployment with metrics support via Telegraf. See the deployment configurations for one-click deployment options.
293
294## License
295
296This project is open source and available under the MIT License.
297
298Copyright (c) 2025 Nick Gerakines
299
300Permission is hereby granted, free of charge, to any person obtaining a copy
301of this software and associated documentation files (the "Software"), to deal
302in the Software without restriction, including without limitation the rights
303to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
304copies of the Software, and to permit persons to whom the Software is
305furnished to do so, subject to the following conditions:
306
307The above copyright notice and this permission notice shall be included in all
308copies or substantial portions of the Software.
309
310THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
311IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
312FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
313AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
314LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
315OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
316SOFTWARE.