documentation: guide to using telegraf and timescaledb for metrics

.gitignore

···
 
 # Docker
 docker-compose.override.yml
+
+# Metrics dev
+/metrics-stack

docs/telegraf-timescaledb-metrics-guide.md

···
+# Telegraf and TimescaleDB Metrics Collection Guide
+
+This guide demonstrates how to set up a metrics collection pipeline using Telegraf to collect StatsD metrics and store them in PostgreSQL with TimescaleDB using Docker Compose.
+
+## Overview
+
+This setup creates a metrics pipeline that:
+- Collects StatsD metrics via Telegraf on UDP port 8125
+- Stores metrics in PostgreSQL with TimescaleDB extensions
+- Automatically creates hypertables for time-series optimization
+- Provides a complete Docker Compose configuration for easy deployment
+
+## Prerequisites
+
+- Docker and Docker Compose installed
+- Basic understanding of StatsD metrics format
+- Familiarity with PostgreSQL/TimescaleDB concepts
+
+## Project Structure
+
+Create the following directory structure:
+
+```
+metrics-stack/
+├── docker-compose.yml
+├── telegraf/
+│   └── telegraf.conf
+└── .env
+```
+
+## Configuration Files
+
+### 1. Environment Variables (.env)
+
+Create a `.env` file to store sensitive configuration:
+
+```env
+# PostgreSQL/TimescaleDB Configuration
+POSTGRES_DB=metrics
+POSTGRES_USER=postgres
+POSTGRES_PASSWORD=secretpassword
+
+# Telegraf Database User
+TELEGRAF_DB_USER=postgres
+TELEGRAF_DB_PASSWORD=secretpassword
+
+# TimescaleDB Settings
+TIMESCALE_TELEMETRY=off
+```
+
+### 2. Telegraf Configuration (telegraf/telegraf.conf)
+
+Create the Telegraf configuration file:
+
+```toml
+# Global Telegraf Agent Configuration
+[agent]
+  interval = "10s"
+  round_interval = true
+  metric_batch_size = 1000
+  metric_buffer_limit = 10000
+  collection_jitter = "0s"
+  flush_interval = "10s"
+  flush_jitter = "0s"
+  precision = ""
+  debug = false
+  quiet = false
+  hostname = "telegraf-agent"
+  omit_hostname = false
+
+# StatsD Input Plugin
+[[inputs.statsd]]
+  service_address = ":8125"  # Listen on UDP port 8125 for StatsD metrics
+  protocol = "udp"
+  delete_gauges = true
+  delete_counters = true
+  delete_sets = true
+  delete_timings = true
+  percentiles = [50, 90, 95, 99]
+  metric_separator = "."
+  allowed_pending_messages = 10000
+  datadog_extensions = true
+  datadog_distributions = true
+
+# PostgreSQL (TimescaleDB) Output Plugin
+[[outputs.postgresql]]
+  connection = "host=timescaledb user=${TELEGRAF_DB_USER} password=${TELEGRAF_DB_PASSWORD} dbname=${POSTGRES_DB} sslmode=disable"
+  schema = "public"
+  create_templates = [
+    '''CREATE TABLE IF NOT EXISTS {{.table}} ({{.columns}})''',
+    '''SELECT create_hypertable({{.table|quoteLiteral}}, 'time', if_not_exists => TRUE)''',
+  ]
+  tags_as_jsonb = true
+
+```
+
+### 3. Docker Compose Configuration (docker-compose.yml)
+
+Create the Docker Compose file:
+
+```yaml
+version: '3.8'
+
+services:
+  timescaledb:
+    image: timescale/timescaledb-ha:pg17
+    container_name: timescaledb
+    restart: unless-stopped
+    environment:
+      POSTGRES_DB: ${POSTGRES_DB}
+      POSTGRES_USER: ${POSTGRES_USER}
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
+      TIMESCALE_TELEMETRY: ${TIMESCALE_TELEMETRY}
+    ports:
+      - "5442:5432"
+    volumes:
+      - timescale_data:/home/postgres/pgdata/data
+      - ./init-scripts:/docker-entrypoint-initdb.d:ro
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    networks:
+      - metrics_network
+
+  telegraf:
+    image: telegraf
+    container_name: telegraf
+    restart: unless-stopped
+    environment:
+      TELEGRAF_DB_USER: ${TELEGRAF_DB_USER}
+      TELEGRAF_DB_PASSWORD: ${TELEGRAF_DB_PASSWORD}
+      POSTGRES_DB: ${POSTGRES_DB}
+    ports:
+      - "8125:8125/udp"  # StatsD UDP port
+    volumes:
+      - ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
+    depends_on:
+      timescaledb:
+        condition: service_healthy
+    networks:
+      - metrics_network
+    command: ["telegraf", "--config", "/etc/telegraf/telegraf.conf"]
+
+  redis:
+    image: redis:7-alpine
+    container_name: redis
+    restart: unless-stopped
+    ports:
+      - "6379:6379"
+    volumes:
+      - redis_data:/data
+    command: redis-server --appendonly yes --appendfsync everysec
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    networks:
+      - metrics_network
+
+networks:
+  metrics_network:
+    driver: bridge
+
+volumes:
+  timescale_data:
+  redis_data:
+```
+
+### 4. Database Initialization Script (optional)
+
+Create `init-scripts/01-init.sql` to set up the Telegraf user and schema:
+
+```sql
+-- Enable TimescaleDB extension
+CREATE EXTENSION IF NOT EXISTS timescaledb;
+```
+
+## Usage
+
+### Starting the Stack
+
+1. Navigate to your project directory:
+```bash
+cd metrics-stack
+```
+
+2. Start the services:
+```bash
+docker-compose up -d
+```
+
+3. Check the logs to ensure everything is running:
+```bash
+docker-compose logs -f
+```
+
+### Sending Metrics
+
+Send test metrics to Telegraf using StatsD protocol:
+
+```bash
+# Send a counter metric
+echo "quickdid.http.request.count:1|c|#method:GET,path:/resolve,status:200" | nc -u -w0 localhost 8125
+
+# Send a gauge metric
+echo "quickdid.resolver.rate_limit.available_permits:10|g" | nc -u -w0 localhost 8125
+
+# Send a timing metric
+echo "quickdid.http.request.duration_ms:42|ms|#method:POST,path:/api,status:201" | nc -u -w0 localhost 8125
+
+# Send a histogram
+echo "quickdid.resolver.resolution_time:100|h|#resolver:redis" | nc -u -w0 localhost 8125
+```
+
+### Querying Metrics
+
+Connect to TimescaleDB to query your metrics:
+
+```bash
+# Connect to the database
+docker exec -it timescaledb psql -U postgres -d metrics
+
+# Query recent metrics
+SELECT 
+    time,
+    tags,
+    value
+FROM "quickdid.http.request.count"
+WHERE time > NOW() - INTERVAL '1 hour'
+ORDER BY time DESC
+LIMIT 10;
+
+# Query specific metric with tags
+SELECT 
+    time_bucket('1 minute', time) AS minute,
+    tags->>'method' as method,
+    tags->>'path' as path,
+    tags->>'status' as status,
+    AVG(mean::numeric) as avg_duration
+FROM "quickdid.http.request.duration_ms"
+WHERE 
+    time > NOW() - INTERVAL '1 hour'
+GROUP BY minute, method, path, status
+ORDER BY minute DESC;
+```
+
+## Advanced Configuration
+
+### Continuous Aggregates
+
+Create continuous aggregates for better query performance:
+
+```sql
+-- Create a continuous aggregate for HTTP metrics
+CREATE MATERIALIZED VIEW http_metrics_hourly
+WITH (timescaledb.continuous) AS
+SELECT
+    time_bucket('1 hour', time) AS hour,
+    name,
+    tags->>'method' as method,
+    tags->>'path' as path,
+    tags->>'status' as status,
+    COUNT(*) as request_count,
+    AVG((fields->>'value')::numeric) as avg_value,
+    MAX((fields->>'value')::numeric) as max_value,
+    MIN((fields->>'value')::numeric) as min_value
+FROM telegraf.metrics
+WHERE name LIKE 'quickdid_http_%'
+GROUP BY hour, name, method, path, status
+WITH NO DATA;
+
+-- Add refresh policy
+SELECT add_continuous_aggregate_policy('http_metrics_hourly',
+    start_offset => INTERVAL '3 hours',
+    end_offset => INTERVAL '1 hour',
+    schedule_interval => INTERVAL '1 hour');
+```
+
+## Monitoring and Maintenance
+
+### Health Checks
+
+Check the health of your TimescaleDB hypertables:
+
+```sql
+-- View hypertable information
+SELECT * FROM timescaledb_information.hypertables;
+
+-- Check chunk information
+SELECT 
+    hypertable_name,
+    chunk_name,
+    range_start,
+    range_end,
+    is_compressed
+FROM timescaledb_information.chunks
+WHERE hypertable_name = 'quickdid.http.request.duration_ms'
+ORDER BY range_start DESC
+LIMIT 10;
+
+-- Check compression status
+SELECT 
+    hypertable_name,
+    uncompressed_total_bytes,
+    compressed_total_bytes,
+    compression_ratio
+FROM timescaledb_information.hypertable_compression_stats;
+```
+
+### Troubleshooting
+
+1. **Telegraf can't connect to TimescaleDB:**
+   - Check that TimescaleDB is healthy: `docker-compose ps`
+   - Verify credentials in `.env` file
+   - Check network connectivity between containers
+
+2. **Metrics not appearing in database:**
+   - Check Telegraf logs: `docker-compose logs telegraf`
+   - Verify StatsD metrics are being received: `docker-compose logs telegraf | grep statsd`
+   - Check table creation: Connect to database and run `\dt telegraf.*`
+
+3. **Performance issues:**
+   - Check chunk sizes and adjust chunk_time_interval if needed
+   - Verify compression is working
+   - Consider creating appropriate indexes on frequently queried columns
+
+## Integration with QuickDID
+
+To integrate with QuickDID, configure it to send metrics to the Telegraf StatsD endpoint:
+
+```bash
+# Set environment variables for QuickDID
+export METRICS_ADAPTER=statsd
+export METRICS_STATSD_HOST=localhost:8125
+export METRICS_PREFIX=quickdid.
+export METRICS_TAGS=env:production,service:quickdid
+
+# Start QuickDID
+cargo run
+```
+
+QuickDID will automatically send metrics to Telegraf, which will store them in TimescaleDB for analysis.
+
+## Backup and Restore
+
+### Backup
+
+Create a backup of your metrics data:
+
+```bash
+# Backup entire database
+docker exec timescaledb pg_dump -U postgres metrics_db > metrics_backup.sql
+
+# Backup specific time range
+docker exec timescaledb pg_dump -U postgres \
+  --table='telegraf.metrics' \
+  --data-only \
+  --where="time >= '2024-01-01' AND time < '2024-02-01'" \
+  metrics_db > metrics_january.sql
+```
+
+### Restore
+
+Restore from backup:
+
+```bash
+# Restore database
+docker exec -i timescaledb psql -U postgres metrics_db < metrics_backup.sql
+```
+
+## Security Considerations
+
+1. **Use strong passwords:** Update the default passwords in `.env`
+2. **Enable SSL:** Configure `sslmode=require` in production
+3. **Network isolation:** Use Docker networks to isolate services
+4. **Access control:** Limit database user permissions to minimum required
+5. **Regular updates:** Keep Docker images updated for security patches
+
+## Performance Tuning
+
+### TimescaleDB Configuration
+
+Add these settings to optimize performance:
+
+```yaml
+# In docker-compose.yml under timescaledb service
+command:
+  - postgres
+  - -c
+  - shared_buffers=1GB
+  - -c
+  - effective_cache_size=3GB
+  - -c
+  - maintenance_work_mem=512MB
+  - -c
+  - work_mem=32MB
+  - -c
+  - timescaledb.max_background_workers=8
+  - -c
+  - max_parallel_workers_per_gather=2
+  - -c
+  - max_parallel_workers=8
+```
+
+### Telegraf Buffer Settings
+
+Adjust buffer settings based on your metric volume:
+
+```toml
+[agent]
+  metric_batch_size = 5000     # Increase for high volume
+  metric_buffer_limit = 100000  # Increase buffer size
+  flush_interval = "5s"         # Decrease for more frequent writes
+```
+
+## Conclusion
+
+This setup provides a robust metrics collection and storage solution using:
+- **Telegraf** for flexible metric collection via StatsD
+- **TimescaleDB** for efficient time-series data storage
+- **Docker Compose** for easy deployment and management
+
+The system is production-ready with support for compression, retention policies, and continuous aggregates for optimal performance at scale.