[![Go Reference](https://pkg.go.dev/badge/go.iscode.ca/mats.svg)](https://pkg.go.dev/go.iscode.ca/mats) # mats: Minimal Numeric Time-Series Engine `mats` is a dedicated, low-resource, high-throughput embedded time-series database engine written in pure Go (`CGO_ENABLED=0`). It is specifically engineered to ingest and store telemetry from **Riemann** monitoring, scaling seamlessly from low-power home nodes (Raspberry Pi/NAS) to large-scale deployments absorbing thousands of metrics per second. For full architectural rationale, see [ADR 0001: Architecture of the Embedded Numeric Time-Series Engine](ADR-0001-mats-architecture.md). --- ## Key Architectural Principles 1. **Dual-Plane Architecture:** * **Control Plane (Metadata):** Pure-Go SQLite (`modernc.org/sqlite`) mapping string tags (`host`, `service`, `tags`) to immutable 64-bit integer `metric_id`s, backed by an in-memory bi-directional cache (`568 ns/op` fast path). * **Data Plane (Telemetry):** Strictly unboxed numeric primitives (`metric_id int64`, `timestamp int64`, `value float64`). No strings or arbitrary JSON in the telemetry stream. 2. **Tier 1 (Hot LSM Buffer):** CockroachDB's `pebble` LSM engine using a composite 20-byte key: $$\text{Key} = [\text{BucketID (4B)}][\text{MetricID (8B)}][\text{Timestamp (8B)}]$$ Enables instant, zero-cost purges via a single `db.DeleteRange([Bucket_N], [Bucket_N+1])` call without tombstone bloat. 3. **Tier 2 (Cold Archive):** Immutable Apache Parquet files stored on S3-compatible cloud storage (AWS S3, Cloudflare R2, MinIO, Versity) or local disk. Compressed with Zstandard, Delta-binary packed timestamps, and Dictionary-encoded metric IDs. 4. **Crash-Safe Handover Protocol:** Drains LSM to an ephemeral scratch file, uploads to S3, commits the manifest to SQLite, and purges the LSM range. Startup recovery is instantaneous ($< 200\text{ ms}$) and cleans dangling state. 5. **WORM Storage Security:** Cold storage is strictly Write-Once-Read-Many. The engine **never requires `s3:DeleteObject` permissions**, preventing accidental or malicious loss of historical archives. 6. **Operating System Sandboxing:** Enforces **Linux Landlock** filesystem containment and `PR_SET_NO_NEW_PRIVS` to lock down process privileges after startup. 7. **Drop & Count Policy:** Drops late-arriving metrics outside the active bucket window to prevent backfilling DoS attacks. 8. **Decoupled Legacy Bridge:** Provides an optional standalone InfluxDB 1.x `/query` proxy for drop-in compatibility with legacy Grafana instances. --- ## Build & Test ### Requirements * Go 1.22+ (verified on Go 1.27.1, ARM64 & AMD64) * Zero external C toolchains or libraries required (`CGO_ENABLED=0`) ### Building ```bash make all ``` Produces: * `bin/mats`: The core daemon (ingestion, storage, hybrid query engine, and embedded dashboard). * `bin/mats-proxy`: Secure, sandboxed public gateway and dashboard proxy. * `bin/mats-influx-proxy`: Standalone InfluxDB 1.x `/query` translation bridge. * `bin/mats-riemann-sse`: Standalone Riemann HTTP SSE forwarding proxy. ### Running Tests & Linter ```bash make test # Run full unit test suite make lint # Run golangci-lint (0 warnings) make bench # Run zero-alloc benchmarks ``` --- ## Running the Engine ```bash # Run with default settings (Daily buckets, local disk storage, Landlock enabled) ./bin/mats -data-dir ./data -scratch-dir ./scratch # Run with S3 Cold Storage & 1-Hour Buckets ./bin/mats \ -data-dir /var/lib/mats/data \ -scratch-dir /var/lib/mats/scratch \ -bucket-duration 1h \ -grace-period 15m \ -s3-endpoint https://s3.us-west-2.amazonaws.com \ -s3-bucket my-telemetry-archive \ -s3-region us-west-2 \ -s3-access-key $AWS_ACCESS_KEY_ID \ -s3-secret-key $AWS_SECRET_ACCESS_KEY ``` ### Configuration (12-Factor Env Variables) All CLI flags can be set via environment variables: * `MATS_DATA_DIR` (default: `./data`) * `MATS_SCRATCH_DIR` (default: `./scratch`) * `MATS_RIEMANN_TCP` (default: `:5555`) * `MATS_RIEMANN_UDP` (default: `:5555`) * `MATS_HTTP_ADDR` (default: `:8080`) * `MATS_BUCKET_DURATION` (default: `24h`) * `MATS_GRACE_PERIOD` (default: `30m`) * `MATS_NODE_ID` (node identifier for multi-ingress storage key namespacing) * `MATS_QUERY_CACHE` (default: `true`) * `MATS_QUERY_CACHE_SIZE` (default: `512`) * `MATS_QUERY_MAX_POINTS` (default: `1500`, 0 to disable) * `MATS_COMPACTION` (default: `true`) * `MATS_COMPACTION_WINDOW` (default: `24h`) * `MATS_S3_ENDPOINT` * `MATS_S3_BUCKET` * `MATS_S3_REGION` (default: `us-east-1`) * `MATS_S3_ACCESS_KEY` * `MATS_S3_SECRET_KEY` * `MATS_S3_DISABLE_TLS` (default: `false`) * `MATS_ENABLE_LANDLOCK` (default: `true`) ### Bucket Duration & Grace Period Tuning `MATS_BUCKET_DURATION` sets the time span for each immutable cold Parquet archive file, while `MATS_GRACE_PERIOD` governs: 1. **Late-arrival tolerance:** Incoming metrics with timestamp `< (now - grace_period)` are discarded to prevent backfilling DoS. 2. **Cold archiving delay:** Closed buckets are retained in the hot Pebble buffer until `bucket_end + grace_period` before being sealed and uploaded to cold storage. When configuring non-default bucket durations, pair them with a proportional grace period: | Bucket Duration | Recommended Grace Period | Typical Use Case | Cold Tier Archival Delay | |:----------------|:-------------------------|:-----------------------------------------|:----------------------------| | `15m` | `2m` – `5m` | Fast cold rotation, low memory footprint | 17m – 20m from bucket start | | `1h` | `5m` – `15m` | Hourly archiving, balanced buffer size | 65m – 75m from bucket start | | `24h` (default) | `30m` – `1h` | Daily consolidation, bulk S3 archiving | 24h 30m from bucket start | > **Note:** If `MATS_GRACE_PERIOD` is significantly larger than `MATS_BUCKET_DURATION` (e.g. `30m` grace on a `15m` bucket), buckets remain open in the hot buffer 2–3x longer than necessary, delaying cold tier upload. Aim for 10%–30% of the bucket duration (or 2m–5m for fast rotation) to absorb client retry jitter while archiving promptly. --- ## Riemann & Ingestion Setup ### In `riemann.config` ```clojure ;; For reliable TCP streaming with flow control: (def mats-sink (tcp-client :host "127.0.0.1" :port 5555)) ;; OR for zero-latency fire-and-forget UDP: (def mats-sink (udp-client :host "127.0.0.1" :port 5555)) (streams (where (not (state "expired")) mats-sink)) ``` ### In `collectd.conf` (Direct Push) ```apache LoadPlugin write_riemann Host "127.0.0.1" Port "5555" Protocol TCP Batch true StoreRates true ``` ### Ad-Hoc Ingestion via HTTP SSE (`mats-riemann-sse`) To stream telemetry from a running Riemann instance **without modifying `riemann.config`**: ```bash ./bin/mats-riemann-sse \ -sse-url "http://127.0.0.1:5556/index?query=true" \ -target-addr "127.0.0.1:5555" \ -protocol tcp ``` * **Zero Riemann Downtime:** Subscribes to Riemann's HTTP Server-Sent Events (SSE) stream. * **Server Isolation:** Enforces a bounded ring buffer (`-buffer-size 10000`). If `mats` is temporarily restarting or compacting, the proxy drops events rather than creating JVM memory backpressure on the upstream Riemann server. * **Micro-Batching:** Packs streamed events into framed Protobuf batches (`-batch-size 100 -batch-timeout 100ms`) over TCP. --- ## Query API & Visualizations ### 1. Embedded Web Dashboard Navigate to `http://localhost:8080/` in any browser to access the zero-dependency embedded monitoring dashboard. ### 2. Columnar JSON API ```bash curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000" ``` Returns parallel flat arrays for instant client-side rendering without parsing overhead: ```json [ { "metric_id": 1, "host": "nas-01", "service": "cpu.idle", "timestamps": [1725537600000, 1725537610000], "values": [94.2, 95.1] } ] ``` ### 3. Server-Side Downsampling & Aggregation (LTTB) When querying large time windows (e.g. days, months, or years with millions of points), `mats` automatically performs server-side downsampling using the **Largest Triangle Three Buckets (LTTB)** algorithm in $O(N)$ linear time. Unlike naive averaging which flattens out critical spikes and dips, LTTB preserves the visual shape, peaks, and troughs of the time-series while drastically shrinking payload sizes (e.g. reducing ~80 MB of raw data down to ~25 KB). #### Query Parameters * **`max_points`** (or **`points`**): Target number of data points per series (default: `1500`, configurable via `-query-max-points` or `MATS_QUERY_MAX_POINTS`). Set to `0` or pass `raw=1` to disable downsampling. * **`raw=1`**: Disables downsampling completely and returns full raw telemetry. * **`step`**: Group points into fixed time intervals (e.g., `10s`, `1m`, `1h`, or milliseconds). * **`method`** (or **`agg`**): Downsampling/aggregation method: * `lttb` (default): Largest Triangle Three Buckets; optimal for graphs and visual inspection. * `avg`: Mean value per bucket. * `min`: Minimum value per bucket. * `max`: Maximum value per bucket. * `sum`: Sum of values per bucket. * `count`: Point count per bucket. #### Examples ```bash # Default LTTB downsampling (up to 1500 points) curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000" # Target 300 points using LTTB curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000&max_points=300" # Aggregate by 5-minute maximums curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000&step=5m&method=max" # Bypass downsampling to retrieve full raw data curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000&raw=1" ``` Downsampling parameters also apply to `/api/v1/render`. By default, SVG charts downsample to $\max(\text{width} \times 2, 1000)$ vertices to prevent oversized SVGs and browser lag, while ASCII sparklines downsample to 200 points. #### In-Memory Response Caching & HTTP 304 Queries and renders are automatically cached in-memory with time-aligned quantization. Repeated requests or dashboard refreshes return `304 Not Modified` via standard `ETag` and `If-None-Match` HTTP headers. Cache keys incorporate downsampling parameters so raw and downsampled queries never collide. ### 4. Pure-Go Vector SVG Charts (Zero JS) ```bash curl -s "http://localhost:8080/api/v1/render?host=nas-01&service=cpu.idle&format=svg" > chart.svg ``` Embed directly in web pages, markdown documents, emails, or issue trackers. ### 5. Machine & LLM ASCII Summaries ```bash curl -s "http://localhost:8080/api/v1/render?host=nas-01&service=cpu.idle&format=ascii" ``` Output: ```text Series: nas-01 / cpu.idle Range: 2026-09-05T19:00:00Z to 2026-09-05T20:00:00Z (60 points) Metrics: Min: 91.20 | Max: 98.40 | Mean: 94.80 | P95: 97.60 | Latest: 95.10 Trend: [ ▂▃▄▅▆▇██▇▆▅▄▃▂ ] ``` --- ## Legacy Grafana Bridge (`mats-influx-proxy`) To connect a legacy Grafana instance (InfluxDB 1.x data source) without changing queries: ```bash ./bin/mats-influx-proxy -listen :8086 -backend http://127.0.0.1:8080 ``` In Grafana: * **Type:** InfluxDB * **URL:** `http://localhost:8086` * **Database:** `mats` The proxy translates InfluxQL `SELECT ... FROM ... WHERE ... GROUP BY time(...)` queries into native `mats` columnar queries and formats the response into legacy Influx 1.x JSON. --- ## Public Gateway Proxy (`mats-proxy`) To securely expose a public demonstration dashboard or status page from an edge device (e.g. Raspberry Pi Zero 2 in a DMZ, Cloudflare Tunnel, or VPS) without exposing private home telemetry, S3 cloud credentials, or host OS integrity: ```bash ./bin/mats-proxy \ -backend http://192.168.1.50:8080 \ -listen :8088 \ -allow-hosts "demo-node,weather-station" \ -allow-services "load.short,temperature,co2" \ -max-query-window 8760h ``` For full architectural details, see [ADR 0002: Architecture and Security Model of the Public Gateway Proxy](ADR-0002-mats-public-proxy.md). ### Security & Operational Guarantees: * **Stateless & Zero Cloud Credentials:** Possesses zero AWS/S3 access keys; communicates strictly via HTTP to the primary `mats` instance. * **Authoritative PII Gate:** Rejects queries for unapproved hosts or services with `403 Forbidden` and automatically rewrites wildcards (`.*`) to the public allow-list. * **Anti-DoS Modulus Quantization:** Public callers cannot force cache misses (`exact=true` and client `no-cache` headers are stripped). Timestamps strictly snap to uniform modulus intervals. * **Minimal Footprint:** Omits Parquet and SQLite dependencies, keeping memory consumption at **~10–15 MB RAM** (ideal for the 512 MB Raspberry Pi Zero 2). * **Multi-OS Sandboxing:** Enforces Linux Landlock and pure-Go Seccomp BPF filters (blocking `execve`, `ptrace`, and privilege escalation), with native `pledge`/`unveil` support on OpenBSD. * **Standalone Edge Hardening:** Active connection limiting (`netutil.LimitListener`), Slowloris timeouts, and in-memory per-IP token bucket rate limiting.