Metric and Time-Series Engine: A Minimal Archiving Time Series Storage Service
README.md

Go Reference

mats: Minimal Numeric Time-Series Engine #

mats is a dedicated, low-resource, high-throughput embedded time-series database engine written in pure Go (CGO_ENABLED=0). It is specifically engineered to ingest and store telemetry from Riemann monitoring, scaling seamlessly from low-power home nodes (Raspberry Pi/NAS) to large-scale deployments absorbing thousands of metrics per second.

For full architectural rationale, see ADR 0001: Architecture of the Embedded Numeric Time-Series Engine.


Key Architectural Principles #

  1. Dual-Plane Architecture:
    • Control Plane (Metadata): Pure-Go SQLite (modernc.org/sqlite) mapping string tags (host, service, tags) to immutable 64-bit integer metric_ids, backed by an in-memory bi-directional cache (568 ns/op fast path).
    • Data Plane (Telemetry): Strictly unboxed numeric primitives (metric_id int64, timestamp int64, value float64). No strings or arbitrary JSON in the telemetry stream.
  2. Tier 1 (Hot LSM Buffer): CockroachDB's pebble LSM engine using a composite 20-byte key: $$\text{Key} = [\text{BucketID (4B)}][\text{MetricID (8B)}][\text{Timestamp (8B)}]$$ Enables instant, zero-cost purges via a single db.DeleteRange([Bucket_N], [Bucket_N+1]) call without tombstone bloat.
  3. Tier 2 (Cold Archive): Immutable Apache Parquet files stored on S3-compatible cloud storage (AWS S3, Cloudflare R2, MinIO, Versity) or local disk. Compressed with Zstandard, Delta-binary packed timestamps, and Dictionary-encoded metric IDs.
  4. Crash-Safe Handover Protocol: Drains LSM to an ephemeral scratch file, uploads to S3, commits the manifest to SQLite, and purges the LSM range. Startup recovery is instantaneous ($< 200\text{ ms}$) and cleans dangling state.
  5. WORM Storage Security: Cold storage is strictly Write-Once-Read-Many. The engine never requires s3:DeleteObject permissions, preventing accidental or malicious loss of historical archives.
  6. Operating System Sandboxing: Enforces Linux Landlock filesystem containment and PR_SET_NO_NEW_PRIVS to lock down process privileges after startup.
  7. Drop & Count Policy: Drops late-arriving metrics outside the active bucket window to prevent backfilling DoS attacks.
  8. Decoupled Legacy Bridge: Provides an optional standalone InfluxDB 1.x /query proxy for drop-in compatibility with legacy Grafana instances.

Build & Test #

Requirements #

  • Go 1.22+ (verified on Go 1.27.1, ARM64 & AMD64)
  • Zero external C toolchains or libraries required (CGO_ENABLED=0)

Building #

make all

Produces:

  • bin/mats: The core daemon (ingestion, storage, hybrid query engine, and embedded dashboard).
  • bin/mats-proxy: Secure, sandboxed public gateway and dashboard proxy.
  • bin/mats-influx-proxy: Standalone InfluxDB 1.x /query translation bridge.
  • bin/mats-riemann-sse: Standalone Riemann HTTP SSE forwarding proxy.

Running Tests & Linter #

make test    # Run full unit test suite
make lint    # Run golangci-lint (0 warnings)
make bench   # Run zero-alloc benchmarks

Running the Engine #

# Run with default settings (Daily buckets, local disk storage, Landlock enabled)
./bin/mats -data-dir ./data -scratch-dir ./scratch

# Run with S3 Cold Storage & 1-Hour Buckets
./bin/mats \
  -data-dir /var/lib/mats/data \
  -scratch-dir /var/lib/mats/scratch \
  -bucket-duration 1h \
  -grace-period 15m \
  -s3-endpoint https://s3.us-west-2.amazonaws.com \
  -s3-bucket my-telemetry-archive \
  -s3-region us-west-2 \
  -s3-access-key $AWS_ACCESS_KEY_ID \
  -s3-secret-key $AWS_SECRET_ACCESS_KEY

Configuration (12-Factor Env Variables) #

All CLI flags can be set via environment variables:

  • MATS_DATA_DIR (default: ./data)
  • MATS_SCRATCH_DIR (default: ./scratch)
  • MATS_RIEMANN_TCP (default: :5555)
  • MATS_RIEMANN_UDP (default: :5555)
  • MATS_HTTP_ADDR (default: :8080)
  • MATS_BUCKET_DURATION (default: 24h)
  • MATS_GRACE_PERIOD (default: 30m)
  • MATS_NODE_ID (node identifier for multi-ingress storage key namespacing)
  • MATS_QUERY_CACHE (default: true)
  • MATS_QUERY_CACHE_SIZE (default: 512)
  • MATS_QUERY_MAX_POINTS (default: 1500, 0 to disable)
  • MATS_COMPACTION (default: true)
  • MATS_COMPACTION_WINDOW (default: 24h)
  • MATS_S3_ENDPOINT
  • MATS_S3_BUCKET
  • MATS_S3_REGION (default: us-east-1)
  • MATS_S3_ACCESS_KEY
  • MATS_S3_SECRET_KEY
  • MATS_S3_DISABLE_TLS (default: false)
  • MATS_ENABLE_LANDLOCK (default: true)

Bucket Duration & Grace Period Tuning #

MATS_BUCKET_DURATION sets the time span for each immutable cold Parquet archive file, while MATS_GRACE_PERIOD governs:

  1. Late-arrival tolerance: Incoming metrics with timestamp < (now - grace_period) are discarded to prevent backfilling DoS.
  2. Cold archiving delay: Closed buckets are retained in the hot Pebble buffer until bucket_end + grace_period before being sealed and uploaded to cold storage.

When configuring non-default bucket durations, pair them with a proportional grace period:

Bucket Duration Recommended Grace Period Typical Use Case Cold Tier Archival Delay
15m 2m – 5m Fast cold rotation, low memory footprint 17m – 20m from bucket start
1h 5m – 15m Hourly archiving, balanced buffer size 65m – 75m from bucket start
24h (default) 30m – 1h Daily consolidation, bulk S3 archiving 24h 30m from bucket start

Note: If MATS_GRACE_PERIOD is significantly larger than MATS_BUCKET_DURATION (e.g. 30m grace on a 15m bucket), buckets remain open in the hot buffer 2–3x longer than necessary, delaying cold tier upload. Aim for 10%–30% of the bucket duration (or 2m–5m for fast rotation) to absorb client retry jitter while archiving promptly.


Riemann & Ingestion Setup #

In riemann.config #

;; For reliable TCP streaming with flow control:
(def mats-sink (tcp-client :host "127.0.0.1" :port 5555))

;; OR for zero-latency fire-and-forget UDP:
(def mats-sink (udp-client :host "127.0.0.1" :port 5555))

(streams
  (where (not (state "expired"))
    mats-sink))

In collectd.conf (Direct Push) #

LoadPlugin write_riemann
<Plugin write_riemann>
    <Node "mats_engine">
        Host "127.0.0.1"
        Port "5555"
        Protocol TCP
        Batch true
        StoreRates true
    </Node>
</Plugin>

Ad-Hoc Ingestion via HTTP SSE (mats-riemann-sse) #

To stream telemetry from a running Riemann instance without modifying riemann.config:

./bin/mats-riemann-sse \
  -sse-url "http://127.0.0.1:5556/index?query=true" \
  -target-addr "127.0.0.1:5555" \
  -protocol tcp
  • Zero Riemann Downtime: Subscribes to Riemann's HTTP Server-Sent Events (SSE) stream.
  • Server Isolation: Enforces a bounded ring buffer (-buffer-size 10000). If mats is temporarily restarting or compacting, the proxy drops events rather than creating JVM memory backpressure on the upstream Riemann server.
  • Micro-Batching: Packs streamed events into framed Protobuf batches (-batch-size 100 -batch-timeout 100ms) over TCP.

Query API & Visualizations #

1. Embedded Web Dashboard #

Navigate to http://localhost:8080/ in any browser to access the zero-dependency embedded monitoring dashboard.

2. Columnar JSON API #

curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000"

Returns parallel flat arrays for instant client-side rendering without parsing overhead:

[
  {
    "metric_id": 1,
    "host": "nas-01",
    "service": "cpu.idle",
    "timestamps": [1725537600000, 1725537610000],
    "values": [94.2, 95.1]
  }
]

3. Server-Side Downsampling & Aggregation (LTTB) #

When querying large time windows (e.g. days, months, or years with millions of points), mats automatically performs server-side downsampling using the Largest Triangle Three Buckets (LTTB) algorithm in $O(N)$ linear time. Unlike naive averaging which flattens out critical spikes and dips, LTTB preserves the visual shape, peaks, and troughs of the time-series while drastically shrinking payload sizes (e.g. reducing ~80 MB of raw data down to ~25 KB).

Query Parameters #

  • max_points (or points): Target number of data points per series (default: 1500, configurable via -query-max-points or MATS_QUERY_MAX_POINTS). Set to 0 or pass raw=1 to disable downsampling.
  • raw=1: Disables downsampling completely and returns full raw telemetry.
  • step: Group points into fixed time intervals (e.g., 10s, 1m, 1h, or milliseconds).
  • method (or agg): Downsampling/aggregation method:
    • lttb (default): Largest Triangle Three Buckets; optimal for graphs and visual inspection.
    • avg: Mean value per bucket.
    • min: Minimum value per bucket.
    • max: Maximum value per bucket.
    • sum: Sum of values per bucket.
    • count: Point count per bucket.

Examples #

# Default LTTB downsampling (up to 1500 points)
curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000"

# Target 300 points using LTTB
curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000&max_points=300"

# Aggregate by 5-minute maximums
curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000&step=5m&method=max"

# Bypass downsampling to retrieve full raw data
curl "http://localhost:8080/api/v1/query?host=nas-01&service=cpu.idle&from=1725537600000&to=1725541200000&raw=1"

Downsampling parameters also apply to /api/v1/render. By default, SVG charts downsample to $\max(\text{width} \times 2, 1000)$ vertices to prevent oversized SVGs and browser lag, while ASCII sparklines downsample to 200 points.

In-Memory Response Caching & HTTP 304 #

Queries and renders are automatically cached in-memory with time-aligned quantization. Repeated requests or dashboard refreshes return 304 Not Modified via standard ETag and If-None-Match HTTP headers. Cache keys incorporate downsampling parameters so raw and downsampled queries never collide.

4. Pure-Go Vector SVG Charts (Zero JS) #

curl -s "http://localhost:8080/api/v1/render?host=nas-01&service=cpu.idle&format=svg" > chart.svg

Embed directly in web pages, markdown documents, emails, or issue trackers.

5. Machine & LLM ASCII Summaries #

curl -s "http://localhost:8080/api/v1/render?host=nas-01&service=cpu.idle&format=ascii"

Output:

Series:   nas-01 / cpu.idle
Range:    2026-09-05T19:00:00Z to 2026-09-05T20:00:00Z (60 points)
Metrics:  Min: 91.20 | Max: 98.40 | Mean: 94.80 | P95: 97.60 | Latest: 95.10
Trend:    [ ▂▃▄▅▆▇██▇▆▅▄▃▂ ]

Legacy Grafana Bridge (mats-influx-proxy) #

To connect a legacy Grafana instance (InfluxDB 1.x data source) without changing queries:

./bin/mats-influx-proxy -listen :8086 -backend http://127.0.0.1:8080

In Grafana:

  • Type: InfluxDB
  • URL: http://localhost:8086
  • Database: mats

The proxy translates InfluxQL SELECT ... FROM ... WHERE ... GROUP BY time(...) queries into native mats columnar queries and formats the response into legacy Influx 1.x JSON.


Public Gateway Proxy (mats-proxy) #

To securely expose a public demonstration dashboard or status page from an edge device (e.g. Raspberry Pi Zero 2 in a DMZ, Cloudflare Tunnel, or VPS) without exposing private home telemetry, S3 cloud credentials, or host OS integrity:

./bin/mats-proxy \
  -backend http://192.168.1.50:8080 \
  -listen :8088 \
  -allow-hosts "demo-node,weather-station" \
  -allow-services "load.short,temperature,co2" \
  -max-query-window 8760h

For full architectural details, see ADR 0002: Architecture and Security Model of the Public Gateway Proxy.

Security & Operational Guarantees: #

  • Stateless & Zero Cloud Credentials: Possesses zero AWS/S3 access keys; communicates strictly via HTTP to the primary mats instance.
  • Authoritative PII Gate: Rejects queries for unapproved hosts or services with 403 Forbidden and automatically rewrites wildcards (.*) to the public allow-list.
  • Anti-DoS Modulus Quantization: Public callers cannot force cache misses (exact=true and client no-cache headers are stripped). Timestamps strictly snap to uniform modulus intervals.
  • Minimal Footprint: Omits Parquet and SQLite dependencies, keeping memory consumption at ~10–15 MB RAM (ideal for the 512 MB Raspberry Pi Zero 2).
  • Multi-OS Sandboxing: Enforces Linux Landlock and pure-Go Seccomp BPF filters (blocking execve, ptrace, and privilege escalation), with native pledge/unveil support on OpenBSD.
  • Standalone Edge Hardening: Active connection limiting (netutil.LimitListener), Slowloris timeouts, and in-memory per-IP token bucket rate limiting.