- Go 100%
Go Riemann Library #
A high-performance Go implementation of the Riemann distributed systems monitoring server. This package (go.iscode.ca/riemann) provides a lightweight, production-grade Go library for event monitoring, indexing, and stream processing.
It is designed for high-throughput stream processing, robust connection management, and low resource overhead—making it ideal for resource-constrained environments like edge/Raspberry Pi devices, as well as core infrastructure services.
Key Features #
- Multi-Protocol Ingestion:
- TCP (Protobuf): Connection-oriented Riemann event streaming.
- UDP (Protobuf): Fire-and-forget high-throughput event ingestion (defaulting to port
5555).
- Real-Time Stream Processing:
- Standard query evaluation and event indexing.
- Stream pipeline helpers:
Wherefiltering,Rollupaggregation, andRatecalculation.
- Modern Web Interfaces:
- WebSockets: Live subscriptions and query-matching streams.
- Server-Sent Events (SSE): Streaming event feeds.
- Multi-Port Listeners: Run WebSocket services and Server-Sent Events on separate ports to match Clojure configurations, or combined on a single interface.
- Robust Security:
- Native TLS/SSL Encryption (
ConfigureTLS) across all TCP and HTTP/WS/SSE listener endpoints.
- Native TLS/SSL Encryption (
- Dynamic Replication (Mirroring):
- Stream synchronization from remote Clojure or Go Riemann instances using WebSocket (
ws://,wss://) or SSE (http://,https://) protocols. - Connection Recovery & Deduplication: Automatic backoff reconnects and deep-field comparisons to prevent duplicate indexing.
- Stream synchronization from remote Clojure or Go Riemann instances using WebSocket (
- Self-Monitoring & Instrumentation:
- Real-time tracking of active TCP, WebSocket, and SSE connections.
- Stream latency calculation (including 99.9th percentile latencies).
Library Usage: Simple Riemann Service #
You can import go.iscode.ca/riemann to build and customize your own stream processing engine.
Below is a complete, runnable example demonstrating how to initialize the core engine, set up a stream pipeline, and query the event index:
package main
import (
"context"
"fmt"
"log"
"time"
"go.iscode.ca/riemann"
)
func main() {
// 1. Initialize the Core service aggregator
core := riemann.NewCore()
// 2. Define your stream pipeline destinations
opsAlerts := riemann.StreamFunc(func(e *riemann.Event) {
fmt.Printf("[ALERT] Service %q on %q entered state %q!\n", e.Service, e.Host, e.State)
})
indexStream := riemann.IndexStream(core.Index)
// 3. Compose the pipeline (equivalent to Clojure's nested stream macros)
// Here, we route critical disk events to opsAlerts, and index all events.
pipeline := riemann.StreamFunc(func(e *riemann.Event) {
// Filter rule (Clojure: (where (and (service "disk") (state "critical"))))
if e.Service == "disk" && e.State == "critical" {
opsAlerts.Send(e)
}
// Index rule
indexStream.Send(e)
})
// Register the pipeline with the core
core.Streams(pipeline)
// 4. Configure and bind TCP and UDP listeners
core.AddTCPServer("0.0.0.0:5555")
core.AddUDPServer("0.0.0.0:5555")
// 5. Configure modern SSE/WS monitoring endpoints
core.AddWebServer("0.0.0.0:5556") // WebSockets endpoint
core.AddSSEServer("0.0.0.0:5558") // Dedicated Server-Sent Events
// 6. Start all transport servers
if err := core.Start(); err != nil {
log.Fatalf("Failed to start Riemann servers: %v", err)
}
defer core.Stop()
log.Println("Riemann service running...")
// 7. Inject a test event into the pipeline
core.Send(&riemann.Event{
Host: "prod-db-1",
Service: "disk",
State: "critical",
Metric: 98.5,
Time: time.Now().Unix(),
TTL: 60,
})
// 8. Query the index for active metrics
events, err := core.Index.Search("service = \"disk\" and state = \"critical\"")
if err != nil {
log.Fatalf("Query search error: %v", err)
}
for _, ev := range events {
fmt.Printf("Query result: Found %s/%s in state %s (metric: %v)\n",
ev.Host, ev.Service, ev.State, ev.Metric)
}
}
Runnable Example Daemon: riemannd #
The library includes a simple standalone Riemann server daemon riemannd under cmd/riemannd/ that logs all incoming events and stores them in the index.
Building #
Compile the example daemon from the repository root:
go build -o riemannd ./cmd/riemannd
Running #
Start the daemon with default ports (TCP/UDP on 5555, WebSockets on 5556, and SSE on 5558):
./riemannd
Options #
You can configure the listening addresses using standard command-line flags:
-tcp-addr string
TCP listen address (default "0.0.0.0:5555")
-udp-addr string
UDP listen address (default "0.0.0.0:5555")
-ws-addr string
WebSocket listen address (default "0.0.0.0:5556")
-sse-addr string
SSE listen address (default "0.0.0.0:5558")