Go port of clojure riemann
README.md

Go Riemann Library #

A high-performance Go implementation of the Riemann distributed systems monitoring server. This package (go.iscode.ca/riemann) provides a lightweight, production-grade Go library for event monitoring, indexing, and stream processing.

It is designed for high-throughput stream processing, robust connection management, and low resource overhead—making it ideal for resource-constrained environments like edge/Raspberry Pi devices, as well as core infrastructure services.


Key Features #

  • Multi-Protocol Ingestion:
    • TCP (Protobuf): Connection-oriented Riemann event streaming.
    • UDP (Protobuf): Fire-and-forget high-throughput event ingestion (defaulting to port 5555).
  • Real-Time Stream Processing:
    • Standard query evaluation and event indexing.
    • Stream pipeline helpers: Where filtering, Rollup aggregation, and Rate calculation.
  • Modern Web Interfaces:
    • WebSockets: Live subscriptions and query-matching streams.
    • Server-Sent Events (SSE): Streaming event feeds.
    • Multi-Port Listeners: Run WebSocket services and Server-Sent Events on separate ports to match Clojure configurations, or combined on a single interface.
  • Robust Security:
    • Native TLS/SSL Encryption (ConfigureTLS) across all TCP and HTTP/WS/SSE listener endpoints.
  • Dynamic Replication (Mirroring):
    • Stream synchronization from remote Clojure or Go Riemann instances using WebSocket (ws://, wss://) or SSE (http://, https://) protocols.
    • Connection Recovery & Deduplication: Automatic backoff reconnects and deep-field comparisons to prevent duplicate indexing.
  • Self-Monitoring & Instrumentation:
    • Real-time tracking of active TCP, WebSocket, and SSE connections.
    • Stream latency calculation (including 99.9th percentile latencies).

Library Usage: Simple Riemann Service #

You can import go.iscode.ca/riemann to build and customize your own stream processing engine.

Below is a complete, runnable example demonstrating how to initialize the core engine, set up a stream pipeline, and query the event index:

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"go.iscode.ca/riemann"
)

func main() {
	// 1. Initialize the Core service aggregator
	core := riemann.NewCore()

	// 2. Define your stream pipeline destinations
	opsAlerts := riemann.StreamFunc(func(e *riemann.Event) {
		fmt.Printf("[ALERT] Service %q on %q entered state %q!\n", e.Service, e.Host, e.State)
	})
	indexStream := riemann.IndexStream(core.Index)

	// 3. Compose the pipeline (equivalent to Clojure's nested stream macros)
	// Here, we route critical disk events to opsAlerts, and index all events.
	pipeline := riemann.StreamFunc(func(e *riemann.Event) {
		// Filter rule (Clojure: (where (and (service "disk") (state "critical"))))
		if e.Service == "disk" && e.State == "critical" {
			opsAlerts.Send(e)
		}
		// Index rule
		indexStream.Send(e)
	})

	// Register the pipeline with the core
	core.Streams(pipeline)

	// 4. Configure and bind TCP and UDP listeners
	core.AddTCPServer("0.0.0.0:5555")
	core.AddUDPServer("0.0.0.0:5555")

	// 5. Configure modern SSE/WS monitoring endpoints
	core.AddWebServer("0.0.0.0:5556") // WebSockets endpoint
	core.AddSSEServer("0.0.0.0:5558") // Dedicated Server-Sent Events

	// 6. Start all transport servers
	if err := core.Start(); err != nil {
		log.Fatalf("Failed to start Riemann servers: %v", err)
	}
	defer core.Stop()

	log.Println("Riemann service running...")

	// 7. Inject a test event into the pipeline
	core.Send(&riemann.Event{
		Host:    "prod-db-1",
		Service: "disk",
		State:   "critical",
		Metric:  98.5,
		Time:    time.Now().Unix(),
		TTL:     60,
	})

	// 8. Query the index for active metrics
	events, err := core.Index.Search("service = \"disk\" and state = \"critical\"")
	if err != nil {
		log.Fatalf("Query search error: %v", err)
	}

	for _, ev := range events {
		fmt.Printf("Query result: Found %s/%s in state %s (metric: %v)\n",
			ev.Host, ev.Service, ev.State, ev.Metric)
	}
}

Runnable Example Daemon: riemannd #

The library includes a simple standalone Riemann server daemon riemannd under cmd/riemannd/ that logs all incoming events and stores them in the index.

Building #

Compile the example daemon from the repository root:

go build -o riemannd ./cmd/riemannd

Running #

Start the daemon with default ports (TCP/UDP on 5555, WebSockets on 5556, and SSE on 5558):

./riemannd

Options #

You can configure the listening addresses using standard command-line flags:

  -tcp-addr string
    	TCP listen address (default "0.0.0.0:5555")
  -udp-addr string
    	UDP listen address (default "0.0.0.0:5555")
  -ws-addr string
    	WebSocket listen address (default "0.0.0.0:5556")
  -sse-addr string
    	SSE listen address (default "0.0.0.0:5558")