prefect server in zig

services #

background workers managed by services.zig. each service is a struct that takes (allocator: Allocator, io: Io) at init and stores io as a field — methods use self.io for mutex lock/unlock, sleeps, timestamps, and uuid generation. see AGENTS.md "io discipline" for the broader pattern.

registry #

Services struct owns one instance of each service plus the DocketWorker that drives the loop services:

pub const Services = struct {
    event_persister: EventPersister,
    event_broadcaster: EventBroadcaster,
    scheduler: Scheduler,
    lease_cleanup: LeaseCleanup,
    late_runs: LateRuns,
    automations: Automations,
    task_run_recorder: TaskRunRecorder,
    docket_worker: DocketWorker,

    pub fn init(allocator: Allocator, io: Io) Services { ... }
    pub fn startAll(self: *Services) !void { ... }
    pub fn stopAll(self: *Services) void { ... }
};

main.zig builds the Services struct once and holds a ?*Services for handlers that need to reach into a service (e.g., event_broadcaster.addSubscriber called from ws handlers via a free-fn shim).

startAll runs forward: every loop service registers + seeds its perpetual task on the shared docket_runtime.instance() BEFORE the docket worker starts (so registrations are visible when the worker first consumes the stream). stopAll runs in reverse: docket worker cancels first, then each service frees its task handle.

the two service shapes #

There are two flavors of background work in prefect-server, and they sit on different substrates:

  1. broker-consumer services: event_persister, event_broadcaster, task_run_recorder, automations reactive. Subscribe to a broker topic with a consumer group; horizontally scalable because the consumer group load-balances messages across replicas.
  2. loop services: scheduler, late_runs, lease_cleanup, automations proactive. Used to be threads with sleep loops. Now docket-driven (tangled.org/zzstoatzz.io/docket) — each registers a perpetual task that fires on an interval, executed by the shared docket_worker consuming docket's stream. Horizontally scalable for the same reason: docket itself uses a consumer group internally.

docket runtime #

src/docket_runtime.zig initializes a module-level Docket singleton early in initInfra. Its URL comes from the broker dialect setting:

  • PREFECT_BROKER_BACKEND=memory → memory://prefect (embedded burner-redis engine)
  • PREFECT_BROKER_BACKEND=redis → redis://{HOST}:{PORT}/{DB} from PREFECT_REDIS_MESSAGING_*

Both backends run the same three Lua scripts for scheduling/promotion/ack — single source of truth for state transitions.

broker callbacks #

the broker's subscribe(topic, group, handler) callback signature is a static fn pointer with no user-data slot. for each service that consumes broker messages, we keep a module-level instance: ?*Service pointer set by start() and a dispatchMessage shim that reads it.

The docket task body uses the same pattern (automations.zig::proactiveTick reads instance to reach the shared cache). docket itself avoids this — its task bodies receive (io, *TaskContext, Params) and decode Params from JSON — but the same-module-pointer trick is the simplest way to share state across the broker callback and the docket task without threading user data through both APIs.

services #

event_persister #

subscribes to broker topic "events" with consumer group "event-persister" (durable; load-balances across replicas).

  • batch size: 100 events
  • flush interval: 1 second
  • deduplication: INSERT OR IGNORE by event id
  • retention: 7 days, trimmed hourly
  • horizontally scalable: N replicas naturally share the consumer group

event_broadcaster #

subscribes to broker topic "events" with an ephemeral consumer group (unique per server instance — broadcast-{ms_timestamp}). fans out events to connected WS subscribers on /api/events/out.

  • subscriber capacity: 256 connections per instance
  • filter support: event prefix, resource_id (parsed with std.json)
  • drop policy: non-blocking writes; slow clients get marked inactive
  • horizontally scalable: each instance has its own ephemeral group, gets all events, fans out to its own subscribers

task_run_recorder #

subscribes to broker topic "events" with consumer group "task-run-recorder" (durable; load-balances across replicas). filters to events matching prefix prefect.task-run AND resource["prefect.orchestration"] == "client".

  • phase 1 (current): subscribe + filter + log. no DB writes.
  • phase 2 (next): dual-key upsert into task_run + task_run_state.
  • horizontally scalable when phase 2 lands: consumer group distributes events; DB-side correctness comes from WHERE state_timestamp < EXCLUDED.state_timestamp on the upsert and ON CONFLICT DO NOTHING on the state row.

scheduler (docket) #

evaluates active deployment schedules (interval + cron) and creates flow runs in SCHEDULED state. Runs as scheduler.evaluate perpetual task.

  • interval: 5s loop (configurable via PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS)
  • max scheduled runs per schedule per tick: 50
  • idempotent via idempotency_key ("scheduled {deployment_id} {schedule_id} {timestamp}")
  • horizontally scalable: docket consumer group distributes ticks across N docket_worker instances

late_runs (docket) #

marks flow runs as "Late" if they haven't started within PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS (default 15) past their scheduled start time. Runs as late_runs.mark perpetual task.

  • interval: 5s loop (configurable via PREFECT_API_SERVICES_LATE_RUNS_LOOP_SECONDS)
  • batch size: 100 runs per cycle
  • horizontally scalable via docket

lease_cleanup (docket) #

expires stale concurrency leases. Runs as lease_cleanup.clean perpetual task.

  • interval: 10s loop
  • batch size: 100 leases per cycle
  • race-safe cleanup via deleteExpiredLease (only deletes if still expired — handles renewal race)
  • horizontally scalable via docket

automations #

evaluates automation triggers against incoming events. Two halves:

  • reactive (event-driven): subscribes to broker topic "events" with consumer group "automations". evaluates each event against the cached automation triggers. horizontally scalable via consumer group.
  • proactive (timer-driven): registered as automations.proactive docket perpetual task. Periodically refreshes cache and evaluates time-based triggers. horizontally scalable via docket. every 5s it evaluates each proactive trigger that is due (python's proactive_evaluation), then deletes every bucket closed for more than 60s (sweep_closed_buckets). the pass runs behind the wall clock by the lag of the newest event the evaluator has been handed (python's events clock). that clock and each trigger's next-due time are per process, as in python: a replica that runs the pass without having consumed events uses the wall clock.
  • cache: 30s refresh interval, loaded from db.automations.listEnabled

docket_worker #

runs docket.Worker.runForever via io.async. The returned Future(anyerror!void) lives on the service; stop() calls future.cancel(io), which propagates error.Canceled through consumeStream and lets the worker exit immediately (no thread.join, no atomic-flag polling).

  • name: docket-worker-1
  • poll_block_ms: 500 (XREADGROUP block timeout, also memory-backend sleep cadence)
  • scheduler_interval_micros: 500ms (how often promote_due fires)
  • horizontally scalable: N docket_workers share the same consumer group docket-workers; XREADGROUP delivers each message to exactly one worker

startup ordering #

1. initInfra()
   1a. db.init() / migrations
   1b. broker.initBroker() (memory or redis dialect)
   1c. lease_storage.init()
   1d. docket_runtime.init() — reads PREFECT_BROKER_BACKEND to pick URL

2. services = Services.init(allocator, io)
   2a. services.startAll():
     - event_persister, event_broadcaster (broker consumers)
     - scheduler / lease_cleanup / late_runs / automations
       (each registers a docket task + seeds the first tick)
     - task_run_recorder (broker consumer)
     - docket_worker (spawns Worker.runForever via io.async)

shutdown reverses: docket_worker.stop() first (cancel + await ≈ 1ms), then each service's stop() frees its handle, then broker / lease_storage / db / docket_runtime teardown.

shutdown latency #

before the docket migration: ~11 seconds (each loop service polled its sleep loop on a 1–5s cadence). after: ~1.3 seconds (docket worker cancel returns immediately, the four loop service .stop()s are free()s).

horizontal scaling #

all background work is now horizontally scalable:

service substrate distribution
event_persister broker consumer group redis stream XREADGROUP
event_broadcaster broker ephemeral group + ws fanout per-instance subscribers
task_run_recorder broker consumer group redis stream XREADGROUP
scheduler docket perpetual docket worker pool
late_runs docket perpetual docket worker pool
lease_cleanup docket perpetual docket worker pool
automations broker + docket both
docket_worker self shared consumer group