services #
background workers managed by services.zig. each service is a struct that takes (allocator: Allocator, io: Io) at init and stores io as a field — methods use self.io for mutex lock/unlock, sleeps, timestamps, and uuid generation. see AGENTS.md "io discipline" for the broader pattern.
registry #
Services struct owns one instance of each service plus the DocketWorker that drives the loop services:
pub const Services = struct {
event_persister: EventPersister,
event_broadcaster: EventBroadcaster,
scheduler: Scheduler,
lease_cleanup: LeaseCleanup,
late_runs: LateRuns,
automations: Automations,
task_run_recorder: TaskRunRecorder,
docket_worker: DocketWorker,
pub fn init(allocator: Allocator, io: Io) Services { ... }
pub fn startAll(self: *Services) !void { ... }
pub fn stopAll(self: *Services) void { ... }
};
main.zig builds the Services struct once and holds a ?*Services for handlers that need to reach into a service (e.g., event_broadcaster.addSubscriber called from ws handlers via a free-fn shim).
startAll runs forward: every loop service registers + seeds its perpetual task on the shared docket_runtime.instance() BEFORE the docket worker starts (so registrations are visible when the worker first consumes the stream). stopAll runs in reverse: docket worker cancels first, then each service frees its task handle.
the two service shapes #
There are two flavors of background work in prefect-server, and they sit on different substrates:
- broker-consumer services:
event_persister,event_broadcaster,task_run_recorder,automationsreactive. Subscribe to a broker topic with a consumer group; horizontally scalable because the consumer group load-balances messages across replicas. - loop services:
scheduler,late_runs,lease_cleanup,automationsproactive. Used to be threads with sleep loops. Now docket-driven (tangled.org/zzstoatzz.io/docket) — each registers a perpetual task that fires on an interval, executed by the shareddocket_workerconsuming docket's stream. Horizontally scalable for the same reason: docket itself uses a consumer group internally.
docket runtime #
src/docket_runtime.zig initializes a module-level Docket singleton early in initInfra. Its URL comes from the broker dialect setting:
PREFECT_BROKER_BACKEND=memory→memory://prefect(embedded burner-redis engine)PREFECT_BROKER_BACKEND=redis→redis://{HOST}:{PORT}/{DB}fromPREFECT_REDIS_MESSAGING_*
Both backends run the same three Lua scripts for scheduling/promotion/ack — single source of truth for state transitions.
broker callbacks #
the broker's subscribe(topic, group, handler) callback signature is a static fn pointer with no user-data slot. for each service that consumes broker messages, we keep a module-level instance: ?*Service pointer set by start() and a dispatchMessage shim that reads it.
The docket task body uses the same pattern (automations.zig::proactiveTick reads instance to reach the shared cache). docket itself avoids this — its task bodies receive (io, *TaskContext, Params) and decode Params from JSON — but the same-module-pointer trick is the simplest way to share state across the broker callback and the docket task without threading user data through both APIs.
services #
event_persister #
subscribes to broker topic "events" with consumer group "event-persister" (durable; load-balances across replicas).
- batch size: 100 events
- flush interval: 1 second
- deduplication:
INSERT OR IGNOREby event id - retention: 7 days, trimmed hourly
- horizontally scalable: N replicas naturally share the consumer group
event_broadcaster #
subscribes to broker topic "events" with an ephemeral consumer group (unique per server instance — broadcast-{ms_timestamp}). fans out events to connected WS subscribers on /api/events/out.
- subscriber capacity: 256 connections per instance
- filter support: event prefix, resource_id (parsed with std.json)
- drop policy: non-blocking writes; slow clients get marked inactive
- horizontally scalable: each instance has its own ephemeral group, gets all events, fans out to its own subscribers
task_run_recorder #
subscribes to broker topic "events" with consumer group "task-run-recorder" (durable; load-balances across replicas). filters to events matching prefix prefect.task-run AND resource["prefect.orchestration"] == "client".
- phase 1 (current): subscribe + filter + log. no DB writes.
- phase 2 (next): dual-key upsert into
task_run+task_run_state. - horizontally scalable when phase 2 lands: consumer group distributes events; DB-side correctness comes from
WHERE state_timestamp < EXCLUDED.state_timestampon the upsert andON CONFLICT DO NOTHINGon the state row.
scheduler (docket) #
evaluates active deployment schedules (interval + cron) and creates flow runs in SCHEDULED state. Runs as scheduler.evaluate perpetual task.
- interval: 5s loop (configurable via
PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS) - max scheduled runs per schedule per tick: 50
- idempotent via
idempotency_key("scheduled {deployment_id} {schedule_id} {timestamp}") - horizontally scalable: docket consumer group distributes ticks across N docket_worker instances
late_runs (docket) #
marks flow runs as "Late" if they haven't started within PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS (default 15) past their scheduled start time. Runs as late_runs.mark perpetual task.
- interval: 5s loop (configurable via
PREFECT_API_SERVICES_LATE_RUNS_LOOP_SECONDS) - batch size: 100 runs per cycle
- horizontally scalable via docket
lease_cleanup (docket) #
expires stale concurrency leases. Runs as lease_cleanup.clean perpetual task.
- interval: 10s loop
- batch size: 100 leases per cycle
- race-safe cleanup via
deleteExpiredLease(only deletes if still expired — handles renewal race) - horizontally scalable via docket
automations #
evaluates automation triggers against incoming events. Two halves:
- reactive (event-driven): subscribes to broker topic "events" with consumer group "automations". evaluates each event against the cached automation triggers. horizontally scalable via consumer group.
- proactive (timer-driven): registered as
automations.proactivedocket perpetual task. Periodically refreshes cache and evaluates time-based triggers. horizontally scalable via docket. every 5s it evaluates each proactive trigger that is due (python'sproactive_evaluation), then deletes every bucket closed for more than 60s (sweep_closed_buckets). the pass runs behind the wall clock by the lag of the newest event the evaluator has been handed (python's events clock). that clock and each trigger's next-due time are per process, as in python: a replica that runs the pass without having consumed events uses the wall clock. - cache: 30s refresh interval, loaded from
db.automations.listEnabled
docket_worker #
runs docket.Worker.runForever via io.async. The returned Future(anyerror!void) lives on the service; stop() calls future.cancel(io), which propagates error.Canceled through consumeStream and lets the worker exit immediately (no thread.join, no atomic-flag polling).
- name:
docket-worker-1 - poll_block_ms: 500 (XREADGROUP block timeout, also memory-backend sleep cadence)
- scheduler_interval_micros: 500ms (how often promote_due fires)
- horizontally scalable: N docket_workers share the same consumer group
docket-workers; XREADGROUP delivers each message to exactly one worker
startup ordering #
1. initInfra()
1a. db.init() / migrations
1b. broker.initBroker() (memory or redis dialect)
1c. lease_storage.init()
1d. docket_runtime.init() — reads PREFECT_BROKER_BACKEND to pick URL
2. services = Services.init(allocator, io)
2a. services.startAll():
- event_persister, event_broadcaster (broker consumers)
- scheduler / lease_cleanup / late_runs / automations
(each registers a docket task + seeds the first tick)
- task_run_recorder (broker consumer)
- docket_worker (spawns Worker.runForever via io.async)
shutdown reverses: docket_worker.stop() first (cancel + await ≈ 1ms), then each service's stop() frees its handle, then broker / lease_storage / db / docket_runtime teardown.
shutdown latency #
before the docket migration: ~11 seconds (each loop service polled its sleep loop on a 1–5s cadence). after: ~1.3 seconds (docket worker cancel returns immediately, the four loop service .stop()s are free()s).
horizontal scaling #
all background work is now horizontally scalable:
| service | substrate | distribution |
|---|---|---|
| event_persister | broker consumer group | redis stream XREADGROUP |
| event_broadcaster | broker ephemeral group + ws fanout | per-instance subscribers |
| task_run_recorder | broker consumer group | redis stream XREADGROUP |
| scheduler | docket perpetual | docket worker pool |
| late_runs | docket perpetual | docket worker pool |
| lease_cleanup | docket perpetual | docket worker pool |
| automations | broker + docket | both |
| docket_worker | self | shared consumer group |