A diet Tap option for handling atproto sync

Add batched webhook delivery #22

Merged socialde.pt
targeting tab from deleted fork:feature/batched-webhook
Merged opened by and assigned to socialde.pt
targeting tab from deleted fork:feature/batched-webhook

Webhook mode posts one request per event, which is a lot of requests during a backfill. This adds an opt-in batched mode:

Setting Default
TAB_WEBHOOK_BATCH_SIZE 0 Events per request. 0 keeps today's one-per-request behavior.
TAB_WEBHOOK_BATCH_FLUSH 500ms Longest a partial batch waits before it is sent.
TAB_WEBHOOK_BATCH_PARALLELISM 1 Batches in flight at once.

A batch is posted as {"events": [...]}, each element exactly as a single-event webhook would receive it, in event ID order.

Delivery and ordering #

  • A 2xx response acks every event in the batch. Any other response retries the whole batch with backoff, so receivers should treat events as at-least-once, as they would with retries today.
  • Ordering is unchanged. The per-DID workers still decide when an event may be sent, so a live event is only handed over once everything before it for that repo is acked, and nothing after it is handed over until its batch is acked.
  • TAB_WEBHOOK_BATCH_PARALLELISM defaults to 1 so the receiver sets the pace. While a batch is in flight, new events queue, and once the queue is full the outbox waits, the same way the websocket channel pushes back.
  • A tab_webhook_batch_size histogram records the size of each delivered batch.

Testing #

  • TestWebhook_BatchedDelivery: events arrive in fewer requests than events, none over the batch size, in ID order within each, each delivered and acked exactly once.
  • TestWebhook_BatchingKeepsLiveEventsInOrder: live events come after earlier historical ones, each in its own batch.
  • TestWebhook_FailedBatchIsRetriedWhole: a failed batch is resent unchanged, and its events are acked once.
  • TestWebhook_BatchingIsOffByDefault.
  • go test ./cmd/tab/ and go vet ./cmd/tab/ pass, and the new code is clean under -race. Note that TestWebhook_BasicDelivery already reports a race under -race on the base branch, in the test server's start and shutdown, unrelated to this change.
  • Run against real repos: three repos holding 11,248 records in the filtered collections were backfilled through a batch size of 100. The receiver got all 11,251 events (records plus identity events) in 114 requests instead of 11,251, each in ID order, and the outbox drained.
Commits 1

Add batched webhook delivery