limit in crates/vibescrobble-index/src/service.rs answers 503 with Retry-After: 1 the moment 64 requests are being answered. Nothing waits. /health, /avatars and /banners take the same slots as /view and /scrobbles.
https://tangled.org/permadeath.com/vibescrobble.com/pulls/286 holds each client to its share of the slots. Within that share, a client's extra requests wait, but they queue only behind that client's own requests. Many clients together still meet the instant 503. On loopback with 100 concurrent /view readers spread over four processes, /view drew these 503s in a 20 s round:
| stack | main | with #285 |
|---|---|---|
| 1x (290 KB /view) | 2.5% | 0 to 12 requests |
| 10x (2.8 MB /view) | 55% | 0.1% |
Most of main's 503s were readers that a slow /view serialization had piled up behind the view lock, and #285 removes that. What is left is the cliff itself. A burst past 64 requests, from enough clients or from a slow upstream behind the picture routes, is refused, and a health check that lands then reads as the service being down.
Options:
- Wait a bounded time for a shared slot, say 1 s, before refusing.
- Give
/healthand pictures their own budget, or exempt/health.
Filed from https://tangled.org/permadeath.com/vibescrobble.com/pulls/286.
Fixed on main by https://tangled.org/permadeath.com/vibescrobble.com/pulls/325: /health is off the request cap, and a request past it waits up to a second (SLOT_WAIT) for a slot.