Monorepo for Tangled tangled.org

knot: requestCrawl is fired immediately and without retry, so knots behind readiness-gating LBs are never crawled #4757

Open opened by fredrik-stock.at.eltele.no
assigned to oyster.cafe

The crawl loop is kicked off before knot_edge::serve(...) is even started: call location

It is also never retried.

So behind a reverse proxy that waits for the service to be ready you end up with a chain of error 400, and no crawl ever (unless you kick it off yourself manually).

Suggested fixes: configurable delay, or retry logic? the crawl request should probably have a default delay anyway, even if not adding a configurable one.

this probably explains some things about the manual crawls we've done with people, thanks :p

Fixed in 54936185 . Need to cut a small release some time soon

@oyster.cafe : The issue is kind of sort of fixed in v1.20.1, but mostly by happenstance, and not in a predictable way.

In the "knot service up, load balancer hasn't picked it up yet" window, the knot's public url will (at least with Traefik) respond with a 503 when the relay comes crawling, which materializes as this in the requestCrawl call: 400 {"error":"HostNotFound","message":"host server unreachable: server is not a PDS: Get [...]" }.

That hits the 4xx && !429 branch, "Hint taken, giving up".

Log:

2026-11-16T22:14:38.623329Z  INFO knot_server: listening on http://[::]:5555 and ssh://[::]:2222
2026-11-16T22:14:38.623342Z  INFO knot_server: import worker running
2026-11-16T22:14:38.960698Z  INFO knot_server: requested crawl crawler=https://index.tangled.network/xrpc/com.atproto.sync.requestCrawl
2026-11-16T22:14:39.231439Z  INFO knot_server: requested crawl crawler=https://atproto.africa/xrpc/com.atproto.sync.requestCrawl
2026-11-16T22:14:39.325783Z ERROR knot_server: requestCrawl got rejected! Hint taken, giving up on this specific endpoint crawler=https://relay.upcloud.world/xrpc/com.atproto.sync.requestCrawl status=400 Bad Request
2026-11-16T22:14:39.567733Z ERROR knot_server: requestCrawl got rejected! Hint taken, giving up on this specific endpoint crawler=https://europe.firehose.network/xrpc/com.atproto.sync.requestCrawl status=400 Bad Request
2026-11-16T22:14:39.616398Z ERROR knot_server: requestCrawl got rejected! Hint taken, giving up on this specific endpoint crawler=https://relay3.fr.hose.cam/xrpc/com.atproto.sync.requestCrawl status=400 Bad Request
2026-11-16T22:14:40.237658Z ERROR knot_server: requestCrawl got rejected! Hint taken, giving up on this specific endpoint crawler=https://northamerica.firehose.network/xrpc/com.atproto.sync.requestCrawl status=400 Bad Request
2026-11-16T22:14:41.025833Z ERROR knot_server: requestCrawl got rejected! Hint taken, giving up on this specific endpoint crawler=https://asia.firehose.network/xrpc/com.atproto.sync.requestCrawl status=400 Bad Request
2026-11-16T22:14:42.692930Z  WARN knot_server: requestCrawl has failed crawler=https://relay.fire.hose.cam/xrpc/com.atproto.sync.requestCrawl error=error sending request for url (https://relay.fire.hose.cam/xrpc/com.atproto.sync.requestCrawl)
2026-11-16T22:14:44.410663Z ERROR knot_server: requestCrawl got rejected! Hint taken, giving up on this specific endpoint crawler=https://bsky.network/xrpc/com.atproto.sync.requestCrawl status=400 Bad Request
2026-11-16T22:15:13.519061Z  INFO knot_server: requested crawl crawler=https://relay.fire.hose.cam/xrpc/com.atproto.sync.requestCrawl

relay.fire.hose.cam gave some other error at request time, so it got retried.

atproto.africa seems to always succeed, and I'm seeing index.tangled.network starting to succeed too. But they succeed before the knot is even online in the LB, so either they're enqueueing the crawl, have a cache or something else ("oh yea 200 fo sho I'll crawl you right away" goes to bed).

I'd reckon the next simple fix without any external changes would be to put HostNotFound in the 429 basket for retry, in lieu of building something robust/configurable?

faaa i was hoping the KISS way would work nicely, my bad

maybe it is worth doing the robust way then

Sign up or Sign in to add to the discussion.