I've accumulated a few cases where I think log messages could be better categorized into different log levels.
- Chain breaks during backfill/ingestion (i.e. the "chain break detected, triggering backfill" message): This is currently logged as a warning, but given that in my understanding chain breaks are really just regular behavior while syncing, I would argue that log should instead be a debug message. Because it's a warning, it's frequently logged while backfilling and pretty much spams the log output, which is a little inconvenient.
- Startup/database recovery ("recovering LSM-tree" messages): When starting up Hydrant with an existing database on disk, there is not really any info-level logging that the database is still being loaded/processed. Instead, there the occasional internal "recovering LSM-tree" messages which don't provide a clear look at the recovery progress. I would argue that those recovery messages should be considered debug-level, and that some info-level logging when database recovery begins and ends would be helpful.
- Backfill queue progress: The crawler does log "throttled: pending queue full" every 60s while idle, but I think that only reflects whether new repos are being discovered and not whether the backfill worker is actually draining the queue. Periodic backfill queue/worker progress summaries would be great to distinguish normal behavior from a stalled backfill worker or something.
1 & 2 makes sense, i think i had 1 as warn because i was debugging things... for 3 we already do have events per sec is that not enough? or you want to distinguish from firehose vs backfill events?