Monorepo for Tangled tangled.org

knot2/knot-cache: expired pack cache entries are only dropped on a same-key lookup #4790

Open willwu.tngl.sh
Open opened by willwu.tngl.sh
assigned to oyster.cafe

The pack cache keeps expired packfiles in memory until size eviction pushes them out. Every push changes the refs digest in the cache key, so the clones that follow are always new keys and old entries are never looked up again. On an unconstrained host, knot's memory grows toward the pack cache cap.

What I saw #

Self-hosted knot, atcr.io/tangled.org/knot:2 (index digest sha256:9357ff769e2401a248046f21d8225721be39999f01172d5346e1939b72a66a3b, built 2026-10-30), Podman, no container memory limit, 7.7 GiB host without swap.

  • RSS (podman stats) was steady at about 20 MB for two weeks with three small repos.
  • After a ~50 MB repo moved onto the knot, memory rose in steps during push-heavy sessions and stayed at each level: 205 → 980 MB over one afternoon (514 git-upload-pack requests in that window, almost all CI clones), then → 1.39 GB the next day (346 requests).
  • Overnight, with only com.atproto.sync.subscribeRepos traffic, memory stayed flat at 1.39 GB.

Controlled test after a restart, measured as anon in the container's memory.stat:

Step anon
baseline 106.9 MB
20× git ls-remote +0.4 MB
20× identical git clone --depth 1 +17.7 MB (one cached pack, then cache hits)
2 minutes later (TTL is 60 s) −3.3 MB, the entry is still held

Cause #

As far as I can tell from master (fdcc06c91; the cache code hasn't changed since 2026-10-29, so it matches the image):

  • knot2/crates/knot-pack/src/cache.rs: PackCache stores whole generated packfiles in an Lru::by_weight_with_ttl keyed by RequestKey (refs digest + request body), with a 60 s TTL by default.
  • knot2/crates/knot-cache/src/lib.rs, Lru::get (line 224): an expired entry is removed only when that same key is read.
  • Lru::insert (line 246): eviction happens only when the cache is over max_weight or max_entries. Expired entries aren't pruned there.
  • knot2/crates/knot-resource/src/mem.rs: pack_cache_bytes gives the cache 25% of the memory budget, capped at pack_cache.max_total_bytes (2 GiB by default). Without a cgroup limit the budget is MemTotal, so here it was about 1.9 GB.
  • cache_shed_warranted reclaims only when available memory drops below 10% of the budget, so on a host with headroom nothing triggers early.

After a push, CI clones produce new keys. Their entries outlive the TTL and stay resident until the cache reaches its weight cap or 1024 entries.

Workaround #

A container memory limit (512 MiB here). knot picks it up (resource governor initialized ... memory_bytes=536870912 memory_source=CgroupV2), and the pack cache drops to 128 MB. Setting KNOT_PACK_CACHE_MAX_TOTAL_BYTES or KNOT_MAX_MEMORY_BYTES should work too.

Possible fix #

Prune expired entries in Lru::insert before the size check. The insert path already holds the lock, and the pack cache has at most 1024 entries, so it can drop every node whose expires_at has passed. LRU order alone isn't enough, because get refreshes the tick but not expires_at. A periodic sweep from the existing allocator/maintenance loop would also cover idle periods.

Sign up or Sign in to add to the discussion.