The pack cache keeps expired packfiles in memory until size eviction pushes them out. Every push changes the refs digest in the cache key, so the clones that follow are always new keys and old entries are never looked up again. On an unconstrained host, knot's memory grows toward the pack cache cap.
What I saw #
Self-hosted knot, atcr.io/tangled.org/knot:2 (index digest sha256:9357ff769e2401a248046f21d8225721be39999f01172d5346e1939b72a66a3b, built 2026-10-30), Podman, no container memory limit, 7.7 GiB host without swap.
- RSS (podman stats) was steady at about 20 MB for two weeks with three small repos.
- After a ~50 MB repo moved onto the knot, memory rose in steps during push-heavy sessions and stayed at each level: 205 → 980 MB over one afternoon (514
git-upload-packrequests in that window, almost all CI clones), then → 1.39 GB the next day (346 requests). - Overnight, with only
com.atproto.sync.subscribeRepostraffic, memory stayed flat at 1.39 GB.
Controlled test after a restart, measured as anon in the container's memory.stat:
| Step | anon |
|---|---|
| baseline | 106.9 MB |
20× git ls-remote |
+0.4 MB |
20× identical git clone --depth 1 |
+17.7 MB (one cached pack, then cache hits) |
| 2 minutes later (TTL is 60 s) | −3.3 MB, the entry is still held |
Cause #
As far as I can tell from master (fdcc06c91; the cache code hasn't changed since 2026-10-29, so it matches the image):
knot2/crates/knot-pack/src/cache.rs:PackCachestores whole generated packfiles in anLru::by_weight_with_ttlkeyed byRequestKey(refs digest + request body), with a 60 s TTL by default.knot2/crates/knot-cache/src/lib.rs,Lru::get(line 224): an expired entry is removed only when that same key is read.Lru::insert(line 246): eviction happens only when the cache is overmax_weightormax_entries. Expired entries aren't pruned there.knot2/crates/knot-resource/src/mem.rs:pack_cache_bytesgives the cache 25% of the memory budget, capped atpack_cache.max_total_bytes(2 GiB by default). Without a cgroup limit the budget is MemTotal, so here it was about 1.9 GB.cache_shed_warrantedreclaims only when available memory drops below 10% of the budget, so on a host with headroom nothing triggers early.
After a push, CI clones produce new keys. Their entries outlive the TTL and stay resident until the cache reaches its weight cap or 1024 entries.
Workaround #
A container memory limit (512 MiB here). knot picks it up (resource governor initialized ... memory_bytes=536870912 memory_source=CgroupV2), and the pack cache drops to 128 MB. Setting KNOT_PACK_CACHE_MAX_TOTAL_BYTES or KNOT_MAX_MEMORY_BYTES should work too.
Possible fix #
Prune expired entries in Lru::insert before the size check. The insert path already holds the lock, and the pack cache has at most 1024 entries, so it can drop every node whose expires_at has passed. LRU order alone isn't enough, because get refreshes the tick but not expires_at. A periodic sweep from the existing allocator/maintenance loop would also cover idle periods.