Git backed by object storage because you can't stop me

feat(s3fs): cache directory listings and object metadata via groupcache

Eliminate redundant HeadObject and ListObjectsV2 round-trips on repeated Stat/Open operations against the same S3 prefixes. Implemented via two groupcache groups with window-encoded TTL and per-prefix local generation for precise in-process invalidation. When a path's parent folder is first listed, the cache populates both: - childEntry metadata (name, kind, size, mtime) for the entire folder - Background prefetch of HeadObject for each file via detached goroutines with semaphore bounding (maxPrefetchInFlight=64, configurable) Stat/Open of a cached folder's absent child answers with zero round-trips (negative hit from the listing). A present file skips HeadObject via the prewarmed head cache and only GetObject-s the body. Positive Stat/ReadDir results are never stale (immutable S3 content); cross-process and post-restart staleness is bounded by the TTL window (-s3-cache-ttl, default 60s; <=0 disables caching entirely). Local writes (Create, Remove, Rename, MkdirAll) bump the parent prefix's generation on success, moving the cache key so the next read re-lists and sees the change immediately (read-after-write correctness). When -groupcache-self and -groupcache-peers are configured, listings are shared across processes via consistent-hash peer pooling. Changes: - New: internal/s3fs/listingcache.go (ListingCache type, groupcache groups, warmer goroutine for refresh at idle TTL boundaries) - New: internal/s3fs/listingcache_test.go (comprehensive table-driven tests with counting stub client, head prefetch tests, window TTL, warmer, chroot integration) - Modified: internal/s3fs/{filesystem,basic,file,dir,chroot}.go to wire cache through Stat/Open/ReadDir/Remove/Rename/MkdirAll with invalidation on writes - Modified: internal/metrics/metrics.go to export groupcache stats via objgit_s3_listing_cache_* collectors - Modified: cmd/objgitd/main.go to add -s3-cache-ttl, -s3-cache-refresh, -s3-cache-idle, -s3-cache-size, -groupcache-self, -groupcache-peers, -groupcache-bind flags; integrate cache into S3FS and errgroup Assisted-by: Claude Opus 4.8 via Claude Code Signed-off-by: Xe Iaso <me@xeiaso.net>


Author xeiaso.net
Date Commit Parent 82da7452