srcball is a project that takes a well-known url and turns it into a oci image
README.md

srcball #

Srcball is a Rust service and client library for turning source repositories into cached OCI image-layout artifacts. It is intended as a foundation for language-specific source proxies: Go-module-style proxying, CUE-style OCI modules, private forge mirrors, or any registry that wants source repositories to remain the package authority.

The service is storage-backed. The current implementation ships a filesystem storage backend for normal volumes; the storage trait is intentionally small so an S3 backend can be added without changing resolver, server, or client code.

What works now #

  • srcball serve runs an HTTP service with notmad lifecycle management.
  • GET /packages/fetch?package=<repo>&ref=<ref> returns an OCI image-layout tar (application/vnd.oci.image.layout.v1.tar) with the source tree as an OCI layer.
  • GET /packages/metadata?package=<repo>&ref=<ref> returns artifact metadata as JSON.
  • OCI Distribution-compatible pull endpoints are exposed under /v2:
    • GET /v2/
    • GET /v2/<package>/manifests/<ref>
    • GET /v2/<package>/blobs/<digest>
  • Missing immutable refs are resolved from Git, packaged as OCI artifacts, and stored in the configured volume.
  • Filesystem storage also materializes source files under state/artifacts/modules, similar to Go's module cache.
  • HEAD or latest requests resolve to Go-style pseudo versions: v0.0.0-yyyymmddhhmmss-<12-char-sha>.
  • noretry wraps transient remote Git operations.
  • nodrift runs recurring service maintenance without timer drift.
  • Filesystem storage tracks access timestamps and can evict least-recently-used OCI artifacts with srcball serve --max-cache-bytes <bytes>.
  • srcball package --url <repo> [--reference <ref>] [--output artifact.oci.tar] fetches through the same service core without HTTP.

Run locally #

docker compose up --build

By default compose bind-mounts the local directory ./.srcball into the container. Override the host-side location with:

SRCBALL_STATE_DIR=./.srcball docker compose up --build

Compose runs the container as ${SRCBALL_UID:-1000}:${SRCBALL_GID:-1000} so the bind-mounted .srcball directory stays writable by the local user. Override these if your local UID/GID differ:

SRCBALL_UID=$(id -u) SRCBALL_GID=$(id -g) docker compose up --build

When running the binary directly, use the same local state directory with:

SRCBALL_ROOT=.srcball cargo run -p srcball -- serve

The registry listens on http://localhost:3000.

Health check:

curl -fsS http://localhost:3000/healthz

Fetch an OCI layout tar directly:

curl -fL 'http://localhost:3000/packages/fetch?package=github.com/cue-lang/cue&ref=v0.15.0' \
  -o cue.oci.tar

Fetch only a subdirectory from a repository with //sub/path:

curl -fL 'http://localhost:3000/packages/fetch?package=github.com/kjuulh/forest//apps/forest&ref=main' \
  -o forest-app.oci.tar

Use the OCI registry surface:

curl -fsS http://localhost:3000/v2/
curl -fsS http://localhost:3000/v2/github.com/cue-lang/cue/manifests/v0.15.0

For private repositories, use Git transport credentials already available to the service process: SSH agent/keys, git@..., ssh://..., HTTPS credential helpers, or mounted Git config.

Local state layout #

The default filesystem backend keeps both OCI artifacts and an unpacked source cache. With the local defaults above, this lives in:

.srcball/
  artifacts/
    blobs/fa/bc/<rest>.tar                 # cached OCI image-layout tar files
    metadata/fa/bc/<rest>.json             # srcball metadata by package/version
    access/fa/bc/<rest>.atime              # LRU access timestamps
    refs/fa/bc/<rest>.json                 # short-TTL HEAD/latest resolution cache
    oci/blobs/sha256/fa/bc/<rest>          # digest-addressed blobs served by /v2/.../blobs/...
    modules/                               # unpacked source trees
  vcs/fa/bc/<rest>/                        # bare Git repositories

Normal module paths are splatted in a Go-module-cache-like shape:

.srcball/artifacts/modules/github.com/acme/project@v1.2.3/

Unsafe or non-registry-like package names, such as absolute local paths, are stored under a hashed _/<hash>@<version> directory. During materialization srcball only writes regular files and directories. Absolute paths, parent-directory traversal, hard links, symlinks, device files, and other special tar entries are skipped.

Eviction removes the cached OCI tar, metadata, access timestamp, materialized source directory, and unreferenced digest blobs. Mutable refs such as HEAD and registry latest are cached briefly before resolving upstream again; configure that with SRCBALL_MUTABLE_REF_TTL_SECS.

Hashed internal keys use two-level fanout directories (fa/bc/<rest>) so large caches do not put every artifact, ref, or Git repository under one flat directory.

Rust library #

In-process service:

use srcball::{FileStorage, GitSourceResolver, PackageSpec, SrcballService};

# async fn example() -> anyhow::Result<()> {
let storage = FileStorage::new(".srcball/artifacts");
let resolver = GitSourceResolver::new(".srcball/vcs");
let service = SrcballService::new(storage, resolver);
let spec = PackageSpec::new("github.com/cue-lang/cue", Some("v0.15.0".into()))?;
let artifact = service.fetch(&spec).await?;
std::fs::write("cue.oci.tar", artifact.body)?;
# Ok(())
# }

HTTP client:

use srcball::{PackageSpec, SrcballClient};

# async fn example() -> Result<(), Box<dyn std::error::Error>> {
let client = SrcballClient::new("http://localhost:3000");
let spec = PackageSpec::new("github.com/cue-lang/cue", Some("v0.15.0".into()))?;
let artifact = client.fetch(&spec).await?;
std::fs::write("cue.oci.tar", artifact.body)?;
# Ok(())
# }

CI #

.github/workflows/ci.yml runs:

  • cargo fmt --all -- --check
  • cargo test --workspace
  • docker build -t srcball:ci .

Milestones #

  1. Core package model: keep PackageSpec, ArtifactMetadata, Storage, and SourceResolver as the narrow service contract.
  2. Storage backends: keep filesystem as the default volume backend; add S3 by implementing Storage.
  3. OCI registry parity: fill in remaining Distribution API behavior needed by stricter clients.
  4. Resolvers: keep Git as the first resolver; add forge-aware resolvers only when they need authentication or provider-specific URL discovery.
  5. Language adapters: layer Go proxy, CUE module, and other protocol endpoints on top of SrcballService rather than duplicating cache or fetch logic.
  6. Private repositories: rely on Git transport credentials first (ssh://, git@, credential helpers); add per-request credential injection as a resolver concern.
  7. Operations: expand health checks and observability around resolver latency, cache hit rate, eviction counts, and artifact sizes.