blobsky #

Blobsky is a small S3-compatible object store written in Go. It streams uploads to local disk and streams them back without loading the whole object into memory. I wrote the storage engine for this project using Go's standard library.

It supports PutObject, GetObject with a single byte range, HeadObject, DeleteObject, and ListObjectsV2 with prefix filtering and pagination. ETags are the quoted MD5 of the object payload.

Run using Flox #

Install Flox 1.19.0 or newer. If Flox asks you to authenticate when resolving packages, run flox auth login.

flox activate
mkdir -p bin
CGO_ENABLED=0 go build -o bin/blobsky ./cmd/blobsky
./bin/blobsky -data-dir ./data -bucket demo

Wait for the listening log before sending requests. Keep ./data between restarts.

The server listens on 127.0.0.1:9000 by default. It has no authentication or TLS, so do not expose it publicly. For a remote demo, use an SSH tunnel or restrict access to trusted hosts.

Try it #

In another terminal, activate Flox and point the AWS CLI at the server:

flox activate
export AWS_ENDPOINT_URL_S3=http://127.0.0.1:9000 # Or your server IP if you set your listening address to the server ip
aws s3api list-objects-v2 --bucket demo

Flox supplies dummy AWS credentials and a default region. It overrides AWS credentials inside the activated environment, so use a separate shell for real AWS work. Set AWS_ENDPOINT_URL_S3 to your server's address when connecting from another machine, and unset it after the demo.

For a full CLI and Go SDK walkthrough, run:

export BLOBSKY_ENDPOINT=http://127.0.0.1:9000
export BLOBSKY_BUCKET=demo
bash scripts/smoke-cli.sh
go test ./tests/integration -run TestSDKRoundTrip -v -count=1

These checks upload, inspect, download, range-read, list, and delete test objects. The CLI script leaves its downloaded files in a temporary directory. Use s3api put-object for large files: aws s3 cp can switch to multipart upload, which Blobsky does not support.

Configuration #

Flags override environment variables.

Flag	Environment variable	Default
-listen	BLOBSKY_LISTEN	127.0.0.1:9000
-data-dir	BLOBSKY_DATA_DIR	./data
-bucket	BLOBSKY_BUCKET	demo
-max-uploads	BLOBSKY_MAX_UPLOADS	8
-shutdown-timeout	BLOBSKY_SHUTDOWN_TIMEOUT	10s

The configured bucket is ready at startup; no CreateBucket call is needed. The server returns 503 when its upload slots are full. Slow clients can hold those slots until they finish or disconnect.

Durability and failures #

I chose self-describing object files so the payload and its metadata are published together, without coordinating a separate database. A successful PUT or DELETE means Blobsky synced the changes needed to recover it. Readers keep a consistent version while another request replaces or deletes the key.

If an upload fails before publication, the previous object remains available. If a rename, unlink, or directory sync has an uncertain outcome, Blobsky stops serving object requests until it restarts and recovers. A failed request can have taken effect, so check the object after recovery. Incomplete upload files are cleaned up at startup; damaged committed files stop startup instead of being silently removed.

These guarantees assume a local filesystem with crash-consistent replacement and working fsync, and a device that honors flushes.

Tests #

flox activate -- go test -race ./...
flox activate -- go vet ./...
flox activate -- golangci-lint run
BLOBSKY_REAL_CLIENTS=1 flox activate -- go test -race ./tests/integration -run TestRealClients -v -count=1 -timeout=5m

The tests cover concurrent reads and writes, interrupted uploads, injected disk errors, recovery, and process kills at PUT and DELETE boundaries. A process kill tests crash recovery, not power loss; injected errors do not simulate a physically full disk.

The real-client tests passed with AWS CLI 2.36.31 and Go SDK S3 module 1.114.8, including 12–13 MiB uploads, overwrites, range reads, pagination, and restarts. They use seekable bodies over HTTP; other clients and upload modes are not covered by those results. The server checks Content-MD5, but does not verify the CLI's CRC64NVME checksum header.

Scope and next steps #

I focused on durability, concurrent access, and partial failures rather than covering every S3 endpoint. Multipart uploads, CopyObject, batch deletion, delimiter listings, conditional requests, and aws-chunked framing are unsupported. Signature verification, authentication, versioning, ACLs, and multi-node storage are out of scope.

The in-memory index is rebuilt at startup and grows with the object count. Aditionally, Blobsky does not verify the payload checksum on ordinary reads. Next, I would focus on testing power cuts at the filesystem/device level, measure memory under large transfers, and add client upload modes based on need.

One piece that could be either implemented directly into Blobsky is a function that would enable automatic backup of buckets (configurable target) and recovery. I would want this to be something that users can start up and move on to the next thing with the confidence that their buckets have backup/recovery already covered out of the box.

The .flox/ manifest and lockfile travel with the source, so another developer can run flox activate with the same locked tools.