juanlu.space / gos3dir
Simple S3-like server for development purposes, written in Go
fork atom
Go 100.0%
26 1 0

Clone this repository

https://tangled.org/juanlu.space/gos3dir
git@knot.juanlu.space:juanlu.space/gos3dir

For self-hosted knots, clone URLs may differ based on your setup.

Download tar.gz
README.md

gos3dir#

A lightweight S3-compatible server that uses a local directory as storage. Perfect for local development, testing, and offline S3 workflows. Do not use in production.

Installation#

go install tangled.org/juanlu.space/gos3dir@latest

Or build from source:

git clone <repository-url>
cd gos3dir
go build

Usage#

Start the server by pointing it to a directory:

gos3dir /path/to/data/dir

The server listens on http://localhost:8041.

Configure AWS CLI v2#

Add a profile to ~/.aws/config:

[profile gos3dir-dev]
endpoint_url = http://localhost:8041
s3 =
    addressing_style = path

Add dummy credentials to ~/.aws/credentials:

[gos3dir-dev]
aws_access_key_id = test
aws_secret_access_key = test

Examples#

List buckets:

aws s3 ls --profile gos3dir-dev

Create a bucket:

aws s3 mb s3://my-bucket --profile gos3dir-dev

Upload a file:

aws s3 cp file.txt s3://my-bucket/ --profile gos3dir-dev

List objects in a bucket:

aws s3 ls s3://my-bucket --profile gos3dir-dev

Delete an object:

aws s3 rm s3://my-bucket/file.txt --profile gos3dir-dev

Delete an empty bucket:

aws s3 rb s3://my-bucket --profile gos3dir-dev

Supported Operations#

  • List buckets (GET /)
  • List objects (GET /{bucket})
  • Download objects (GET /{bucket}/{key})
  • Create bucket (PUT /{bucket})
  • Upload object (PUT /{bucket}/{key})
  • Delete empty bucket (DELETE /{bucket})
  • Delete object (DELETE /{bucket}/{key})

Lakehouse formats#

This has been tested with Polars + Delta Lake:

In [25]: df.write_delta("s3://deltalake/df", storage_options={"AWS_ENDPOINT_URL": "http://localhost:8041", "AWS_ALLOW_HTTP": "true"})

In [26]: pl.read_delta("s3://deltalake/df", storage_options={"AWS_ENDPOINT_URL": "http://localhost:8041", "AWS_ALLOW_HTTP": "true"})
Out[26]:
shape: (4, 2)
┌─────┬─────┐
│ id  ┆ col │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 0   ┆ a   │
│ 1   ┆ b   │
│ 2   ┆ c   │
│ 3   ┆ d   │
└─────┴─────┘

Future work#

We would like to fix these at some point:

  • Proper deletion of dangling "directories"
  • Whatever is needed for open table formats (DuckLake, Apache Iceberg, Delta Lake) to work almost perfectly
  • Virtual-hosted-style addressing (for now, only path-style addressing is supported)

If you see more gaps, feel free to open an issue. But it might be deemed out of scope (see below).

Limitations#

These are by design and will not be fixed:

  • No authentication or authorization
  • No multipart uploads
  • No versioning
  • No . and .. in full key names
  • No bit-by-bit compatibility with Amazon S3 server responses