Homebrew bottle builder and tap manager for OCaml monorepos
README.md

bottler #

Homebrew bottle builder and tap manager for OCaml monorepos.

Overview #

Homebrew installs a package from a formula, a Ruby file that says where the package comes from and how to install it. A bottle is a prebuilt binary of that package for one platform, a gzipped tar archive that brew downloads instead of compiling. A tap is a git repository of formulae that brew tap adds to the local list. Shipping an OCaml binary through Homebrew takes several steps that each fail in their own way: build a bottle per platform, upload it somewhere brew can download it, write a formula that carries the bottle's SHA-256, and push that formula to a tap. Bottler runs all of them from one homebrew.yml at the root of a dune monorepo. It builds the bottles, uploads them to S3-compatible storage (Scaleway), checks that the store serves exactly the bytes it uploaded, writes the formulae and pushes the tap.

The formula and tap primitives come from the ocaml-homebrew library. Bottler adds the parts specific to a monorepo: the dune release build, the YAML configuration, the upload and the GitHub Actions workflow that runs a release in CI.

Bottles are byte-reproducible for a given release version. Bottler writes the tar archive itself, with fixed metadata (mtime 0, uid and gid 0, empty user and group names), and compresses it at level 9 through zlib's gzip wrapper, which writes a header with modification time 0 and no file name; the one header byte that varies, zlib's operating-system code, is constant for one platform. No external tar or gzip is involved. The release version is part of the archived path (NAME/VERSION/bin/NAME) and of the binary's --version output, so two builds give the same bytes when they share a version: a fixed PROJECT_VERSION, or the same commit built on the same day (see Configuration).

Installation #

Install with opam:

$ opam install bottler

If opam cannot find the package, it may not yet be released in the public opam-repository. Add the overlay repository, then install it:

$ opam repo add samoht https://tangled.org/gazagnaire.org/opam-overlay.git
$ opam update
$ opam install bottler

Usage #

$ bottler config          # Show parsed homebrew.yml
$ bottler build           # Build bottles for build.platforms
$ bottler upload          # Upload bottles to S3
$ bottler formula NAME    # Emit a Ruby formula
$ bottler release         # Full pipeline: build + upload + tap push
$ bottler workflow        # Generate GitHub Actions release workflow
$ bottler doctor          # Verify credentials and connectivity
$ bottler login           # Set up Scaleway + rclone

bottler release is the command most people run. It builds and uploads the bottles, reads each one back from the store and compares its digest before any formula is written, then pushes the tap. When several homebrew.yml sources share one tap and another release pushes first, bottler resets its checkout to the new tip, writes its formulae again and retries.

The other commands look after the tap and the bucket. bottler install installs or upgrades packages from the tap, and bottler install --local builds bottles from the working tree and installs them without uploading anything. bottler list shows every formula in the tap and bottler remove NAME deletes one. bottler gc deletes old bottle versions from the bucket, bottler usage reports how much space it holds, and bottler keys provisions upload-only Scaleway credentials for CI. bottler init writes the bin/version helper that makes a binary report the release version from --version.

Configuration #

Bottler reads homebrew.yml at the project root:

handle: samoht
storage:
  bucket: homebrew-bottles
  region: fr-par
  profile: homebrew-samoht
tap:
  clone_url: https://github.com/samoht/homebrew-tap.git
build:
  switch: lto-5.4.1      # opam switch the release build runs under
packages: [merlint, prune]

A package is its name when the tree says the rest. Its target is the dune executable whose public_name is that name, found by reading the tree's dune files as dune does, and its description is the synopsis of the opam package that installs the executable: the (package P) of its stanza, or its project's only package, read from P.opam, or from the dune-project when there is no such file. The object form sets what the tree does not say or says otherwise, beside the per-package keys:

packages:
  - merlint
  - name: mq                      # a binary no public_name installs as mq
    target: ocaml-merge-queue/bin/main.exe
    description: "Local merge queue"
    companions:
      - name: daemon-supervisor
        target: ocaml-daemon/bin/supervisor.exe

A name that no executable or several are installed under is refused with the candidates and a request to set target. tap.push_url is the SSH form of tap.clone_url on github.com and gitlab.com, and git@KNOT:HANDLE/REPO for a tangled.org clone URL on the knot tap.knot names; another forge needs tap.push_url. bottler config prints each of these values with where it came from.

build:
  linux: static          # or linuxbrew
  switch: lto-5.4.1
  checks: [switch-pins]  # root aliases the release build also builds
  platforms: [arm64_sonoma, x86_64_linux, arm64_linux]

build.platforms lists the bottles a release builds, every one in the same release so that the formula carries them all; without it, a release builds the host's platform alone. The host builds its own platform. With build.linux: static, a Linux platform is built in a container of build.image (ocaml/opam:alpine-ocaml-5.4 unless given) with Docker, for that platform's architecture, from any host: the working tree is copied in, its dependencies installed with opam, and the release build linked with -ccopt -static, so the binary runs on any Linux distribution and can be shipped outside Homebrew too. The container's opam root is a named volume that the next release reuses. The image's current switch builds the release, so a tree whose release profile needs -use-lto names an image whose switch has it. A macOS platform builds only on a Mac of its architecture, and a linuxbrew bottle only on its Linux machine.

build.compiler lists the packages build.switch is created with where it does not exist, as on a CI runner or in a Linux container: the compiler, then its options. build.compiler_url pins the first from elsewhere, such as a fork whose compiler has -use-lto. Both are read from build.switch on the machine that has it, through opam switch export: its ocaml-base-compiler or ocaml-variants package, its ocaml-option-* packages and the compiler's pin. That machine hands them on: a release passes them to the Linux containers it starts, and bottler workflow writes them into the workflow it generates. Where the switch does not exist and nothing is given, the compiler is ocaml-base-compiler.5.4.1. Given, they are used as they are:

build:
  switch: lto-5.4.1
  compiler: [ocaml-variants.5.4.1, ocaml-option-flambda]
  compiler_url: git+https://github.com/samoht/ocaml.git#6ca6a45
  lto: true

A tree released to opam cannot put -use-lto in its own dune files, since a stock compiler rejects the flag: build.lto: true links its release with the flag through a workspace file of bottler's, and needs a build.compiler that has it. A tree with its own dune-workspace turns the flag on there instead. bottler workflow generates a GitHub Actions workflow that builds each platform on a runner of its own with bottler build --platform, uploads the bottles, and writes the formulas into the tap the workflow runs in.

A macOS bottle is labelled arm64_sonoma or sonoma, so brew pours it on macOS 14 and later. Bottler builds it with MACOSX_DEPLOYMENT_TARGET=14.0, and refuses to build under a switch whose archives are built for a later macOS: a compiler configured on a later macOS uses the functions it found there, such as pipe2 and dup3 on macOS 27, and the binary would not start on macOS 14. The refusal gives the command that rebuilds the switch for macOS 14. Each binary is checked before it is bottled: it must record no later minimum macOS, and import no function weakly, which is how a function later than the deployment target is linked.

build.switch is required and has no default. Bottles are built with --profile=release, which links with -use-lto. LTO is link-time optimisation: the compiler keeps an intermediate form of every module and optimises across modules when the program is linked. The stock OCaml compiler has no -use-lto flag and refuses it; only a compiler built with that extension accepts it, so this key names a switch whose compiler has it, usually not the switch used for day-to-day development. Bottler runs opam exec --switch <build.switch> -- dune build --profile=release <targets>, and bottler doctor checks that the switch exists and that its ocamlopt accepts -use-lto.

build.checks names root-directory dune aliases the release build builds beside the first package's binaries, as @@<alias> in the same command. A failing check fails the build, so no bottle is made, and its output says why. The monorepo names switch-pins, which refuses a release switch whose pins drifted from root.opam; no other build depends on that alias.

Each package's primary binary must report the release version from --version, and bottler checks this with PROJECT_VERSION cleared from the environment. The packages are built one at a time and each primary is checked before the next is built, so a binary that reports another version stops the release at its own build. Companion binaries are built and bottled at the same version, then installed beside the primary binary by source-built formulae. The release version is PROJECT_VERSION when it is set, and otherwise YYYYMMDD-<commit> from git HEAD, with -dirty appended when the tree has uncommitted changes. The date is the build machine's local date at build time, not the commit date.

Bottler refuses a configuration it cannot use before building anything. A missing file, a missing build: section or a missing build.switch each produce an error that names the file it resolved against the current directory, the section or key that is absent, and the lines to add. An unknown key is an error too, so a misspelt field cannot silently drop out of a formula.

Licence #

ISC License. See LICENSE.md for details.