# odd — design

Keep per-switch HTML documentation continuously up to date by hooking into
opam's session lifecycle and rebuilding only the packages each session
actually changed, using `odd-driver` (from the odoc driver, opam
mode) as the per-package generator.

## Goals

- After any successful `opam install` / `opam upgrade` / `opam remove`, the
  switch's HTML docs reflect the new state, with no manual step.
- Incremental: only packages (re)built or removed in the session are
  reprocessed.
- Per-switch: all state and output live under the switch prefix and vanish
  with `opam switch remove`.
- Fail-safe: a documentation failure never causes the opam invocation to
  fail.

## Non-goals

- Multi-universe handling (docs.ocaml.org's problem). A switch is a single
  coherent universe: exactly one version of each package, all built
  together — which is why opam mode drops voodoo mode's
  universe/blessed machinery and uses the flat `<package>/` layout.
- Parallel doc building across packages (possible later; see Open
  questions).

## The generator: `odd-driver`

A variant of `odoc_driver` (opam mode) designed for exactly this job. It is
**vendored** into this repo under `driver/` (the whole `odoc_driver_lib` plus
the opam-mode entry point, built and installed as the `odd-driver` binary
alongside `odd`) — a temporary measure pending upstreaming of the changes into
`odoc-driver`; until then `odd` carries its own copy so it is self-contained.
Per invocation it documents **one package installed in the current switch**:

- Package file lists come from opam's own record,
  `<prefix>/.opam-switch/install/<pkg>.changes` — no prepared input tree,
  no file copying; installed artifacts are read in place.
- The switch is discovered by shelling out to `opam switch show` /
  `opam var prefix` (both read-only operations).
- `--odoc-dir`, `--odocl-dir` and `--html-dir` all default to
  `<switch-prefix>/odoc`. Intermediate `.odoc`/`.odocl` files and HTML
  share that tree, with each package's output directly under
  `odoc/<package>/` (flat layout, same as plain `odoc_driver`). This is an
  odd-owned tree, kept separate from opam's own `<prefix>/doc`, where
  packages install their own documentation.
- Previously built dependencies are discovered by scanning `--odoc-dir`
  for the `.odoc_pkg_marker` / `.odoc_lib_marker` files written on earlier
  runs — this is what makes incremental, package-at-a-time building work.
- Like the voodoo driver, it must be run package-by-package, **with each
  package's dependencies already compiled** in `--odoc-dir`. Ordering is
  our job (see Ordering).

So the wrapper tool's responsibilities reduce to: ordering the work and
invoking the driver once per stale package (recording what changed and
cleaning up removals are done directly by the coreutils hooks, see below).

## Components

### 1. opam hooks (configured in `~/.opam/config`, shipped via opamrc)

Three wrapper fields drive everything. The two per-package hooks are plain
coreutils — crucially they invoke **no odd binary** — so they keep
working even in a switch where odd isn't installed (or is mid-upgrade,
or where a *global* hook fires for a switch that never had it). All real work
happens once per session, in the one hook that does need the binary.

```
post-install-commands: [
  [ "sh" "-c" "mkdir -p \"$1/odoc/$2\" && touch \"$1/odoc/$2/.odd-stale\""
    "--" "%{prefix}%" "%{name}%" ] { error-code = 0 }
]
post-remove-commands: [
  [ "rm" "-rf" "%{prefix}%/odoc/%{name}%" ] { error-code = 0 }
]
post-session-commands: [
  [ "%{hooks}%/odd" "sync" "--prefix" "%{prefix}%" ] { success }
]
```

- **post-install** marks the package stale by `touch`-ing
  `odoc/<pkg>/.odd-stale` (after `mkdir -p`-ing the dir). It fires for
  every install action including same-version rebuilds — which matters,
  because a dependency-triggered rebuild changes the `.cmti`s but is invisible
  to session-level `%{new}%`/`%{removed}%`. A marker is a single empty file per
  package, so opam's parallel package actions can't corrupt anything (distinct
  files; `touch` is idempotent). **The set of markers is the work list** — no
  ordering or content is implied (order is recomputed at sync time, see
  Ordering below).
- **post-remove** deletes the package's doc subtree outright. Removal needs no
  marker and no deferral: the deletion *is* the action, done immediately and
  binary-free. (This also clears any stale marker that lived in that subtree.)
- **post-session** runs `odd sync`, the one hook that needs the binary.
  It always exits 0: doc failures are logged, never propagated, because opam
  aborts the invocation with a configuration error if a session hook fails
  (`opamSolution.ml`, post-session handling). If odd is absent, this
  hook simply doesn't run; the markers persist and are drained by the next
  session that does have it — deferred, never lost.

### 2. State and layout (all under `$OPAM_SWITCH_PREFIX`)

```
var/cache/odd/
  log              # sync output, since hooks must stay quiet
  lock             # guards a manual sync against a hook-invoked one
odoc/
  <package>/
    .odd-stale  # marker: post-install touched it, sync rebuilds
    ...                # per-package odoc, odocl and HTML (driver defaults)
  index.html       # switch-wide landing page (ours)
  odoc-search/...  # driver support files, search assets
```

The driver's defaults are taken as-is: one `odoc/` tree per switch holding
both intermediates and HTML, separate from opam's own `<prefix>/doc`. The
work list lives in that tree too — a `.odd-stale` marker file inside
each stale package's dir — so the post-install hook can write it without
odd. Our only other state is the log and lock.

### 3. The `sync` step

Runs once at session end, under opam's switch lock (post-session executes
inside the opam invocation), so no extra locking is needed against
concurrent opam sessions. Take a private lock file anyway to guard against
a manually-invoked `sync` racing a hook-invoked one. The hook environment
is the switch environment, so the driver's `opam switch show` resolves to
the right switch; being read-only, the nested opam calls don't contend
with the lock the surrounding session holds.

1. **Collect the stale set**: the package names whose `odoc/<pkg>/` holds a
   `.odd-stale` marker, intersected with the installed set. (A marker
   for a no-longer-installed package can only survive an install+remove in the
   same session — the remove hook's `rm -rf` normally clears it — so tidy it
   away.) Removals need no handling here: the post-remove hook already deleted
   the docs.
2. **Compute dependency order** over the stale set (see Ordering).
3. **For each stale package, in order**: delete `odoc/<pkg>/` (a clean build —
   the flat layout has no version in the path, so this prevents files from
   modules that no longer exist surviving from a previous version, and it
   removes the marker), then run `odd-driver <pkg> --actions all` (all
   directory options left at their switch defaults). On success the marker
   stays gone; on failure re-create it (`mkdir -p` + `touch`) so the next
   session retries.
4. **Regenerate the top-level index** *every session*: a landing page listing
   all installed packages with built docs (directory listing of `odoc/`,
   filtered). Doing this unconditionally is what lets a session that only
   *removed* packages drop them from the page — there's no removal marker to
   trigger on. The write is skipped when the content is unchanged, so idle
   sessions stay cheap. The per-package pages and redirects are the driver's
   job; only this one page is ours.

### 4. Ordering

`odd-driver` must see a package's dependencies already compiled in
`--odoc-dir`, so stale packages are processed in dependency order. Two
facts make this tractable:

- **The stale set is closed under reverse dependencies.** If A's rebuild
  could affect B's docs, opam rebuilt B too (that is what triggers
  recompilation), so B was marked stale by the post-install hook. We never
  need to add packages to the work list ourselves.
- **Order is computed at sync time, never inferred from the markers.** The
  markers are an unordered set (filesystem entries with no meaningful
  timestamps to trust), so order is always recomputed from package metadata —
  necessary anyway, since a marker left over from a failed earlier session can
  coexist with a freshly touched one for its dependency.

The graph is built from opam's installed-package metadata:
`$OPAM_SWITCH_PREFIX/.opam-switch/packages/<pkg>.<ver>/opam` — the opam
file *as installed*, correct for pins and local packages. Parse with the
`opam-format` library (never by hand):

- collect every package name in `depends:` via `OpamFormula.fold_left`,
  ignoring version constraints and disjunction structure;
- **exclude** dependencies marked `{post}` — they are allowed to be
  circular and opam itself excludes them from build order;
- **include** `depopts:` — an installed optional dependency influenced the
  build, hence the docs;
- intersect the collected names with the installed set (the directory
  listing of `.opam-switch/packages/` is that set). Whichever disjunct or
  optional dep is installed is the one that was used. Spurious edges from
  installed `with-test`/`with-doc` deps only constrain order; they cannot
  make it wrong.

Topologically sort `stale` against this graph (edges to non-stale packages
are dropped — their docs are already current in `--odoc-dir`). A cycle
(shouldn't happen with `{post}` excluded) is logged and broken arbitrarily
rather than failing the run.

### 5. The compiler / stdlib

The compiler's libraries (stdlib, compiler-libs, threads…) live under
`lib/ocaml` and everything depends on them, so they must be documented
first whenever they change. In our favour, they are installed by an
ordinary package (`ocaml-base-compiler` / `ocaml-variants` /
`ocaml-system`) whose `.changes` file covers `lib/ocaml`, so
`odd-driver`'s normal discovery path applies — no synthetic
package needed, just the ordinary ordering edge, and a compiler change
stales the whole world (correct: opam rebuilds the world anyway).
Needs verification early in implementation: that the driver's library
discovery (META-less lib-dir scanning) handles the `lib/ocaml` layout,
and how the `ocaml` wrapper package (which installs almost nothing
itself) behaves — possibly just a no-op build to skip.

## Tool shape

One OCaml executable, `odd`, with subcommands:

- `odd sync` — the session worker described above (the post-session hook).
- `odd rebuild [--all | <pkg>...]` — manual escape hatch: mark
  packages (or everything installed) stale and run sync. `--all` works on
  pre-existing switches because the installed-package metadata is always
  present, whether or not hooks were configured at install time.
- `odd setup` — write the three wrapper fields into `~/.opam/config`
  with `opam option --global` (idempotent).

There is deliberately **no** `record` subcommand: recording a change (a
`touch`) and cleaning up a removal (an `rm -rf`) are plain coreutils in the
hooks, so they never depend on odd being installed.

Dependencies: `opam-format` (opam file parsing), `bos`, `fpath`,
`cmdliner`. The vendored driver (`driver/`) pulls in its own cone (eio,
progress, yojson, ocamlfind, logs, sexplib, ppx_sexp_conv, and the `odoc` /
`odoc-md` / `sherlodoc` binaries it drives); `odd sync` still invokes
`odd-driver` as a subprocess, so the driver's runtime cost stays in a separate
process. `odd-driver` is built and installed by this package, so it is always
on `PATH` next to `odd` (overridable with `--driver` / `$ODD_DRIVER`).

Distribution: an opamrc adding the three `*-commands` fields for
`opam init --config`; `odd setup` for retrofitting existing roots.

## Failure handling summary

| Failure | Behaviour |
|---|---|
| package build fails | hooks filtered on `error-code = 0` / `{ success }`; the package isn't marked stale, no sync |
| odd not installed in the switch | post-install/post-remove still work (coreutils); markers accumulate, drained by the next session that has odd — deferred, never lost |
| doc build fails for one package | logged; its marker is re-created; later packages still attempted (their deps' docs may be stale — accepted, fixed on retry) |
| sync interrupted | markers intact (a marker is only cleared by a successful build); next session resumes |
| `odd-driver` not found (only if `--driver` points elsewhere; it ships with `odd`) | sync logs and exits 0; markers survive until the driver is available |
| docs failure overall | `sync` exits 0 regardless; opam reports its own success untouched |

## Open questions

- **Parallelism**: stale-set topo order admits parallel builds of
  independent packages; `odd-driver` already parallelises units
  internally (`nb_workers`), which is probably enough for session-sized
  work lists.
- **Search/occurrences**: the driver emits per-package occurrence files
  and json search indexes; merging them into a switch-wide search index is
  a later `sync` step.
- **Driver self-staleness**: when the switch's odoc/driver itself is
  upgraded, every package's intermediates were produced by the old odoc.
  The closure property covers packages opam rebuilt, but odoc version
  changes may warrant a full `rebuild --all` — detectable by recording the
  driver version used per run.