+18
-22
README.md
+18
-22
README.md
···
9
9
- `test/` integration tests, Wasm conformance tests, etc.
10
10
- `website/` the website (TODO)
11
11
12
-
## naming convention
12
+
## naming convention
13
13
14
14
- all crates use *kebab case* both in the folder name and crate name
15
15
- OS subsystems and components are prefixed with `k` e.g. `kasync` and `kmem` (these go into `sys/`) while regular, potentially publish-able crates are not prefixed.
16
16
17
17
## buck2 techniques
18
18
19
-
Buck2 Techniques for `no_std`/Kernel Rust Projects
19
+
### Toolchains
20
20
21
-
### Toolchain
22
-
- **Nix flake-provided toolchains** — Rust nightly, clang, lld all come from Nix flake packages; a custom `flake.package()` Starlark rule calls `nix build` and exposes outputs as Buck2 targets, giving hermetic toolchains without Buck2 having to manage downloads
23
-
- **Explicit sysroot deps** — `core`, `alloc`, `compiler_builtins`, and panic runtimes declared as named deps on `rust_toolchain`, completely replacing the implicit std sysroot
24
-
- **Bootstrap mode** — a constraint + transition flips the toolchain into "build `core`/`alloc` from source" mode, so you can rebuild the standard library with custom flags without a separate toolchain definition
25
-
26
-
### Configuration
27
-
- **Custom constraints for compiler knobs** — `panic` (unwind/abort), `opt-level`, `debuginfo`, `strip` are all modeled as Buck2 constraint settings, not hardcoded flags; one target graph produces many configurations
28
-
- **Configuration transitions** — `incoming_transition` rules (e.g. `panic_abort`, `rust_bootstrap`) flip constraint values on dependency edges, so a single `deps` list can mix panic strategies or bootstrap/standard builds
29
-
- **Profile aliases** — `debug`/`release` map to constraint tuples in `PACKAGE`, giving `buck2 build -m release` without duplicating target definitions
30
-
- **Dynamic `rustc_target_triple` via `select()`** — a single toolchain definition covers 7+ target triples by selecting on constraint values at analysis time
31
-
32
-
### Cross-Compilation
33
-
- **Custom JSON target spec** — baremetal targets defined as a JSON file (ABI, linker, panic strategy, code model) referenced via `rust_target_path`; no upstream rustc target needed
34
-
- **Execution platform ≠ target platform** — host builds tools, baremetal is the target; `target_compatible_with` constraints prevent accidentally building/testing kernel code on the host
35
-
36
-
### Third-Party Crates
37
-
- **Reindeer with per-platform feature selection** — generates Buck2 targets from `Cargo.lock`; a custom `riscv64` platform definition adds `["no_std"]` features, so crates like `libc` are only deps on hosted platforms
38
-
- **Per-crate fixups** — `fixups/<crate>/fixups.toml` patches individual generated rules without full regeneration
21
+
- Sources rust, clang, lld **from the repos Nix flake**. The flake uses [oxalica's rust overlay](https://github.com/oxalica/rust-overlay) to fetch the Rust version declared in `rust-toolchain.toml`. We then obtain references to these tools using (a tweaked (tweaged??) copy of) [buck2.nix](https://github.com/tweag/buck2.nix).
22
+
- We use `reindeer` for crates.io integration. feature resolution was a bit complicated; we essentially have two partially overlapping dependency graphs: one for _target_ builds and one for _execution_ builds. By enabling the `no_std` cargo feature for _target_ builds and `std`/`default` feature for _execution_ builds we can force dependency resolution to not include incompatible crates for _target_ builds. But this part of the integration remains cumbersome.
23
+
- Compiler settings are exposed as Buck2 constraints (`opt-level`, `debuginfo`, `strip` at the moment). This way we can have crate specific overrides later for example. Currently these constraints are just exposed via `debug`/`release` aliases that emulate the Rust profiles. I.e. you can run `buck2 build -m release` to produce an optimized binary.
24
+
25
+
### build-std
26
+
27
+
- k23 needs to build the `core`/`alloc`/`compiler_builtins` crates from source. Both because we use custom JSON target specs to control the precise code generation settings and because we have a custom _unwinding_ machine that works on `no_std` builds. Rebuilding stdlib crates also allows us to apply compiler settings to the whole graph. This is just a good idea in general. It is a bit more involved to set up though.
28
+
29
+
- We defined a filegroup of our custom JSON targets, added it as the Rust toolchains `rust_target_path` and then in a `select` instruct `rustc` to build with the custom target when `os:none` and the `cpu` type matches. Otherwise we always fall back to the hosts target triple (e.g. for tests).
30
+
- `rust-toolchain.toml` conntains the `rust-src` component which means we can obtain the stdlibs source code from the nix flake too. We then just define `rust_library` targets for each required dependency and add them as **explicit sysroot deps** to the toolchain.
31
+
- Uncoditionally adding these sysroot deps would lead to a circular dependency problem when building the stdlib deps themselves (since they rely on the rust toolchain themselves of course!) so we define a _bootstrap_ mode transition. When applied, the stdlib crates are compiled without any sysroot deps.
32
+
- Similarly not all crates can be compiled using the custom unwind machinery: the `loader` stage for example cannot unwind because not enough of the machine state is set up yet. We define a second transition that allows `kernel` and dependencies to be compiled with `-Cpanic=unwind` while `loader` and dependencies are compiled using `-Cpanic=abort`.
39
33
40
34
### Testing
41
-
- **Auto-generated host-only test targets** — a `k23_rust_library` macro creates a companion `rust_test` constrained to `HOSTED` platforms, so `buck2 test //...` skips baremetal targets automatically
42
-
- **Subtarget suffixes for analysis tools** — `[check]`, `[clippy.txt]`, `[miri]` suffixes on any Rust target give lint/analysis outputs without separate rule definitions
35
+
36
+
- we have lots of _different_ tests (regular Rust tests, miri, loom, fuzz tests, test running in emulator, etc.) but not all tests run on all targets. Crates that e.g. include riscv specific assembly cannot be tested on the host and need to be run in QEMU while loom tests for example cannot be run on target.
37
+
- This was for the longest time the biggest source of headaches for k23 builds but buck2 constraints make this relatively easy: tests are marked with the platform they are compatible with and buck2 will ignore all incompatible tests by default. This still needs some cleaning up but we're finally on a good path…
38
+
- buck2 commands can get quite large, so we have a `justfile` with quick aliases for common tasks (`check` builds, running `clippy`, running `tests`, running `miri` tests, etc.)
History
2 rounds
0 comments
jonaskruckenberg.de
submitted
#1
1 commit
expand
collapse
add buck2 techniques overview
expand 0 comments
pull request successfully merged
jonaskruckenberg.de
submitted
#0
1 commit
expand
collapse
add buck2 techniques overview