+26
-1
Diff
round #1
+26
-1
README.md
+26
-1
README.md
···
1
+
## folder structure
2
+
1
3
- `build/` build system files
2
4
- `doc/` developer documentation
3
5
- `platforms/` build system configurations for various target platforms
···
7
9
- `test/` integration tests, Wasm conformance tests, etc.
8
10
- `website/` the website (TODO)
9
11
10
-
# naming convention
12
+
## naming convention
11
13
12
14
- all crates use *kebab case* both in the folder name and crate name
13
15
- OS subsystems and components are prefixed with `k` e.g. `kasync` and `kmem` (these go into `sys/`) while regular, potentially publish-able crates are not prefixed.
16
+
17
+
## buck2 techniques
18
+
19
+
### Toolchains
20
+
21
+
- Sources rust, clang, lld **from the repos Nix flake**. The flake uses [oxalica's rust overlay](https://github.com/oxalica/rust-overlay) to fetch the Rust version declared in `rust-toolchain.toml`. We then obtain references to these tools using (a tweaked (tweaged??) copy of) [buck2.nix](https://github.com/tweag/buck2.nix).
22
+
- We use `reindeer` for crates.io integration. feature resolution was a bit complicated; we essentially have two partially overlapping dependency graphs: one for _target_ builds and one for _execution_ builds. By enabling the `no_std` cargo feature for _target_ builds and `std`/`default` feature for _execution_ builds we can force dependency resolution to not include incompatible crates for _target_ builds. But this part of the integration remains cumbersome.
23
+
- Compiler settings are exposed as Buck2 constraints (`opt-level`, `debuginfo`, `strip` at the moment). This way we can have crate specific overrides later for example. Currently these constraints are just exposed via `debug`/`release` aliases that emulate the Rust profiles. I.e. you can run `buck2 build -m release` to produce an optimized binary.
24
+
25
+
### build-std
26
+
27
+
- k23 needs to build the `core`/`alloc`/`compiler_builtins` crates from source. Both because we use custom JSON target specs to control the precise code generation settings and because we have a custom _unwinding_ machine that works on `no_std` builds. Rebuilding stdlib crates also allows us to apply compiler settings to the whole graph. This is just a good idea in general. It is a bit more involved to set up though.
28
+
29
+
- We defined a filegroup of our custom JSON targets, added it as the Rust toolchains `rust_target_path` and then in a `select` instruct `rustc` to build with the custom target when `os:none` and the `cpu` type matches. Otherwise we always fall back to the hosts target triple (e.g. for tests).
30
+
- `rust-toolchain.toml` conntains the `rust-src` component which means we can obtain the stdlibs source code from the nix flake too. We then just define `rust_library` targets for each required dependency and add them as **explicit sysroot deps** to the toolchain.
31
+
- Uncoditionally adding these sysroot deps would lead to a circular dependency problem when building the stdlib deps themselves (since they rely on the rust toolchain themselves of course!) so we define a _bootstrap_ mode transition. When applied, the stdlib crates are compiled without any sysroot deps.
32
+
- Similarly not all crates can be compiled using the custom unwind machinery: the `loader` stage for example cannot unwind because not enough of the machine state is set up yet. We define a second transition that allows `kernel` and dependencies to be compiled with `-Cpanic=unwind` while `loader` and dependencies are compiled using `-Cpanic=abort`.
33
+
34
+
### Testing
35
+
36
+
- we have lots of _different_ tests (regular Rust tests, miri, loom, fuzz tests, test running in emulator, etc.) but not all tests run on all targets. Crates that e.g. include riscv specific assembly cannot be tested on the host and need to be run in QEMU while loom tests for example cannot be run on target.
37
+
- This was for the longest time the biggest source of headaches for k23 builds but buck2 constraints make this relatively easy: tests are marked with the platform they are compatible with and buck2 will ignore all incompatible tests by default. This still needs some cleaning up but we're finally on a good path…
38
+
- buck2 commands can get quite large, so we have a `justfile` with quick aliases for common tasks (`check` builds, running `clippy`, running `tests`, running `miri` tests, etc.)
History
2 rounds
0 comments
jonaskruckenberg.de
submitted
#1
1 commit
expand
collapse
add buck2 techniques overview
expand 0 comments
pull request successfully merged
jonaskruckenberg.de
submitted
#0
1 commit
expand
collapse
add buck2 techniques overview