pyrox.dev
1# Nixpkgs CI evaluation
2
3The code in this directory is used by the [eval.yml](../../.github/workflows/eval.yml) GitHub Actions workflow to evaluate the majority of Nixpkgs for all PRs, effectively making sure that when the development branches are processed by Hydra, no evaluation failures are encountered.
4
5Furthermore it also allows local evaluation using:
6
7```
8nix-build ci -A eval.baseline
9```
10
11The most important two arguments are:
12- `--arg evalSystems`: The set of systems for which `nixpkgs` should be evaluated.
13 Defaults to the four official platforms (`x86_64-linux`, `aarch64-linux`, `x86_64-darwin` and `aarch64-darwin`).
14 Example: `--arg evalSystems '["x86_64-linux" "aarch64-darwin"]'`
15- `--arg quickTest`: Enables testing a single chunk of the current system only for quick iteration.
16 Example: `--arg quickTest true`
17
18The following arguments can be used to fine-tune performance:
19- `--max-jobs`: The maximum number of derivations to run at the same time.
20 Only each [supported system](../supportedSystems.json) gets a separate derivation, so it doesn't make sense to set this higher than that number.
21- `--cores`: The number of cores to use for each job.
22 Recommended to set this to the amount of cores on your system divided by `--max-jobs`.
23- `--arg chunkSize`: The number of attributes that are evaluated simultaneously on a single core.
24 Lowering this decreases memory usage at the cost of increased evaluation time.
25 If this is too high, there won't be enough chunks to process them in parallel, and will also increase evaluation time.
26 The default is 5000.
27 Example: `--arg chunkSize 10000`
28
29Note that 16GB memory is the recommended minimum, while with less than 8GB memory evaluation time suffers greatly.
30
31## Local eval with rebuilds / comparison
32
33To compare two commits locally, first run the following on the baseline commit:
34
35```
36nix-build ci -A eval.baseline --out-link baseline
37```
38
39Then, on the commit with your changes:
40
41```
42nix-build ci -A eval.full --arg baseline ./baseline
43```
44
45Keep in mind to otherwise pass the same set of arguments for both commands (`evalSystems`, `quickTest`, `chunkSize`).
46Running this command will evaluate the difference between the baseline statistics and the ones at the time of running the command.
47From that difference, it will produce a human-readable report in `$out/step-summary.md`.
48If no packages were added or removed, then performance statistics will also be generated as part of this report.