···1+2+# Developing the NixOS Test Driver {#chap-developing-the-test-driver}
3+4+The NixOS test framework is a project of its own.
5+6+It consists of roughly the following components:
7+8+ - `nixos/lib/test-driver`: The Python framework that sets up the test and runs the [`testScript`](#test-opt-testScript)
9+ - `nixos/lib/testing`: The Nix code responsible for the wiring, written using the (NixOS) Module System.
10+11+These components are exposed publicly through:
12+13+ - `nixos/lib/default.nix`: The public interface that exposes the `nixos/lib/testing` entrypoint.
14+ - `flake.nix`: Exposes the `lib.nixos`, including the public test interface.
15+16+Beyond the test driver itself, its integration into NixOS and Nixpkgs is important.
17+18+ - `pkgs/top-level/all-packages.nix`: Defines the `nixosTests` attribute, used
19+ by the package `tests` attributes and OfBorg.
20+ - `nixos/release.nix`: Defines the `tests` attribute built by Hydra, independently, but analogous to `nixosTests`
21+ - `nixos/release-combined.nix`: Defines which tests are channel blockers.
22+23+Finally, we have legacy entrypoints that users should move away from, but are cared for on a best effort basis.
24+These include `pkgs.nixosTest`, `testing-python.nix` and `make-test-python.nix`.
25+26+## Testing changes to the test framework {#sec-test-the-test-framework}
27+28+When making significant changes to the test framework, we run the tests on Hydra, to avoid disrupting the larger NixOS project.
29+30+For this, we use the `python-test-refactoring` branch in the `NixOS/nixpkgs` repository, and its [corresponding Hydra jobset](https://hydra.nixos.org/jobset/nixos/python-test-refactoring).
31+This branch is used as a pointer, and not as a feature branch.
32+33+1. Rebase the PR onto a recent, good evaluation of `nixos-unstable`
34+2. Create a baseline evaluation by force-pushing this revision of `nixos-unstable` to `python-test-refactoring`.
35+3. Note the evaluation number (we'll call it `<previous>`)
36+4. Push the PR to `python-test-refactoring` and evaluate the PR on Hydra
37+5. Create a comparison URL by navigating to the latest build of the PR and adding to the URL `?compare=<previous>`. This is not necessary for the evaluation that comes right after the baseline.
38+39+Review the removed tests and newly failed tests using the constructed URL; otherwise you will accidentally compare iterations of the PR instead of changes to the PR base.
40+41+As we currently have some flaky tests, newly failing tests are expected, but should be reviewed to make sure that
42+ - The number of failures did not increase significantly.
43+ - All failures that do occur can reasonably be assumed to fail for a different reason than the changes.
···15 description = "Quickly get ANSI escape sequences";
16 longDescription = ''
17 CLI utility called "ansi" to quickly get ANSI escape sequences. Supports
18- the colors and styles, such as bold or italic.";
19 '';
20 homepage = "https://github.com/phip1611/ansi-escape-sequences-cli";
21 license = with licenses; [ mit ];
···15 description = "Quickly get ANSI escape sequences";
16 longDescription = ''
17 CLI utility called "ansi" to quickly get ANSI escape sequences. Supports
18+ the colors and styles, such as bold or italic.
19 '';
20 homepage = "https://github.com/phip1611/ansi-escape-sequences-cli";
21 license = with licenses; [ mit ];
+3-3
pkgs/servers/plex/raw.nix
···12# server, and the FHS userenv and corresponding NixOS module should
13# automatically pick up the changes.
14stdenv.mkDerivation rec {
15- version = "1.30.2.6563-3d4dc0cce";
16 pname = "plexmediaserver";
1718 # Fetch the source
19 src = if stdenv.hostPlatform.system == "aarch64-linux" then fetchurl {
20 url = "https://downloads.plex.tv/plex-media-server-new/${version}/debian/plexmediaserver_${version}_arm64.deb";
21- sha256 = "0sz6xc484flh1cnlrvwin7x34bl118yy2mwj034f8p9ngiy5hrkw";
22 } else fetchurl {
23 url = "https://downloads.plex.tv/plex-media-server-new/${version}/debian/plexmediaserver_${version}_amd64.deb";
24- sha256 = "0ymxfy3s9nygv9syiy2bdwmjfqg8m4i5n8c37z1ib6393iwj8mgi";
25 };
2627 outputs = [ "out" "basedb" ];
···12# server, and the FHS userenv and corresponding NixOS module should
13# automatically pick up the changes.
14stdenv.mkDerivation rec {
15+ version = "1.31.0.6654-02189b09f";
16 pname = "plexmediaserver";
1718 # Fetch the source
19 src = if stdenv.hostPlatform.system == "aarch64-linux" then fetchurl {
20 url = "https://downloads.plex.tv/plex-media-server-new/${version}/debian/plexmediaserver_${version}_arm64.deb";
21+ sha256 = "sha256-ttkvYD+ALxfZpQutI1VyTbmQi/7hmvZ+YMUv3lskeWU=";
22 } else fetchurl {
23 url = "https://downloads.plex.tv/plex-media-server-new/${version}/debian/plexmediaserver_${version}_amd64.deb";
24+ sha256 = "sha256-TTEcyIBFiuJTNHeJ9wu+4o2ol72oCvM9FdDPC83J3Mc=";
25 };
2627 outputs = [ "out" "basedb" ];