···1088108810891089## Environment Helpers {#ssec-pkgs-dockerTools-helpers}
1090109010911091-Some packages expect certain files to be available globally.
10921092-When building an image from scratch (i.e. without `fromImage`), these files are missing.
10931093-`pkgs.dockerTools` provides some helpers to set up an environment with the necessary files.
10941094-You can include them in `copyToRoot` like this:
10911091+When building Docker images with Nix, you might also want to add certain files that are expected to be available globally by the software you're packaging.
10921092+Simple examples are the `env` utility in `/usr/bin/env`, or trusted root TLS/SSL certificates.
10931093+Such files will most likely not be included if you're building a Docker image from scratch with Nix, and they might also not be included if you're starting from a Docker image that doesn't include them.
10941094+The helpers in this section are packages that provide some of these commonly-needed global files.
1095109510961096-```nix
10971097-buildImage {
10981098- name = "environment-example";
10991099- copyToRoot = with pkgs.dockerTools; [
11001100- usrBinEnv
11011101- binSh
11021102- caCertificates
11031103- fakeNss
11041104- ];
11051105-}
11061106-```
10961096+Most of these helpers are packages, which means you have to add them to the list of contents to be included in the image (this changes depending on the function you're using to build the image).
10971097+[](#ex-dockerTools-helpers-buildImage) and [](#ex-dockerTools-helpers-buildLayeredImage) show how to include these packages on `dockerTools` functions that build an image.
10981098+For more details on how that works, see the documentation for the function you're using.
1107109911081100### usrBinEnv {#sssec-pkgs-dockerTools-helpers-usrBinEnv}
1109110111101102This provides the `env` utility at `/usr/bin/env`.
11031103+This is currently implemented by linking to the `env` binary from the `coreutils` package, but is considered an implementation detail that could change in the future.
1111110411121105### binSh {#sssec-pkgs-dockerTools-helpers-binSh}
1113110611141114-This provides `bashInteractive` at `/bin/sh`.
11071107+This provides a `/bin/sh` link to the `bash` binary from the `bashInteractive` package.
11081108+Because of this, it supports cases such as running a command interactively in a container (for example by running `docker run -it <image_name>`).
1115110911161110### caCertificates {#sssec-pkgs-dockerTools-helpers-caCertificates}
1117111111181118-This sets up `/etc/ssl/certs/ca-certificates.crt`.
11121112+This adds trusted root TLS/SSL certificates from the `cacert` package in multiple locations in an attempt to be compatible with binaries built for multiple Linux distributions.
11131113+The locations currently used are:
1119111411151115+- `/etc/ssl/certs/ca-bundle.crt`
11161116+- `/etc/ssl/certs/ca-certificates.crt`
11171117+- `/etc/pki/tls/certs/ca-bundle.crt`
11181118+11191119+[]{#ssec-pkgs-dockerTools-fakeNss}
11201120### fakeNss {#sssec-pkgs-dockerTools-helpers-fakeNss}
1121112111221122-Provides `/etc/passwd` and `/etc/group` that contain root and nobody.
11231123-Useful when packaging binaries that insist on using nss to look up
11241124-username/groups (like nginx).
11221122+This is a re-export of the `fakeNss` package from Nixpkgs.
11231123+See [](#sec-fakeNss).
1125112411261125### shadowSetup {#ssec-pkgs-dockerTools-shadowSetup}
1127112611281128-This constant string is a helper for setting up the base files for managing users and groups, only if such files don't exist already. It is suitable for being used in a [`buildImage` `runAsRoot`](#ex-dockerTools-buildImage-runAsRoot) script for cases like in the example below:
11271127+This is a string containing a script that sets up files needed for [`shadow`](https://github.com/shadow-maint/shadow) to work (using the `shadow` package from Nixpkgs), and alters `PATH` to make all its utilities available in the same script.
11281128+It is intended to be used with other dockerTools functions in attributes that expect scripts.
11291129+After the script in `shadowSetup` runs, you'll then be able to add more commands that make use of the utilities in `shadow`, such as adding any extra users and/or groups.
11301130+See [](#ex-dockerTools-shadowSetup-buildImage) and [](#ex-dockerTools-shadowSetup-buildLayeredImage) to better understand how to use it.
11311131+11321132+`shadowSetup` achieves a result similar to [`fakeNss`](#sssec-pkgs-dockerTools-helpers-fakeNss), but only sets up a `root` user with different values for the home directory and the shell to use, in addition to setting up files for [PAM](https://en.wikipedia.org/wiki/Linux_PAM) and a {manpage}`login.defs(5)` file.
11331133+11341134+:::{.caution}
11351135+Using both `fakeNss` and `shadowSetup` at the same time will either cause your build to break or produce unexpected results.
11361136+Use either `fakeNss` or `shadowSetup` depending on your use case, but avoid using both.
11371137+:::
11381138+11391139+:::{.note}
11401140+When used with [`buildLayeredImage`](#ssec-pkgs-dockerTools-buildLayeredImage) or [`streamLayeredImage`](#ssec-pkgs-dockerTools-streamLayeredImage), you will have to set the `enableFakechroot` attribute to `true`, or else the script in `shadowSetup` won't run properly.
11411141+See [](#ex-dockerTools-shadowSetup-buildLayeredImage).
11421142+:::
11431143+11441144+### Examples {#ssec-pkgs-dockerTools-helpers-examples}
11451145+11461146+:::{.example #ex-dockerTools-helpers-buildImage}
11471147+# Using `dockerTools`'s environment helpers with `buildImage`
11481148+11491149+This example adds the [`binSh`](#sssec-pkgs-dockerTools-helpers-binSh) helper to a basic Docker image built with [`dockerTools.buildImage`](#ssec-pkgs-dockerTools-buildImage).
11501150+This helper makes it possible to enter a shell inside the container.
11511151+This is the `buildImage` equivalent of [](#ex-dockerTools-helpers-buildLayeredImage).
11521152+11531153+```nix
11541154+{ dockerTools, hello }:
11551155+dockerTools.buildImage {
11561156+ name = "env-helpers";
11571157+ tag = "latest";
11581158+11591159+ copyToRoot = [
11601160+ hello
11611161+ dockerTools.binSh
11621162+ ];
11631163+```
11641164+11651165+After building the image and loading it in Docker, we can create a container based on it and enter a shell inside the container.
11661166+This is made possible by `binSh`.
11671167+11681168+```shell
11691169+$ nix-build
11701170+(some output removed for clarity)
11711171+/nix/store/2p0i3i04cgjlk71hsn7ll4kxaxxiv4qg-docker-image-env-helpers.tar.gz
11721172+$ docker load -i /nix/store/2p0i3i04cgjlk71hsn7ll4kxaxxiv4qg-docker-image-env-helpers.tar.gz
11731173+(output removed for clarity)
11741174+$ docker run --rm -it env-helpers:latest /bin/sh
11751175+sh-5.2# help
11761176+GNU bash, version 5.2.21(1)-release (x86_64-pc-linux-gnu)
11771177+(rest of output removed for clarity)
11781178+```
11791179+:::
11801180+11811181+:::{.example #ex-dockerTools-helpers-buildLayeredImage}
11821182+# Using `dockerTools`'s environment helpers with `buildLayeredImage`
11831183+11841184+This example adds the [`binSh`](#sssec-pkgs-dockerTools-helpers-binSh) helper to a basic Docker image built with [`dockerTools.buildLayeredImage`](#ssec-pkgs-dockerTools-buildLayeredImage).
11851185+This helper makes it possible to enter a shell inside the container.
11861186+This is the `buildLayeredImage` equivalent of [](#ex-dockerTools-helpers-buildImage).
1129118711301188```nix
11311131-buildImage {
11891189+{ dockerTools, hello }:
11901190+dockerTools.buildLayeredImage {
11911191+ name = "env-helpers";
11921192+ tag = "latest";
11931193+11941194+ contents = [
11951195+ hello
11961196+ dockerTools.binSh
11971197+ ];
11981198+11991199+ config = {
12001200+ Cmd = [ "/bin/hello" ];
12011201+ };
12021202+}
12031203+```
12041204+12051205+After building the image and loading it in Docker, we can create a container based on it and enter a shell inside the container.
12061206+This is made possible by `binSh`.
12071207+12081208+```shell
12091209+$ nix-build
12101210+(some output removed for clarity)
12111211+/nix/store/rpf47f4z5b9qr4db4ach9yr4b85hjhxq-env-helpers.tar.gz
12121212+$ docker load -i /nix/store/rpf47f4z5b9qr4db4ach9yr4b85hjhxq-env-helpers.tar.gz
12131213+(output removed for clarity)
12141214+$ docker run --rm -it env-helpers:latest /bin/sh
12151215+sh-5.2# help
12161216+GNU bash, version 5.2.21(1)-release (x86_64-pc-linux-gnu)
12171217+(rest of output removed for clarity)
12181218+```
12191219+:::
12201220+12211221+:::{.example #ex-dockerTools-shadowSetup-buildImage}
12221222+# Using `dockerTools.shadowSetup` with `dockerTools.buildImage`
12231223+12241224+This is an example that shows how to use `shadowSetup` with `dockerTools.buildImage`.
12251225+Note that the extra script in `runAsRoot` uses `groupadd` and `useradd`, which are binaries provided by the `shadow` package.
12261226+These binaries are added to the `PATH` by the `shadowSetup` script, but only for the duration of `runAsRoot`.
12271227+12281228+```nix
12291229+{ dockerTools, hello }:
12301230+dockerTools.buildImage {
11321231 name = "shadow-basic";
12321232+ tag = "latest";
12331233+12341234+ copyToRoot = [ hello ];
1133123511341236 runAsRoot = ''
11351135- #!${pkgs.runtimeShell}
11361136- ${pkgs.dockerTools.shadowSetup}
11371137- groupadd -r redis
11381138- useradd -r -g redis redis
12371237+ ${dockerTools.shadowSetup}
12381238+ groupadd -r hello
12391239+ useradd -r -g hello hello
11391240 mkdir /data
11401140- chown redis:redis /data
12411241+ chown hello:hello /data
11411242 '';
12431243+12441244+ config = {
12451245+ Cmd = [ "/bin/hello" ];
12461246+ WorkingDir = "/data";
12471247+ };
11421248}
11431249```
12501250+:::
1144125111451145-Creating base files like `/etc/passwd` or `/etc/login.defs` is necessary for shadow-utils to manipulate users and groups.
12521252+:::{.example #ex-dockerTools-shadowSetup-buildLayeredImage}
12531253+# Using `dockerTools.shadowSetup` with `dockerTools.buildLayeredImage`
1146125411471147-When using `buildLayeredImage`, you can put this in `fakeRootCommands` if you `enableFakechroot`:
12551255+It accomplishes the same thing as [](#ex-dockerTools-shadowSetup-buildImage), but using `buildLayeredImage` instead.
12561256+12571257+Note that the extra script in `fakeRootCommands` uses `groupadd` and `useradd`, which are binaries provided by the `shadow` package.
12581258+These binaries are added to the `PATH` by the `shadowSetup` script, but only for the duration of `fakeRootCommands`.
12591259+11481260```nix
11491149-buildLayeredImage {
11501150- name = "shadow-layered";
12611261+{ dockerTools, hello }:
12621262+dockerTools.buildLayeredImage {
12631263+ name = "shadow-basic";
12641264+ tag = "latest";
12651265+12661266+ contents = [ hello ];
1151126711521268 fakeRootCommands = ''
11531153- ${pkgs.dockerTools.shadowSetup}
12691269+ ${dockerTools.shadowSetup}
12701270+ groupadd -r hello
12711271+ useradd -r -g hello hello
12721272+ mkdir /data
12731273+ chown hello:hello /data
11541274 '';
11551275 enableFakechroot = true;
12761276+12771277+ config = {
12781278+ Cmd = [ "/bin/hello" ];
12791279+ WorkingDir = "/data";
12801280+ };
11561281}
11571282```
12831283+:::
1158128411591159-## fakeNss {#ssec-pkgs-dockerTools-fakeNss}
12851285+[]{#ssec-pkgs-dockerTools-buildNixShellImage-arguments}
12861286+## buildNixShellImage {#ssec-pkgs-dockerTools-buildNixShellImage}
1160128711611161-If your primary goal is providing a basic skeleton for user lookups to work,
11621162-and/or a lesser privileged user, adding `pkgs.fakeNss` to
11631163-the container image root might be the better choice than a custom script
11641164-running `useradd` and friends.
12881288+`buildNixShellImage` uses [`streamNixShellImage`](#ssec-pkgs-dockerTools-streamNixShellImage) underneath to build a compressed Docker-compatible repository tarball of an image that sets up an environment similar to that of running `nix-shell` on a derivation.
12891289+Basically, `buildNixShellImage` runs the script created by `streamNixShellImage` to save the compressed image in the Nix store.
1165129011661166-It provides a `/etc/passwd` and `/etc/group`, containing `root` and `nobody`
11671167-users and groups.
12911291+`buildNixShellImage` supports the same options as `streamNixShellImage`, see [`streamNixShellImage`](#ssec-pkgs-dockerTools-streamNixShellImage) for details.
1168129211691169-It also provides a `/etc/nsswitch.conf`, configuring NSS host resolution to
11701170-first check `/etc/hosts`, before checking DNS, as the default in the absence of
11711171-a config file (`dns [!UNAVAIL=return] files`) is quite unexpected.
12931293+[]{#ssec-pkgs-dockerTools-buildNixShellImage-example}
12941294+### Examples {#ssec-pkgs-dockerTools-buildNixShellImage-examples}
1172129511731173-You can pair it with `binSh`, which provides `bin/sh` as a symlink
11741174-to `bashInteractive` (as `/bin/sh` is configured as a shell).
12961296+:::{.example #ex-dockerTools-buildNixShellImage-hello}
12971297+# Building a Docker image with `buildNixShellImage` with the build environment for the `hello` package
1175129811761176-```nix
11771177-buildImage {
11781178- name = "shadow-basic";
12991299+This example shows how to build the `hello` package inside a Docker container built with `buildNixShellImage`.
13001300+The Docker image generated will have a name like `hello-<version>-env` and tag `latest`.
13011301+This example is the `buildNixShellImage` equivalent of [](#ex-dockerTools-streamNixShellImage-hello).
1179130211801180- copyToRoot = pkgs.buildEnv {
11811181- name = "image-root";
11821182- paths = [ binSh pkgs.fakeNss ];
11831183- pathsToLink = [ "/bin" "/etc" "/var" ];
11841184- };
13031303+```nix
13041304+{ dockerTools, hello }:
13051305+dockerTools.buildNixShellImage {
13061306+ drv = hello;
13071307+ tag = "latest";
11851308}
11861309```
1187131011881188-## buildNixShellImage {#ssec-pkgs-dockerTools-buildNixShellImage}
13111311+The result of building this package is a `.tar.gz` file that can be loaded into Docker:
13121312+13131313+```shell
13141314+$ nix-build
13151315+(some output removed for clarity)
13161316+/nix/store/pkj1sgzaz31wl0pbvbg3yp5b3kxndqms-hello-2.12.1-env.tar.gz
13171317+13181318+$ docker load -i /nix/store/pkj1sgzaz31wl0pbvbg3yp5b3kxndqms-hello-2.12.1-env.tar.gz
13191319+(some output removed for clarity)
13201320+Loaded image: hello-2.12.1-env:latest
13211321+```
1189132211901190-Create a Docker image that sets up an environment similar to that of running `nix-shell` on a derivation.
11911191-When run in Docker, this environment somewhat resembles the Nix sandbox typically used by `nix-build`, with a major difference being that access to the internet is allowed.
11921192-It additionally also behaves like an interactive `nix-shell`, running things like `shellHook` and setting an interactive prompt.
11931193-If the derivation is fully buildable (i.e. `nix-build` can be used on it), running `buildDerivation` inside such a Docker image will build the derivation, with all its outputs being available in the correct `/nix/store` paths, pointed to by the respective environment variables like `$out`, etc.
13231323+After starting an interactive container, the derivation can be built by running `buildDerivation`, and the output can be executed as expected:
1194132411951195-::: {.warning}
11961196-The behavior doesn't match `nix-shell` or `nix-build` exactly and this function is known not to work correctly for e.g. fixed-output derivations, content-addressed derivations, impure derivations and other special types of derivations.
13251325+```shell
13261326+$ docker run -it hello-2.12.1-env:latest
13271327+[nix-shell:~]$ buildDerivation
13281328+Running phase: unpackPhase
13291329+unpacking source archive /nix/store/pa10z4ngm0g83kx9mssrqzz30s84vq7k-hello-2.12.1.tar.gz
13301330+source root is hello-2.12.1
13311331+(some output removed for clarity)
13321332+Running phase: fixupPhase
13331333+shrinking RPATHs of ELF executables and libraries in /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1
13341334+shrinking /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1/bin/hello
13351335+checking for references to /build/ in /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1...
13361336+gzipping man pages under /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1/share/man/
13371337+patching script interpreter paths in /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1
13381338+stripping (with command strip and flags -S -p) in /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1/bin
13391339+13401340+[nix-shell:~]$ $out/bin/hello
13411341+Hello, world!
13421342+```
11971343:::
1198134411991199-### Arguments {#ssec-pkgs-dockerTools-buildNixShellImage-arguments}
13451345+## streamNixShellImage {#ssec-pkgs-dockerTools-streamNixShellImage}
1200134612011201-`drv`
13471347+`streamNixShellImage` builds a **script** which, when run, will stream to stdout a Docker-compatible repository tarball of an image that sets up an environment similar to that of running `nix-shell` on a derivation.
13481348+This means that `streamNixShellImage` does not output an image into the Nix store, but only a script that builds the image, saving on IO and disk/cache space, particularly with large images.
13491349+See [](#ex-dockerTools-streamNixShellImage-hello) to understand how to load in Docker the image generated by this script.
1202135012031203-: The derivation on which to base the Docker image.
13511351+The environment set up by `streamNixShellImage` somewhat resembles the Nix sandbox typically used by `nix-build`, with a major difference being that access to the internet is allowed.
13521352+It also behaves like an interactive `nix-shell`, running things like `shellHook` (see [](#ex-dockerTools-streamNixShellImage-addingShellHook)) and setting an interactive prompt.
13531353+If the derivation is buildable (i.e. `nix-build` can be used on it), running `buildDerivation` in the container will build the derivation, with all its outputs being available in the correct `/nix/store` paths, pointed to by the respective environment variables (e.g. `$out`).
1204135412051205- Adding packages to the Docker image is possible by e.g. extending the list of `nativeBuildInputs` of this derivation like
13551355+::: {.caution}
13561356+The environment in the image doesn't match `nix-shell` or `nix-build` exactly, and this function is known not to work correctly for fixed-output derivations, content-addressed derivations, impure derivations and other special types of derivations.
13571357+:::
1206135812071207- ```nix
12081208- buildNixShellImage {
12091209- drv = someDrv.overrideAttrs (old: {
12101210- nativeBuildInputs = old.nativeBuildInputs or [] ++ [
12111211- somethingExtra
12121212- ];
12131213- });
12141214- # ...
12151215- }
12161216- ```
13591359+### Inputs {#ssec-pkgs-dockerTools-streamNixShellImage-inputs}
1217136012181218- Similarly, you can extend the image initialization script by extending `shellHook`
13611361+`streamNixShellImage` expects one argument with the following attributes:
1219136212201220-`name` _optional_
13631363+`drv` (Attribute Set)
1221136412221222-: The name of the resulting image.
13651365+: The derivation for which the environment in the image will be set up.
13661366+ Adding packages to the Docker image is possible by extending the list of `nativeBuildInputs` of this derivation.
13671367+ See [](#ex-dockerTools-streamNixShellImage-extendingBuildInputs) for how to do that.
13681368+ Similarly, you can extend the image initialization script by extending `shellHook`.
13691369+ [](#ex-dockerTools-streamNixShellImage-addingShellHook) shows how to do that.
1223137012241224- *Default:* `drv.name + "-env"`
13711371+`name` (String; _optional_)
1225137212261226-`tag` _optional_
13731373+: The name of the generated image.
13741374+13751375+ _Default value:_ the value of `drv.name + "-env"`.
13761376+13771377+`tag` (String or Null; _optional_)
1227137812281379: Tag of the generated image.
13801380+ If `null`, the hash of the nix derivation that builds the Docker image will be used as the tag.
13811381+13821382+ _Default value:_ `null`.
1229138312301230- *Default:* the resulting image derivation output path's hash
13841384+`uid` (Number; _optional_)
13851385+13861386+: The user ID to run the container as.
13871387+ This can be seen as a `nixbld` build user.
13881388+13891389+ _Default value:_ 1000.
13901390+13911391+`gid` (Number; _optional_)
1231139212321232-`uid`/`gid` _optional_
13931393+: The group ID to run the container as.
13941394+ This can be seen as a `nixbld` build group.
1233139512341234-: The user/group ID to run the container as. This is like a `nixbld` build user.
13961396+ _Default value:_ 1000.
1235139712361236- *Default:* 1000/1000
13981398+`homeDirectory` (String; _optional_)
1237139912381238-`homeDirectory` _optional_
14001400+: The home directory of the user the container is running as.
1239140112401240-: The home directory of the user the container is running as
14021402+ _Default value:_ `/build`.
1241140312421242- *Default:* `/build`
14041404+`shell` (String; _optional_)
1243140512441244-`shell` _optional_
14061406+: The path to the `bash` binary to use as the shell.
14071407+ This shell is started when running the image.
14081408+ This can be seen as an equivalent of the `NIX_BUILD_SHELL` [environment variable](https://nixos.org/manual/nix/stable/command-ref/nix-shell.html#environment-variables) for {manpage}`nix-shell(1)`.
1245140912461246-: The path to the `bash` binary to use as the shell. This shell is started when running the image.
14101410+ _Default value:_ the `bash` binary from the `bashInteractive` package.
1247141112481248- *Default:* `pkgs.bashInteractive + "/bin/bash"`
14121412+`command` (String or Null; _optional_)
1249141312501250-`command` _optional_
14141414+: If specified, this command will be run in the environment of the derivation in an interactive shell.
14151415+ A call to `exit` will be added after the command if it is specified, so the shell will exit after it's finished running.
14161416+ This can be seen as an equivalent of the `--command` option in {manpage}`nix-shell(1)`.
1251141712521252-: Run this command in the environment of the derivation, in an interactive shell. See the `--command` option in the [`nix-shell` documentation](https://nixos.org/manual/nix/stable/command-ref/nix-shell.html?highlight=nix-shell#options).
14181418+ _Default value:_ `null`.
1253141912541254- *Default:* (none)
14201420+`run` (String or Null; _optional_)
1255142112561256-`run` _optional_
14221422+: Similar to the `command` attribute, but runs the command in a non-interactive shell instead.
14231423+ A call to `exit` will be added after the command if it is specified, so the shell will exit after it's finished running.
14241424+ This can be seen as an equivalent of the `--run` option in {manpage}`nix-shell(1)`.
1257142512581258-: Same as `command`, but runs the command in a non-interactive shell instead. See the `--run` option in the [`nix-shell` documentation](https://nixos.org/manual/nix/stable/command-ref/nix-shell.html?highlight=nix-shell#options).
14261426+ _Default value:_ `null`.
1259142712601260- *Default:* (none)
14281428+### Examples {#ssec-pkgs-dockerTools-streamNixShellImage-examples}
1261142912621262-### Example {#ssec-pkgs-dockerTools-buildNixShellImage-example}
14301430+:::{.example #ex-dockerTools-streamNixShellImage-hello}
14311431+# Building a Docker image with `streamNixShellImage` with the build environment for the `hello` package
1263143212641264-The following shows how to build the `pkgs.hello` package inside a Docker container built with `buildNixShellImage`.
14331433+This example shows how to build the `hello` package inside a Docker container built with `streamNixShellImage`.
14341434+The Docker image generated will have a name like `hello-<version>-env` and tag `latest`.
14351435+This example is the `streamNixShellImage` equivalent of [](#ex-dockerTools-buildNixShellImage-hello).
1265143612661437```nix
12671267-with import <nixpkgs> {};
12681268-dockerTools.buildNixShellImage {
14381438+{ dockerTools, hello }:
14391439+dockerTools.streamNixShellImage {
12691440 drv = hello;
14411441+ tag = "latest";
12701442}
12711443```
1272144412731273-Build the derivation:
14451445+The result of building this package is a script.
14461446+Running this script and piping it into `docker load` gives you the same image that was built in [](#ex-dockerTools-buildNixShellImage-hello).
14471447+14481448+```shell
14491449+$ nix-build
14501450+(some output removed for clarity)
14511451+/nix/store/8vhznpz2frqazxnd8pgdvf38jscdypax-stream-hello-2.12.1-env
1274145212751275-```console
12761276-nix-build hello.nix
14531453+$ /nix/store/8vhznpz2frqazxnd8pgdvf38jscdypax-stream-hello-2.12.1-env | docker load
14541454+(some output removed for clarity)
14551455+Loaded image: hello-2.12.1-env:latest
12771456```
1278145712791279- these 8 derivations will be built:
12801280- /nix/store/xmw3a5ln29rdalavcxk1w3m4zb2n7kk6-nix-shell-rc.drv
12811281- ...
12821282- Creating layer 56 from paths: ['/nix/store/crpnj8ssz0va2q0p5ibv9i6k6n52gcya-stdenv-linux']
12831283- Creating layer 57 with customisation...
12841284- Adding manifests...
12851285- Done.
12861286- /nix/store/cpyn1lc897ghx0rhr2xy49jvyn52bazv-hello-2.12-env.tar.gz
14581458+After starting an interactive container, the derivation can be built by running `buildDerivation`, and the output can be executed as expected:
1287145912881288-Load the image:
14601460+```shell
14611461+$ docker run -it hello-2.12.1-env:latest
14621462+[nix-shell:~]$ buildDerivation
14631463+Running phase: unpackPhase
14641464+unpacking source archive /nix/store/pa10z4ngm0g83kx9mssrqzz30s84vq7k-hello-2.12.1.tar.gz
14651465+source root is hello-2.12.1
14661466+(some output removed for clarity)
14671467+Running phase: fixupPhase
14681468+shrinking RPATHs of ELF executables and libraries in /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1
14691469+shrinking /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1/bin/hello
14701470+checking for references to /build/ in /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1...
14711471+gzipping man pages under /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1/share/man/
14721472+patching script interpreter paths in /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1
14731473+stripping (with command strip and flags -S -p) in /nix/store/f2vs29jibd7lwxyj35r9h87h6brgdysz-hello-2.12.1/bin
1289147412901290-```console
12911291-docker load -i result
14751475+[nix-shell:~]$ $out/bin/hello
14761476+Hello, world!
12921477```
14781478+:::
1293147912941294- 0d9f4c4cd109: Loading layer [==================================================>] 2.56MB/2.56MB
12951295- ...
12961296- ab1d897c0697: Loading layer [==================================================>] 10.24kB/10.24kB
12971297- Loaded image: hello-2.12-env:pgj9h98nal555415faa43vsydg161bdz
14801480+:::{.example #ex-dockerTools-streamNixShellImage-extendingBuildInputs}
14811481+# Adding extra packages to a Docker image built with `streamNixShellImage`
14821482+14831483+This example shows how to add extra packages to an image built with `streamNixShellImage`.
14841484+In this case, we'll add the `cowsay` package.
14851485+The Docker image generated will have a name like `hello-<version>-env` and tag `latest`.
14861486+This example uses [](#ex-dockerTools-streamNixShellImage-hello) as a starting point.
14871487+14881488+```nix
14891489+{ dockerTools, cowsay, hello }:
14901490+dockerTools.streamNixShellImage {
14911491+ tag = "latest";
14921492+ drv = hello.overrideAttrs (old: {
14931493+ nativeBuildInputs = old.nativeBuildInputs or [] ++ [
14941494+ cowsay
14951495+ ];
14961496+ });
14971497+}
14981498+```
14991499+15001500+The result of building this package is a script which can be run and piped into `docker load` to load the generated image.
15011501+15021502+```shell
15031503+$ nix-build
15041504+(some output removed for clarity)
15051505+/nix/store/h5abh0vljgzg381lna922gqknx6yc0v7-stream-hello-2.12.1-env
15061506+15071507+$ /nix/store/h5abh0vljgzg381lna922gqknx6yc0v7-stream-hello-2.12.1-env | docker load
15081508+(some output removed for clarity)
15091509+Loaded image: hello-2.12.1-env:latest
15101510+```
1298151112991299-Run the container:
15121512+After starting an interactive container, we can verify the extra package is available by running `cowsay`:
1300151313011301-```console
13021302-docker run -it hello-2.12-env:pgj9h98nal555415faa43vsydg161bdz
15141514+```shell
15151515+$ docker run -it hello-2.12.1-env:latest
15161516+[nix-shell:~]$ cowsay "Hello, world!"
15171517+ _______________
15181518+< Hello, world! >
15191519+ ---------------
15201520+ \ ^__^
15211521+ \ (oo)\_______
15221522+ (__)\ )\/\
15231523+ ||----w |
15241524+ || ||
13031525```
15261526+:::
1304152713051305- [nix-shell:/build]$
15281528+:::{.example #ex-dockerTools-streamNixShellImage-addingShellHook}
15291529+# Adding a `shellHook` to a Docker image built with `streamNixShellImage`
1306153013071307-In the running container, run the build:
15311531+This example shows how to add a `shellHook` command to an image built with `streamNixShellImage`.
15321532+In this case, we'll simply output the string `Hello, world!`.
15331533+The Docker image generated will have a name like `hello-<version>-env` and tag `latest`.
15341534+This example uses [](#ex-dockerTools-streamNixShellImage-hello) as a starting point.
1308153513091309-```console
13101310-buildDerivation
15361536+```nix
15371537+{ dockerTools, hello }:
15381538+dockerTools.streamNixShellImage {
15391539+ tag = "latest";
15401540+ drv = hello.overrideAttrs (old: {
15411541+ shellHook = ''
15421542+ ${old.shellHook or ""}
15431543+ echo "Hello, world!"
15441544+ '';
15451545+ });
15461546+}
13111547```
1312154813131313- unpacking sources
13141314- unpacking source archive /nix/store/8nqv6kshb3vs5q5bs2k600xpj5bkavkc-hello-2.12.tar.gz
13151315- ...
13161316- patching script interpreter paths in /nix/store/z5wwy5nagzy15gag42vv61c2agdpz2f2-hello-2.12
13171317- checking for references to /build/ in /nix/store/z5wwy5nagzy15gag42vv61c2agdpz2f2-hello-2.12...
15491549+The result of building this package is a script which can be run and piped into `docker load` to load the generated image.
1318155013191319-Check the build result:
15511551+```shell
15521552+$ nix-build
15531553+(some output removed for clarity)
15541554+/nix/store/iz4dhdvgzazl5vrgyz719iwjzjy6xlx1-stream-hello-2.12.1-env
1320155513211321-```console
13221322-$out/bin/hello
15561556+$ /nix/store/iz4dhdvgzazl5vrgyz719iwjzjy6xlx1-stream-hello-2.12.1-env | docker load
15571557+(some output removed for clarity)
15581558+Loaded image: hello-2.12.1-env:latest
13231559```
1324156013251325- Hello, world!
15611561+After starting an interactive container, we can see the result of the `shellHook`:
15621562+15631563+```shell
15641564+$ docker run -it hello-2.12.1-env:latest
15651565+Hello, world!
15661566+15671567+[nix-shell:~]$
15681568+```
15691569+:::
+1
doc/build-helpers/special.md
···33This chapter describes several special build helpers.
4455```{=include=} sections
66+special/fakenss.section.md
67special/fhs-environments.section.md
78special/makesetuphook.section.md
89special/mkshell.section.md
+77
doc/build-helpers/special/fakenss.section.md
···11+# fakeNss {#sec-fakeNss}
22+33+Provides `/etc/passwd` and `/etc/group` files that contain `root` and `nobody`, allowing user/group lookups to work in binaries that insist on doing those.
44+This might be a better choice than a custom script running `useradd` and related utilities if you only need those files to exist with some entries.
55+66+`fakeNss` also provides `/etc/nsswitch.conf`, configuring NSS host resolution to first check `/etc/hosts` before checking DNS, since the default in the absence of a config file (`dns [!UNAVAIL=return] files`) is quite unexpected.
77+88+It also creates an empty directory at `/var/empty` because it uses that as the home directory for the `root` and `nobody` users.
99+The `/var/empty` directory can also be used as a `chroot` target to prevent file access in processes that do not need to access files, if your container runs such processes.
1010+1111+The user entries created by `fakeNss` use the `/bin/sh` shell, which is not provided by `fakeNss` because in most cases it won't be used.
1212+If you need that to be available, see [`dockerTools.binSh`](#sssec-pkgs-dockerTools-helpers-binSh) or provide your own.
1313+1414+## Inputs {#sec-fakeNss-inputs}
1515+1616+`fakeNss` is made available in Nixpkgs as a package rather than a function, but it has two attributes that can be overridden and might be useful in particular cases.
1717+For more details on how overriding works, see [](#ex-fakeNss-overriding) and [](#sec-pkg-override).
1818+1919+`extraPasswdLines` (List of Strings; _optional_)
2020+2121+: A list of lines that will be added to `/etc/passwd`.
2222+ Useful if extra users need to exist in the output of `fakeNss`.
2323+ If `extraPasswdLines` is specified, it will **not** override the `root` and `nobody` entries created by `fakeNss`.
2424+ Those entries will always exist.
2525+2626+ Lines specified here must follow the format in {manpage}`passwd(5)`.
2727+2828+ _Default value:_ `[]`.
2929+3030+`extraGroupLines` (List of Strings; _optional_)
3131+3232+: A list of lines that will be added to `/etc/group`.
3333+ Useful if extra groups need to exist in the output of `fakeNss`.
3434+ If `extraGroupLines` is specified, it will **not** override the `root` and `nobody` entries created by `fakeNss`.
3535+ Those entries will always exist.
3636+3737+ Lines specified here must follow the format in {manpage}`group(5)`.
3838+3939+ _Default value:_ `[]`.
4040+4141+## Examples {#sec-fakeNss-examples}
4242+4343+:::{.example #ex-fakeNss-dockerTools-buildImage}
4444+# Using `fakeNss` with `dockerTools.buildImage`
4545+4646+This example shows how to use `fakeNss` as-is.
4747+It is useful with functions in `dockerTools` to allow building Docker images that have the `/etc/passwd` and `/etc/group` files.
4848+This example includes the `hello` binary in the image so it can do something besides just have the extra files.
4949+5050+```nix
5151+{ dockerTools, fakeNss, hello }:
5252+dockerTools.buildImage {
5353+ name = "image-with-passwd";
5454+ tag = "latest";
5555+5656+ copyToRoot = [ fakeNss hello ];
5757+5858+ config = {
5959+ Cmd = [ "/bin/hello" ];
6060+ };
6161+}
6262+```
6363+:::
6464+6565+:::{.example #ex-fakeNss-overriding}
6666+# Using `fakeNss` with an override to add extra lines
6767+6868+The following code uses `override` to add extra lines to `/etc/passwd` and `/etc/group` to create another user and group entry.
6969+7070+```nix
7171+{ fakeNss }:
7272+fakeNss.override {
7373+ extraPasswdLines = ["newuser:x:9001:9001:new user:/var/empty:/bin/sh"];
7474+ extraGroupLines = ["newuser:x:9001:"];
7575+}
7676+```
7777+:::
+1-1
doc/hooks/installShellFiles.section.md
···2233This hook helps with installing manpages and shell completion files. It exposes 2 shell functions `installManPage` and `installShellCompletion` that can be used from your `postInstall` hook.
4455-The `installManPage` function takes one or more paths to manpages to install. The manpages must have a section suffix, and may optionally be compressed (with `.gz` suffix). This function will place them into the correct directory.
55+The `installManPage` function takes one or more paths to manpages to install. The manpages must have a section suffix, and may optionally be compressed (with `.gz` suffix). This function will place them into the correct `share/man/man<section>/` directory, in [`outputMan`](#outputman).
6677The `installShellCompletion` function takes one or more paths to shell completion files. By default it will autodetect the shell type from the completion file extension, but you may also specify it by passing one of `--bash`, `--fish`, or `--zsh`. These flags apply to all paths listed after them (up until another shell flag is given). Each path may also have a custom installation name provided by providing a flag `--name NAME` before the path. If this flag is not provided, zsh completions will be renamed automatically such that `foobar.zsh` becomes `_foobar`. A root name may be provided for all paths using the flag `--cmd NAME`; this synthesizes the appropriate name depending on the shell (e.g. `--cmd foo` will synthesize the name `foo.bash` for bash and `_foo` for zsh). The path may also be a fifo or named fd (such as produced by `<(cmd)`), in which case the shell and name must be provided.
88
+46-2
doc/languages-frameworks/dotnet.section.md
···9393To package Dotnet applications, you can use `buildDotnetModule`. This has similar arguments to `stdenv.mkDerivation`, with the following additions:
94949595* `projectFile` is used for specifying the dotnet project file, relative to the source root. These have `.sln` (entire solution) or `.csproj` (single project) file extensions. This can be a list of multiple projects as well. When omitted, will attempt to find and build the solution (`.sln`). If running into problems, make sure to set it to a file (or a list of files) with the `.csproj` extension - building applications as entire solutions is not fully supported by the .NET CLI.
9696-* `nugetDeps` takes either a path to a `deps.nix` file, or a derivation. The `deps.nix` file can be generated using the script attached to `passthru.fetch-deps`. This file can also be generated manually using `nuget-to-nix` tool, which is available in nixpkgs. If the argument is a derivation, it will be used directly and assume it has the same output as `mkNugetDeps`.
9696+* `nugetDeps` takes either a path to a `deps.nix` file, or a derivation. The `deps.nix` file can be generated using the script attached to `passthru.fetch-deps`. If the argument is a derivation, it will be used directly and assume it has the same output as `mkNugetDeps`.
9797+::: {.note}
9898+For more detail about managing the `deps.nix` file, see [Generating and updating NuGet dependencies](#generating-and-updating-nuget-dependencies)
9999+:::
100100+97101* `packNupkg` is used to pack project as a `nupkg`, and installs it to `$out/share`. If set to `true`, the derivation can be used as a dependency for another dotnet project by adding it to `projectReferences`.
98102* `projectReferences` can be used to resolve `ProjectReference` project items. Referenced projects can be packed with `buildDotnetModule` by setting the `packNupkg = true` attribute and passing a list of derivations to `projectReferences`. Since we are sharing referenced projects as NuGets they must be added to csproj/fsproj files as `PackageReference` as well.
99103 For example, your project has a local dependency:
···156160}
157161```
158162163163+Keep in mind that you can tag the [`@NixOS/dotnet`](https://github.com/orgs/nixos/teams/dotnet) team for help and code review.
164164+159165## Dotnet global tools {#dotnet-global-tools}
160166161167[.NET Global tools](https://learn.microsoft.com/en-us/dotnet/core/tools/global-tools) are a mechanism provided by the dotnet CLI to install .NET binaries from Nuget packages.
···212218 };
213219}
214220```
221221+## Generating and updating NuGet dependencies {#generating-and-updating-nuget-dependencies}
215222216216-When packaging a new .NET application in nixpkgs, you can tag the [`@NixOS/dotnet`](https://github.com/orgs/nixos/teams/dotnet) team for help and code review.
223223+First, restore the packages to the `out` directory, ensure you have cloned
224224+the upstream repository and you are inside it.
225225+226226+```bash
227227+$ dotnet restore --packages out
228228+ Determining projects to restore...
229229+ Restored /home/lychee/Celeste64/Celeste64.csproj (in 1.21 sec).
230230+```
231231+232232+Next, use `nuget-to-nix` tool provided in nixpkgs to generate a lockfile to `deps.nix` from
233233+the packages inside the `out` directory.
234234+235235+```bash
236236+$ nuget-to-nix out > deps.nix
237237+```
238238+Which `nuget-to-nix` will generate an output similar to below
239239+```
240240+{ fetchNuGet }: [
241241+ (fetchNuGet { pname = "FosterFramework"; version = "0.1.15-alpha"; sha256 = "0pzsdfbsfx28xfqljcwy100xhbs6wyx0z1d5qxgmv3l60di9xkll"; })
242242+ (fetchNuGet { pname = "Microsoft.AspNetCore.App.Runtime.linux-x64"; version = "8.0.1"; sha256 = "1gjz379y61ag9whi78qxx09bwkwcznkx2mzypgycibxk61g11da1"; })
243243+ (fetchNuGet { pname = "Microsoft.NET.ILLink.Tasks"; version = "8.0.1"; sha256 = "1drbgqdcvbpisjn8mqfgba1pwb6yri80qc4mfvyczqwrcsj5k2ja"; })
244244+ (fetchNuGet { pname = "Microsoft.NETCore.App.Runtime.linux-x64"; version = "8.0.1"; sha256 = "1g5b30f4l8a1zjjr3b8pk9mcqxkxqwa86362f84646xaj4iw3a4d"; })
245245+ (fetchNuGet { pname = "SharpGLTF.Core"; version = "1.0.0-alpha0031"; sha256 = "0ln78mkhbcxqvwnf944hbgg24vbsva2jpih6q3x82d3h7rl1pkh6"; })
246246+ (fetchNuGet { pname = "SharpGLTF.Runtime"; version = "1.0.0-alpha0031"; sha256 = "0lvb3asi3v0n718qf9y367km7qpkb9wci38y880nqvifpzllw0jg"; })
247247+ (fetchNuGet { pname = "Sledge.Formats"; version = "1.2.2"; sha256 = "1y0l66m9rym0p1y4ifjlmg3j9lsmhkvbh38frh40rpvf1axn2dyh"; })
248248+ (fetchNuGet { pname = "Sledge.Formats.Map"; version = "1.1.5"; sha256 = "1bww60hv9xcyxpvkzz5q3ybafdxxkw6knhv97phvpkw84pd0jil6"; })
249249+ (fetchNuGet { pname = "System.Numerics.Vectors"; version = "4.5.0"; sha256 = "1kzrj37yzawf1b19jq0253rcs8hsq1l2q8g69d7ipnhzb0h97m59"; })
250250+]
251251+```
252252+253253+Finally, you move the `deps.nix` file to the appropriate location to be used by `nugetDeps`, then you're all set!
254254+255255+If you ever need to update the dependencies of a package, you instead do
256256+257257+* `nix-build -A package.fetch-deps` to generate the update script for `package`
258258+* Run `./result deps.nix` to regenerate the lockfile to `deps.nix`, keep in mind if a location isn't provided, it will write to a temporary path instead
259259+* Finally, move the file where needed and look at its contents to confirm it has updated the dependencies.
260260+
···805805 '';
806806807807 # This provides /bin/sh, pointing to bashInteractive.
808808+ # The use of bashInteractive here is intentional to support cases like `docker run -it <image_name>`, so keep these use cases in mind if making any changes to how this works.
808809 binSh = runCommand "bin-sh" { } ''
809810 mkdir -p $out/bin
810811 ln -s ${bashInteractive}/bin/bash $out/bin/sh
···77in
88stdenv.mkDerivation rec {
99 pname = "oil";
1010- version = "0.19.0";
1010+ version = "0.20.0";
11111212 src = fetchurl {
1313 url = "https://www.oilshell.org/download/oil-${version}.tar.xz";
1414- hash = "sha256-iCoEFudFqxjYZerhOe7u6bPzN5EUOpwSpWCbTzUmF8U=";
1414+ hash = "sha256-QrhfUru6Sju44W8j/DlMQwK8/ZY48GfwHDfSPy7kSaA=";
1515 };
16161717 postPatch = ''
···24242525 strictDeps = true;
2626 buildInputs = lib.optional withReadline readline;
2727- # As of 0.19.0 the build generates an error on MacOS (using clang version 16.0.6 in the builder),
2727+ # As of 0.20.0 the build generates an error on MacOS (using clang version 16.0.6 in the builder),
2828 # whereas running it outside of Nix with clang version 15.0.0 generates just a warning. The shell seems to
2929 # work just fine though, so we disable the error here.
3030 env.NIX_CFLAGS_COMPILE = lib.optionalString stdenv.cc.isClang "-Wno-error=incompatible-function-pointer-types";
+38-38
pkgs/test/nixpkgs-check-by-name/src/eval.nix
···11-# Takes a path to nixpkgs and a path to the json-encoded list of attributes to check.
22-# Returns an value containing information on each requested attribute,
11+# Takes a path to nixpkgs and a path to the json-encoded list of `pkgs/by-name` attributes.
22+# Returns a value containing information on all Nixpkgs attributes
33# which is decoded on the Rust side.
44# See ./eval.rs for the meaning of the returned values
55{
···99let
1010 attrs = builtins.fromJSON (builtins.readFile attrsPath);
11111212- nixpkgsPathLength = builtins.stringLength (toString nixpkgsPath) + 1;
1313- removeNixpkgsPrefix = builtins.substring nixpkgsPathLength (-1);
1414-1515- # We need access to the `callPackage` arguments of each attribute.
1616- # The only way to do so is to override `callPackage` with our own version that adds this information to the result,
1717- # and then try to access this information.
1212+ # We need to check whether attributes are defined manually e.g. in
1313+ # `all-packages.nix`, automatically by the `pkgs/by-name` overlay, or
1414+ # neither. The only way to do so is to override `callPackage` and
1515+ # `_internalCallByNamePackageFile` with our own version that adds this
1616+ # information to the result, and then try to access it.
1817 overlay = final: prev: {
19182020- # Information for attributes defined using `callPackage`
1919+ # Adds information to each attribute about whether it's manually defined using `callPackage`
2120 callPackage = fn: args:
2221 addVariantInfo (prev.callPackage fn args) {
2323- Manual = {
2424- path =
2525- if builtins.isPath fn then
2626- removeNixpkgsPrefix (toString fn)
2727- else
2828- null;
2929- empty_arg =
3030- args == { };
3131- };
2222+ # This is a manual definition of the attribute, and it's a callPackage, specifically a semantic callPackage
2323+ ManualDefinition.is_semantic_call_package = true;
3224 };
33253434- # Information for attributes that are auto-called from pkgs/by-name.
3535- # This internal attribute is only used by pkgs/by-name
2626+ # Adds information to each attribute about whether it's automatically
2727+ # defined by the `pkgs/by-name` overlay. This internal attribute is only
2828+ # used by that overlay.
2929+ # This overrides the above `callPackage` information (we don't need that
3030+ # one, since `pkgs/by-name` always uses `callPackage` underneath.
3631 _internalCallByNamePackageFile = file:
3732 addVariantInfo (prev._internalCallByNamePackageFile file) {
3838- Auto = null;
3333+ AutoDefinition = null;
3934 };
40354136 };
···5045 else
5146 # It's very rare that callPackage doesn't return an attribute set, but it can occur.
5247 # In such a case we can't really return anything sensible that would include the info,
5353- # so just don't return the info and let the consumer handle it.
4848+ # so just don't return the value directly and treat it as if it wasn't a callPackage.
5449 value;
55505651 pkgs = import nixpkgsPath {
···6257 system = "x86_64-linux";
6358 };
64596565- attrInfo = name: value:
6666- if ! builtins.isAttrs value then
6767- {
6868- NonAttributeSet = null;
6969- }
7070- else if ! value ? _callPackageVariant then
7171- {
7272- NonCallPackage = null;
7373- }
7474- else
7575- {
7676- CallPackage = {
7777- call_package_variant = value._callPackageVariant;
7878- is_derivation = pkgs.lib.isDerivation value;
7979- location = builtins.unsafeGetAttrPos name pkgs;
6060+ # See AttributeInfo in ./eval.rs for the meaning of this
6161+ attrInfo = name: value: {
6262+ location = builtins.unsafeGetAttrPos name pkgs;
6363+ attribute_variant =
6464+ if ! builtins.isAttrs value then
6565+ { NonAttributeSet = null; }
6666+ else
6767+ {
6868+ AttributeSet = {
6969+ is_derivation = pkgs.lib.isDerivation value;
7070+ definition_variant =
7171+ if ! value ? _callPackageVariant then
7272+ { ManualDefinition.is_semantic_call_package = false; }
7373+ else
7474+ value._callPackageVariant;
7575+ };
8076 };
8181- };
7777+ };
82787979+ # Information on all attributes that are in pkgs/by-name.
8380 byNameAttrs = builtins.listToAttrs (map (name: {
8481 inherit name;
8582 value.ByName =
8683 if ! pkgs ? ${name} then
8784 { Missing = null; }
8885 else
8686+ # Evaluation failures are not allowed, so don't try to catch them
8987 { Existing = attrInfo name pkgs.${name}; };
9088 }) attrs);
9189···9391 # We need this to enforce pkgs/by-name for new packages
9492 nonByNameAttrs = builtins.mapAttrs (name: value:
9593 let
9494+ # Packages outside `pkgs/by-name` often fail evaluation,
9595+ # so we need to handle that
9696 output = attrInfo name value;
9797 result = builtins.tryEval (builtins.deepSeq output null);
9898 in
+278-181
pkgs/test/nixpkgs-check-by-name/src/eval.rs
···3737}
38383939#[derive(Deserialize)]
4040-enum AttributeInfo {
4141- /// The attribute exists, but its value isn't an attribute set
4242- NonAttributeSet,
4343- /// The attribute exists, but its value isn't defined using callPackage
4444- NonCallPackage,
4545- /// The attribute exists and its value is an attribute set
4646- CallPackage(CallPackageInfo),
4747-}
4848-4949-#[derive(Deserialize)]
5050-struct CallPackageInfo {
5151- call_package_variant: CallPackageVariant,
5252- /// Whether the attribute is a derivation (`lib.isDerivation`)
5353- is_derivation: bool,
4040+struct AttributeInfo {
4141+ /// The location of the attribute as returned by `builtins.unsafeGetAttrPos`
5442 location: Option<Location>,
4343+ attribute_variant: AttributeVariant,
5544}
56455757-/// The structure returned by `builtins.unsafeGetAttrPos`
4646+/// The structure returned by a successful `builtins.unsafeGetAttrPos`
5847#[derive(Deserialize, Clone, Debug)]
5948struct Location {
6049 pub file: PathBuf,
···6352}
64536554#[derive(Deserialize)]
6666-enum CallPackageVariant {
6767- /// The attribute is auto-called as pkgs.callPackage using pkgs/by-name,
6868- /// and it is not overridden by a definition in all-packages.nix
6969- Auto,
7070- /// The attribute is defined as a pkgs.callPackage <path> <args>,
7171- /// and overridden by all-packages.nix
7272- Manual {
7373- /// The <path> argument or None if it's not a path
7474- path: Option<PathBuf>,
7575- /// true if <args> is { }
7676- empty_arg: bool,
5555+pub enum AttributeVariant {
5656+ /// The attribute is not an attribute set, we're limited in the amount of information we can get
5757+ /// from it (though it's obviously not a derivation)
5858+ NonAttributeSet,
5959+ AttributeSet {
6060+ /// Whether the attribute is a derivation (`lib.isDerivation`)
6161+ is_derivation: bool,
6262+ /// The type of callPackage
6363+ definition_variant: DefinitionVariant,
6464+ },
6565+}
6666+6767+#[derive(Deserialize)]
6868+pub enum DefinitionVariant {
6969+ /// An automatic definition by the `pkgs/by-name` overlay
7070+ /// Though it's detected using the internal _internalCallByNamePackageFile attribute,
7171+ /// which can in theory also be used by other code
7272+ AutoDefinition,
7373+ /// A manual definition of the attribute, typically in `all-packages.nix`
7474+ ManualDefinition {
7575+ /// Whether the attribute is defined as `pkgs.callPackage ...` or something else.
7676+ is_semantic_call_package: bool,
7777 },
7878}
7979···165165 nix_file_store,
166166 non_by_name_attribute,
167167 )?,
168168- Attribute::ByName(by_name_attribute) => {
169169- by_name(&attribute_name, by_name_attribute)
170170- }
168168+ Attribute::ByName(by_name_attribute) => by_name(
169169+ nix_file_store,
170170+ nixpkgs_path,
171171+ &attribute_name,
172172+ by_name_attribute,
173173+ )?,
171174 };
172175 Ok::<_, anyhow::Error>(check_result.map(|value| (attribute_name.clone(), value)))
173176 })
···183186/// Handles the evaluation result for an attribute in `pkgs/by-name`,
184187/// turning it into a validation result.
185188fn by_name(
189189+ nix_file_store: &mut NixFileStore,
190190+ nixpkgs_path: &Path,
186191 attribute_name: &str,
187192 by_name_attribute: ByNameAttribute,
188188-) -> validation::Validation<ratchet::Package> {
193193+) -> validation::Result<ratchet::Package> {
189194 use ratchet::RatchetState::*;
190190- use AttributeInfo::*;
191195 use ByNameAttribute::*;
192192- use CallPackageVariant::*;
193196194197 let relative_package_file = structure::relative_file_for_package(attribute_name);
198198+ let absolute_package_file = nixpkgs_path.join(&relative_package_file);
195199196196- match by_name_attribute {
197197- Missing => NixpkgsProblem::UndefinedAttr {
198198- relative_package_file: relative_package_file.to_owned(),
199199- package_name: attribute_name.to_owned(),
200200+ // At this point we know that `pkgs/by-name/fo/foo/package.nix` has to exists.
201201+ // This match decides whether the attribute `foo` is defined accordingly
202202+ // and whether a legacy manual definition could be removed
203203+ let manual_definition_result = match by_name_attribute {
204204+ // The attribute is missing
205205+ Missing => {
206206+ // This indicates a bug in the `pkgs/by-name` overlay, because it's supposed to
207207+ // automatically defined attributes in `pkgs/by-name`
208208+ NixpkgsProblem::UndefinedAttr {
209209+ relative_package_file: relative_package_file.to_owned(),
210210+ package_name: attribute_name.to_owned(),
211211+ }
212212+ .into()
200213 }
201201- .into(),
202202- Existing(NonAttributeSet) => NixpkgsProblem::NonDerivation {
203203- relative_package_file: relative_package_file.to_owned(),
204204- package_name: attribute_name.to_owned(),
205205- }
206206- .into(),
207207- Existing(NonCallPackage) => NixpkgsProblem::WrongCallPackage {
208208- relative_package_file: relative_package_file.to_owned(),
209209- package_name: attribute_name.to_owned(),
214214+ // The attribute exists
215215+ Existing(AttributeInfo {
216216+ // But it's not an attribute set, which limits the amount of information we can get
217217+ // about this attribute (see ./eval.nix)
218218+ attribute_variant: AttributeVariant::NonAttributeSet,
219219+ location: _location,
220220+ }) => {
221221+ // The only thing we know is that it's definitely not a derivation, since those are
222222+ // always attribute sets.
223223+ //
224224+ // We can't know whether the attribute is automatically or manually defined for sure,
225225+ // and while we could check the location, the error seems clear enough as is.
226226+ NixpkgsProblem::NonDerivation {
227227+ relative_package_file: relative_package_file.to_owned(),
228228+ package_name: attribute_name.to_owned(),
229229+ }
230230+ .into()
210231 }
211211- .into(),
212212- Existing(CallPackage(CallPackageInfo {
213213- is_derivation,
214214- call_package_variant,
215215- ..
216216- })) => {
217217- let check_result = if !is_derivation {
232232+ // The attribute exists
233233+ Existing(AttributeInfo {
234234+ // And it's an attribute set, which allows us to get more information about it
235235+ attribute_variant:
236236+ AttributeVariant::AttributeSet {
237237+ is_derivation,
238238+ definition_variant,
239239+ },
240240+ location,
241241+ }) => {
242242+ // Only derivations are allowed in `pkgs/by-name`
243243+ let is_derivation_result = if is_derivation {
244244+ Success(())
245245+ } else {
218246 NixpkgsProblem::NonDerivation {
219247 relative_package_file: relative_package_file.to_owned(),
220248 package_name: attribute_name.to_owned(),
221249 }
222250 .into()
223223- } else {
224224- Success(())
225251 };
226252227227- check_result.and(match &call_package_variant {
228228- Auto => Success(ratchet::Package {
229229- manual_definition: Tight,
230230- uses_by_name: Tight,
231231- }),
232232- // TODO: Use the call_package_argument_info_at instead/additionally and
233233- // simplify the eval.nix code
234234- Manual { path, empty_arg } => {
235235- let correct_file = if let Some(call_package_path) = path {
236236- relative_package_file == *call_package_path
253253+ // If the definition looks correct
254254+ let variant_result = match definition_variant {
255255+ // An automatic `callPackage` by the `pkgs/by-name` overlay.
256256+ // Though this gets detected by checking whether the internal
257257+ // `_internalCallByNamePackageFile` was used
258258+ DefinitionVariant::AutoDefinition => {
259259+ if let Some(_location) = location {
260260+ // Such an automatic definition should definitely not have a location
261261+ // Having one indicates that somebody is using `_internalCallByNamePackageFile`,
262262+ NixpkgsProblem::InternalCallPackageUsed {
263263+ attr_name: attribute_name.to_owned(),
264264+ }
265265+ .into()
237266 } else {
238238- false
239239- };
267267+ Success(Tight)
268268+ }
269269+ }
270270+ // The attribute is manually defined, e.g. in `all-packages.nix`.
271271+ // This means we need to enforce it to look like this:
272272+ // callPackage ../pkgs/by-name/fo/foo/package.nix { ... }
273273+ DefinitionVariant::ManualDefinition {
274274+ is_semantic_call_package,
275275+ } => {
276276+ // We should expect manual definitions to have a location, otherwise we can't
277277+ // enforce the expected format
278278+ if let Some(location) = location {
279279+ // Parse the Nix file in the location and figure out whether it's an
280280+ // attribute definition of the form `= callPackage <arg1> <arg2>`,
281281+ // returning the arguments if so.
282282+ let optional_syntactic_call_package = nix_file_store
283283+ .get(&location.file)?
284284+ .call_package_argument_info_at(
285285+ location.line,
286286+ location.column,
287287+ // We're passing `pkgs/by-name/fo/foo/package.nix` here, which causes
288288+ // the function to verify that `<arg1>` is the same path,
289289+ // making `syntactic_call_package.relative_path` end up as `""`
290290+ // TODO: This is confusing and should be improved
291291+ &absolute_package_file,
292292+ )?;
240293241241- if correct_file {
242242- Success(ratchet::Package {
243243- // Empty arguments for non-auto-called packages are not allowed anymore.
244244- manual_definition: if *empty_arg { Loose(()) } else { Tight },
245245- uses_by_name: Tight,
246246- })
294294+ // At this point, we completed two different checks for whether it's a
295295+ // `callPackage`
296296+ match (is_semantic_call_package, optional_syntactic_call_package) {
297297+ // Something like `<attr> = { ... }`
298298+ // or a `pkgs.callPackage` but with the wrong file
299299+ (false, None)
300300+ // Something like `<attr> = pythonPackages.callPackage ./pkgs/by-name/...`
301301+ | (false, Some(_))
302302+ // Something like `<attr> = bar` where `bar = pkgs.callPackage ...`
303303+ // or a `callPackage` but with the wrong file
304304+ | (true, None) => {
305305+ // All of these are not of the expected form, so error out
306306+ // TODO: Make error more specific, don't lump everything together
307307+ NixpkgsProblem::WrongCallPackage {
308308+ relative_package_file: relative_package_file.to_owned(),
309309+ package_name: attribute_name.to_owned(),
310310+ }.into()
311311+ }
312312+ // Something like `<attr> = pkgs.callPackage ./pkgs/by-name/...`,
313313+ // with the correct file
314314+ (true, Some(syntactic_call_package)) => {
315315+ Success(
316316+ // Manual definitions with empty arguments are not allowed
317317+ // anymore
318318+ if syntactic_call_package.empty_arg {
319319+ Loose(())
320320+ } else {
321321+ Tight
322322+ }
323323+ )
324324+ }
325325+ }
247326 } else {
248248- NixpkgsProblem::WrongCallPackage {
249249- relative_package_file: relative_package_file.to_owned(),
250250- package_name: attribute_name.to_owned(),
327327+ // If manual definitions don't have a location, it's likely `mapAttrs`'d
328328+ // over, e.g. if it's defined in aliases.nix.
329329+ // We can't verify whether its of the expected `callPackage`, so error out
330330+ NixpkgsProblem::CannotDetermineAttributeLocation {
331331+ attr_name: attribute_name.to_owned(),
251332 }
252333 .into()
253334 }
254335 }
255255- })
336336+ };
337337+338338+ // Independently report problems about whether it's a derivation and the callPackage variant
339339+ is_derivation_result.and(variant_result)
256340 }
257257- }
341341+ };
342342+ Ok(
343343+ // Packages being checked in this function are _always_ already defined in `pkgs/by-name`,
344344+ // so instead of repeating ourselves all the time to define `uses_by_name`, just set it
345345+ // once at the end with a map
346346+ manual_definition_result.map(|manual_definition| ratchet::Package {
347347+ manual_definition,
348348+ uses_by_name: Tight,
349349+ }),
350350+ )
258351}
259352260353/// Handles the evaluation result for an attribute _not_ in `pkgs/by-name`,
···265358 non_by_name_attribute: NonByNameAttribute,
266359) -> validation::Result<ratchet::Package> {
267360 use ratchet::RatchetState::*;
268268- use AttributeInfo::*;
269269- use CallPackageVariant::*;
270361 use NonByNameAttribute::*;
271362272272- Ok(match non_by_name_attribute {
273273- // The attribute succeeds evaluation and is NOT defined in pkgs/by-name
274274- EvalSuccess(attribute_info) => {
275275- let uses_by_name = match attribute_info {
276276- // In these cases the package doesn't qualify for being in pkgs/by-name,
277277- // so the UsesByName ratchet is already as tight as it can be
278278- NonAttributeSet => Success(NonApplicable),
279279- NonCallPackage => Success(NonApplicable),
280280- // This is the case when the `pkgs/by-name`-internal _internalCallByNamePackageFile
281281- // is used for a package outside `pkgs/by-name`
282282- CallPackage(CallPackageInfo {
283283- call_package_variant: Auto,
284284- ..
285285- }) => {
286286- // With the current detection mechanism, this also triggers for aliases
287287- // to pkgs/by-name packages, and there's no good method of
288288- // distinguishing alias vs non-alias.
289289- // Using `config.allowAliases = false` at least currently doesn't work
290290- // because there's nothing preventing people from defining aliases that
291291- // are present even with that disabled.
292292- // In the future we could kind of abuse this behavior to have better
293293- // enforcement of conditional aliases, but for now we just need to not
294294- // give an error.
295295- Success(NonApplicable)
296296- }
297297- // Only derivations can be in pkgs/by-name,
298298- // so this attribute doesn't qualify
299299- CallPackage(CallPackageInfo {
300300- is_derivation: false,
301301- ..
302302- }) => Success(NonApplicable),
303303- // A location of None indicates something weird, we can't really know where
304304- // this attribute is defined, probably an alias
305305- CallPackage(CallPackageInfo { location: None, .. }) => Success(Tight),
306306- // The case of an attribute that qualifies:
307307- // - Uses callPackage
308308- // - Is a derivation
309309- CallPackage(CallPackageInfo {
310310- is_derivation: true,
311311- call_package_variant: Manual { .. },
312312- location: Some(location),
313313- }) =>
314314- // We'll use the attribute's location to parse the file that defines it
315315- {
316316- match nix_file_store
317317- .get(&location.file)?
318318- .call_package_argument_info_at(
319319- location.line,
320320- location.column,
321321- nixpkgs_path,
322322- )? {
323323- // If the definition is not of the form `<attr> = callPackage <arg1> <arg2>;`,
324324- // it's generally not possible to migrate to `pkgs/by-name`
325325- None => Success(NonApplicable),
326326- Some(call_package_argument_info) => {
327327- if let Some(ref rel_path) = call_package_argument_info.relative_path {
328328- if rel_path.starts_with(utils::BASE_SUBPATH) {
329329- // Package variants of by-name packages are explicitly allowed according to RFC 140
330330- // https://github.com/NixOS/rfcs/blob/master/rfcs/0140-simple-package-paths.md#package-variants:
331331- //
332332- // foo-variant = callPackage ../by-name/fo/foo/package.nix {
333333- // someFlag = true;
334334- // }
335335- //
336336- // While such definitions could be moved to `pkgs/by-name` by using
337337- // `.override { someFlag = true; }` instead, this changes the semantics in
338338- // relation with overlays.
339339- Success(NonApplicable)
340340- } else {
341341- Success(Loose(call_package_argument_info))
342342- }
343343- } else {
344344- Success(Loose(call_package_argument_info))
345345- }
346346- }
363363+ // The ratchet state whether this attribute uses `pkgs/by-name`.
364364+ // This is never `Tight`, because we only either:
365365+ // - Know that the attribute _could_ be migrated to `pkgs/by-name`, which is `Loose`
366366+ // - Or we're unsure, in which case we use NonApplicable
367367+ let uses_by_name =
368368+ // This is a big ol' match on various properties of the attribute
369369+370370+ // First, it needs to succeed evaluation. We can't know whether an attribute could be
371371+ // migrated to `pkgs/by-name` if it doesn't evaluate, since we need to check that it's a
372372+ // derivation.
373373+ //
374374+ // This only has the minor negative effect that if a PR that breaks evaluation
375375+ // gets merged, fixing those failures won't force anything into `pkgs/by-name`.
376376+ //
377377+ // For now this isn't our problem, but in the future we
378378+ // might have another check to enforce that evaluation must not be broken.
379379+ //
380380+ // The alternative of assuming that failing attributes would have been fit for `pkgs/by-name`
381381+ // has the problem that if a package evaluation gets broken temporarily,
382382+ // fixing it requires a move to pkgs/by-name, which could happen more
383383+ // often and isn't really justified.
384384+ if let EvalSuccess(AttributeInfo {
385385+ // We're only interested in attributes that are attribute sets (which includes
386386+ // derivations). Anything else can't be in `pkgs/by-name`.
387387+ attribute_variant: AttributeVariant::AttributeSet {
388388+ // Indeed, we only care about derivations, non-derivation attribute sets can't be
389389+ // in `pkgs/by-name`
390390+ is_derivation: true,
391391+ // Of the two definition variants, really only the manual one makes sense here.
392392+ // Special cases are:
393393+ // - Manual aliases to auto-called packages are not treated as manual definitions,
394394+ // due to limitations in the semantic callPackage detection. So those should be
395395+ // ignored.
396396+ // - Manual definitions using the internal _internalCallByNamePackageFile are
397397+ // not treated as manual definitions, since _internalCallByNamePackageFile is
398398+ // used to detect automatic ones. We can't distinguish from the above case, so we
399399+ // just need to ignore this one too, even if that internal attribute should never
400400+ // be called manually.
401401+ definition_variant: DefinitionVariant::ManualDefinition { is_semantic_call_package }
402402+ },
403403+ // We need the location of the manual definition, because otherwise
404404+ // we can't figure out whether it's a syntactic callPackage
405405+ location: Some(location),
406406+ }) = non_by_name_attribute {
407407+408408+ // Parse the Nix file in the location and figure out whether it's an
409409+ // attribute definition of the form `= callPackage <arg1> <arg2>`,
410410+ // returning the arguments if so.
411411+ let optional_syntactic_call_package = nix_file_store
412412+ .get(&location.file)?
413413+ .call_package_argument_info_at(
414414+ location.line,
415415+ location.column,
416416+ // Passing the Nixpkgs path here both checks that the <arg1> is within Nixpkgs, and
417417+ // strips the absolute Nixpkgs path from it, such that
418418+ // syntactic_call_package.relative_path is relative to Nixpkgs
419419+ nixpkgs_path
420420+ )?;
421421+422422+ // At this point, we completed two different checks for whether it's a
423423+ // `callPackage`
424424+ match (is_semantic_call_package, optional_syntactic_call_package) {
425425+ // Something like `<attr> = { }`
426426+ (false, None)
427427+ // Something like `<attr> = pythonPackages.callPackage ...`
428428+ | (false, Some(_))
429429+ // Something like `<attr> = bar` where `bar = pkgs.callPackage ...`
430430+ | (true, None) => {
431431+ // In all of these cases, it's not possible to migrate the package to `pkgs/by-name`
432432+ NonApplicable
433433+ }
434434+ // Something like `<attr> = pkgs.callPackage ...`
435435+ (true, Some(syntactic_call_package)) => {
436436+ // It's only possible to migrate such a definitions if..
437437+ match syntactic_call_package.relative_path {
438438+ Some(ref rel_path) if rel_path.starts_with(utils::BASE_SUBPATH) => {
439439+ // ..the path is not already within `pkgs/by-name` like
440440+ //
441441+ // foo-variant = callPackage ../by-name/fo/foo/package.nix {
442442+ // someFlag = true;
443443+ // }
444444+ //
445445+ // While such definitions could be moved to `pkgs/by-name` by using
446446+ // `.override { someFlag = true; }` instead, this changes the semantics in
447447+ // relation with overlays, so migration is generally not possible.
448448+ //
449449+ // See also "package variants" in RFC 140:
450450+ // https://github.com/NixOS/rfcs/blob/master/rfcs/0140-simple-package-paths.md#package-variants
451451+ NonApplicable
452452+ }
453453+ _ => {
454454+ // Otherwise, the path is outside `pkgs/by-name`, which means it can be
455455+ // migrated
456456+ Loose(syntactic_call_package)
347457 }
348458 }
349349- };
350350- uses_by_name.map(|x| ratchet::Package {
351351- manual_definition: Tight,
352352- uses_by_name: x,
353353- })
459459+ }
354460 }
355355- EvalFailure => {
356356- // We don't know anything about this attribute really
357357- Success(ratchet::Package {
358358- // We'll assume that we can't remove any manual definitions, which has the
359359- // minimal drawback that if there was a manual definition that could've
360360- // been removed, fixing the package requires removing the definition, no
361361- // big deal, that's a minor edit.
362362- manual_definition: Tight,
363363-364364- // Regarding whether this attribute could `pkgs/by-name`, we don't really
365365- // know, so return NonApplicable, which has the effect that if a
366366- // package evaluation gets broken temporarily, the fix can remove it from
367367- // pkgs/by-name again. For now this isn't our problem, but in the future we
368368- // might have another check to enforce that evaluation must not be broken.
369369- // The alternative of assuming that it's using `pkgs/by-name` already
370370- // has the problem that if a package evaluation gets broken temporarily,
371371- // fixing it requires a move to pkgs/by-name, which could happen more
372372- // often and isn't really justified.
373373- uses_by_name: NonApplicable,
374374- })
375375- }
376376- })
461461+ } else {
462462+ // This catches all the cases not matched by the above `if let`, falling back to not being
463463+ // able to migrate such attributes
464464+ NonApplicable
465465+ };
466466+ Ok(Success(ratchet::Package {
467467+ // Packages being checked in this function _always_ need a manual definition, because
468468+ // they're not using `pkgs/by-name` which would allow avoiding it.
469469+ // so instead of repeating ourselves all the time to define `manual_definition`,
470470+ // just set it once at the end here
471471+ manual_definition: Tight,
472472+ uses_by_name,
473473+ }))
377474}
···11+pkgs.foo: This attribute is manually defined (most likely in pkgs/top-level/all-packages.nix), which is only allowed if the definition is of the form `pkgs.callPackage pkgs/by-name/fo/foo/package.nix { ... }` with a non-empty second argument.