1<chapter xmlns="http://docbook.org/ns/docbook"
2 xmlns:xlink="http://www.w3.org/1999/xlink"
3 xmlns:xi="http://www.w3.org/2001/XInclude"
4 xml:id="chap-functions">
5 <title>Functions reference</title>
6 <para>
7 The nixpkgs repository has several utility functions to manipulate Nix
8 expressions.
9 </para>
10 <section xml:id="sec-overrides">
11 <title>Overriding</title>
12
13 <para>
14 Sometimes one wants to override parts of <literal>nixpkgs</literal>, e.g.
15 derivation attributes, the results of derivations or even the whole package
16 set.
17 </para>
18
19 <section xml:id="sec-pkg-override">
20 <title><pkg>.override</title>
21
22 <para>
23 The function <varname>override</varname> is usually available for all the
24 derivations in the nixpkgs expression (<varname>pkgs</varname>).
25 </para>
26
27 <para>
28 It is used to override the arguments passed to a function.
29 </para>
30
31 <para>
32 Example usages:
33<programlisting>pkgs.foo.override { arg1 = val1; arg2 = val2; ... }</programlisting>
34<programlisting>import pkgs.path { overlays = [ (self: super: {
35 foo = super.foo.override { barSupport = true ; };
36 })]};</programlisting>
37<programlisting>mypkg = pkgs.callPackage ./mypkg.nix {
38 mydep = pkgs.mydep.override { ... };
39 }</programlisting>
40 </para>
41
42 <para>
43 In the first example, <varname>pkgs.foo</varname> is the result of a
44 function call with some default arguments, usually a derivation. Using
45 <varname>pkgs.foo.override</varname> will call the same function with the
46 given new arguments.
47 </para>
48 </section>
49
50 <section xml:id="sec-pkg-overrideAttrs">
51 <title><pkg>.overrideAttrs</title>
52
53 <para>
54 The function <varname>overrideAttrs</varname> allows overriding the
55 attribute set passed to a <varname>stdenv.mkDerivation</varname> call,
56 producing a new derivation based on the original one. This function is
57 available on all derivations produced by the
58 <varname>stdenv.mkDerivation</varname> function, which is most packages in
59 the nixpkgs expression <varname>pkgs</varname>.
60 </para>
61
62 <para>
63 Example usage:
64<programlisting>helloWithDebug = pkgs.hello.overrideAttrs (oldAttrs: rec {
65 separateDebugInfo = true;
66 });</programlisting>
67 </para>
68
69 <para>
70 In the above example, the <varname>separateDebugInfo</varname> attribute is
71 overridden to be true, thus building debug info for
72 <varname>helloWithDebug</varname>, while all other attributes will be
73 retained from the original <varname>hello</varname> package.
74 </para>
75
76 <para>
77 The argument <varname>oldAttrs</varname> is conventionally used to refer to
78 the attr set originally passed to <varname>stdenv.mkDerivation</varname>.
79 </para>
80
81 <note>
82 <para>
83 Note that <varname>separateDebugInfo</varname> is processed only by the
84 <varname>stdenv.mkDerivation</varname> function, not the generated, raw
85 Nix derivation. Thus, using <varname>overrideDerivation</varname> will not
86 work in this case, as it overrides only the attributes of the final
87 derivation. It is for this reason that <varname>overrideAttrs</varname>
88 should be preferred in (almost) all cases to
89 <varname>overrideDerivation</varname>, i.e. to allow using
90 <varname>sdenv.mkDerivation</varname> to process input arguments, as well
91 as the fact that it is easier to use (you can use the same attribute names
92 you see in your Nix code, instead of the ones generated (e.g.
93 <varname>buildInputs</varname> vs <varname>nativeBuildInputs</varname>,
94 and involves less typing.
95 </para>
96 </note>
97 </section>
98
99 <section xml:id="sec-pkg-overrideDerivation">
100 <title><pkg>.overrideDerivation</title>
101
102 <warning>
103 <para>
104 You should prefer <varname>overrideAttrs</varname> in almost all cases,
105 see its documentation for the reasons why.
106 <varname>overrideDerivation</varname> is not deprecated and will continue
107 to work, but is less nice to use and does not have as many abilities as
108 <varname>overrideAttrs</varname>.
109 </para>
110 </warning>
111
112 <warning>
113 <para>
114 Do not use this function in Nixpkgs as it evaluates a Derivation before
115 modifying it, which breaks package abstraction and removes error-checking
116 of function arguments. In addition, this evaluation-per-function
117 application incurs a performance penalty, which can become a problem if
118 many overrides are used. It is only intended for ad-hoc customisation,
119 such as in <filename>~/.config/nixpkgs/config.nix</filename>.
120 </para>
121 </warning>
122
123 <para>
124 The function <varname>overrideDerivation</varname> creates a new derivation
125 based on an existing one by overriding the original's attributes with the
126 attribute set produced by the specified function. This function is
127 available on all derivations defined using the
128 <varname>makeOverridable</varname> function. Most standard
129 derivation-producing functions, such as
130 <varname>stdenv.mkDerivation</varname>, are defined using this function,
131 which means most packages in the nixpkgs expression,
132 <varname>pkgs</varname>, have this function.
133 </para>
134
135 <para>
136 Example usage:
137<programlisting>mySed = pkgs.gnused.overrideDerivation (oldAttrs: {
138 name = "sed-4.2.2-pre";
139 src = fetchurl {
140 url = ftp://alpha.gnu.org/gnu/sed/sed-4.2.2-pre.tar.bz2;
141 sha256 = "11nq06d131y4wmf3drm0yk502d2xc6n5qy82cg88rb9nqd2lj41k";
142 };
143 patches = [];
144 });</programlisting>
145 </para>
146
147 <para>
148 In the above example, the <varname>name</varname>, <varname>src</varname>,
149 and <varname>patches</varname> of the derivation will be overridden, while
150 all other attributes will be retained from the original derivation.
151 </para>
152
153 <para>
154 The argument <varname>oldAttrs</varname> is used to refer to the attribute
155 set of the original derivation.
156 </para>
157
158 <note>
159 <para>
160 A package's attributes are evaluated *before* being modified by the
161 <varname>overrideDerivation</varname> function. For example, the
162 <varname>name</varname> attribute reference in <varname>url =
163 "mirror://gnu/hello/${name}.tar.gz";</varname> is filled-in *before* the
164 <varname>overrideDerivation</varname> function modifies the attribute set.
165 This means that overriding the <varname>name</varname> attribute, in this
166 example, *will not* change the value of the <varname>url</varname>
167 attribute. Instead, we need to override both the <varname>name</varname>
168 *and* <varname>url</varname> attributes.
169 </para>
170 </note>
171 </section>
172
173 <section xml:id="sec-lib-makeOverridable">
174 <title>lib.makeOverridable</title>
175
176 <para>
177 The function <varname>lib.makeOverridable</varname> is used to make the
178 result of a function easily customizable. This utility only makes sense for
179 functions that accept an argument set and return an attribute set.
180 </para>
181
182 <para>
183 Example usage:
184<programlisting>f = { a, b }: { result = a+b; }
185 c = lib.makeOverridable f { a = 1; b = 2; }</programlisting>
186 </para>
187
188 <para>
189 The variable <varname>c</varname> is the value of the <varname>f</varname>
190 function applied with some default arguments. Hence the value of
191 <varname>c.result</varname> is <literal>3</literal>, in this example.
192 </para>
193
194 <para>
195 The variable <varname>c</varname> however also has some additional
196 functions, like <link linkend="sec-pkg-override">c.override</link> which
197 can be used to override the default arguments. In this example the value of
198 <varname>(c.override { a = 4; }).result</varname> is 6.
199 </para>
200 </section>
201 </section>
202 <section xml:id="sec-generators">
203 <title>Generators</title>
204
205 <para>
206 Generators are functions that create file formats from nix data structures,
207 e. g. for configuration files. There are generators available for:
208 <literal>INI</literal>, <literal>JSON</literal> and <literal>YAML</literal>
209 </para>
210
211 <para>
212 All generators follow a similar call interface: <code>generatorName
213 configFunctions data</code>, where <literal>configFunctions</literal> is a
214 set of user-defined functions that format variable parts of the content.
215 They each have common defaults, so often they do not need to be set
216 manually. An example is <code>mkSectionName ? (name: libStr.escape [ "[" "]"
217 ] name)</code> from the <literal>INI</literal> generator. It gets the name
218 of a section and returns a sanitized name. The default
219 <literal>mkSectionName</literal> escapes <literal>[</literal> and
220 <literal>]</literal> with a backslash.
221 </para>
222
223 <note>
224 <para>
225 Nix store paths can be converted to strings by enclosing a derivation
226 attribute like so: <code>"${drv}"</code>.
227 </para>
228 </note>
229
230 <para>
231 Detailed documentation for each generator can be found in
232 <literal>lib/generators.nix</literal>.
233 </para>
234 </section>
235 <section xml:id="sec-fhs-environments">
236 <title>buildFHSUserEnv</title>
237
238 <para>
239 <function>buildFHSUserEnv</function> provides a way to build and run
240 FHS-compatible lightweight sandboxes. It creates an isolated root with bound
241 <filename>/nix/store</filename>, so its footprint in terms of disk space
242 needed is quite small. This allows one to run software which is hard or
243 unfeasible to patch for NixOS -- 3rd-party source trees with FHS
244 assumptions, games distributed as tarballs, software with integrity checking
245 and/or external self-updated binaries. It uses Linux namespaces feature to
246 create temporary lightweight environments which are destroyed after all
247 child processes exit, without root user rights requirement. Accepted
248 arguments are:
249 </para>
250
251 <variablelist>
252 <varlistentry>
253 <term>
254 <literal>name</literal>
255 </term>
256 <listitem>
257 <para>
258 Environment name.
259 </para>
260 </listitem>
261 </varlistentry>
262 <varlistentry>
263 <term>
264 <literal>targetPkgs</literal>
265 </term>
266 <listitem>
267 <para>
268 Packages to be installed for the main host's architecture (i.e. x86_64 on
269 x86_64 installations). Along with libraries binaries are also installed.
270 </para>
271 </listitem>
272 </varlistentry>
273 <varlistentry>
274 <term>
275 <literal>multiPkgs</literal>
276 </term>
277 <listitem>
278 <para>
279 Packages to be installed for all architectures supported by a host (i.e.
280 i686 and x86_64 on x86_64 installations). Only libraries are installed by
281 default.
282 </para>
283 </listitem>
284 </varlistentry>
285 <varlistentry>
286 <term>
287 <literal>extraBuildCommands</literal>
288 </term>
289 <listitem>
290 <para>
291 Additional commands to be executed for finalizing the directory
292 structure.
293 </para>
294 </listitem>
295 </varlistentry>
296 <varlistentry>
297 <term>
298 <literal>extraBuildCommandsMulti</literal>
299 </term>
300 <listitem>
301 <para>
302 Like <literal>extraBuildCommands</literal>, but executed only on multilib
303 architectures.
304 </para>
305 </listitem>
306 </varlistentry>
307 <varlistentry>
308 <term>
309 <literal>extraOutputsToInstall</literal>
310 </term>
311 <listitem>
312 <para>
313 Additional derivation outputs to be linked for both target and
314 multi-architecture packages.
315 </para>
316 </listitem>
317 </varlistentry>
318 <varlistentry>
319 <term>
320 <literal>extraInstallCommands</literal>
321 </term>
322 <listitem>
323 <para>
324 Additional commands to be executed for finalizing the derivation with
325 runner script.
326 </para>
327 </listitem>
328 </varlistentry>
329 <varlistentry>
330 <term>
331 <literal>runScript</literal>
332 </term>
333 <listitem>
334 <para>
335 A command that would be executed inside the sandbox and passed all the
336 command line arguments. It defaults to <literal>bash</literal>.
337 </para>
338 </listitem>
339 </varlistentry>
340 </variablelist>
341
342 <para>
343 One can create a simple environment using a <literal>shell.nix</literal>
344 like that:
345 </para>
346
347<programlisting><![CDATA[
348{ pkgs ? import <nixpkgs> {} }:
349
350(pkgs.buildFHSUserEnv {
351 name = "simple-x11-env";
352 targetPkgs = pkgs: (with pkgs;
353 [ udev
354 alsaLib
355 ]) ++ (with pkgs.xorg;
356 [ libX11
357 libXcursor
358 libXrandr
359 ]);
360 multiPkgs = pkgs: (with pkgs;
361 [ udev
362 alsaLib
363 ]);
364 runScript = "bash";
365}).env
366]]></programlisting>
367
368 <para>
369 Running <literal>nix-shell</literal> would then drop you into a shell with
370 these libraries and binaries available. You can use this to run
371 closed-source applications which expect FHS structure without hassles:
372 simply change <literal>runScript</literal> to the application path, e.g.
373 <filename>./bin/start.sh</filename> -- relative paths are supported.
374 </para>
375 </section>
376 <xi:include href="shell.section.xml" />
377 <section xml:id="sec-pkgs-dockerTools">
378 <title>pkgs.dockerTools</title>
379
380 <para>
381 <varname>pkgs.dockerTools</varname> is a set of functions for creating and
382 manipulating Docker images according to the
383 <link xlink:href="https://github.com/moby/moby/blob/master/image/spec/v1.2.md#docker-image-specification-v120">
384 Docker Image Specification v1.2.0 </link>. Docker itself is not used to
385 perform any of the operations done by these functions.
386 </para>
387
388 <warning>
389 <para>
390 The <varname>dockerTools</varname> API is unstable and may be subject to
391 backwards-incompatible changes in the future.
392 </para>
393 </warning>
394
395 <section xml:id="ssec-pkgs-dockerTools-buildImage">
396 <title>buildImage</title>
397
398 <para>
399 This function is analogous to the <command>docker build</command> command,
400 in that can used to build a Docker-compatible repository tarball containing
401 a single image with one or multiple layers. As such, the result is suitable
402 for being loaded in Docker with <command>docker load</command>.
403 </para>
404
405 <para>
406 The parameters of <varname>buildImage</varname> with relative example
407 values are described below:
408 </para>
409
410 <example xml:id='ex-dockerTools-buildImage'>
411 <title>Docker build</title>
412<programlisting>
413 buildImage {
414 name = "redis"; <co xml:id='ex-dockerTools-buildImage-1' />
415 tag = "latest"; <co xml:id='ex-dockerTools-buildImage-2' />
416
417 fromImage = someBaseImage; <co xml:id='ex-dockerTools-buildImage-3' />
418 fromImageName = null; <co xml:id='ex-dockerTools-buildImage-4' />
419 fromImageTag = "latest"; <co xml:id='ex-dockerTools-buildImage-5' />
420
421 contents = pkgs.redis; <co xml:id='ex-dockerTools-buildImage-6' />
422 runAsRoot = '' <co xml:id='ex-dockerTools-buildImage-runAsRoot' />
423 #!${stdenv.shell}
424 mkdir -p /data
425 '';
426
427 config = { <co xml:id='ex-dockerTools-buildImage-8' />
428 Cmd = [ "/bin/redis-server" ];
429 WorkingDir = "/data";
430 Volumes = {
431 "/data" = {};
432 };
433 };
434 }
435 </programlisting>
436 </example>
437
438 <para>
439 The above example will build a Docker image <literal>redis/latest</literal>
440 from the given base image. Loading and running this image in Docker results
441 in <literal>redis-server</literal> being started automatically.
442 </para>
443
444 <calloutlist>
445 <callout arearefs='ex-dockerTools-buildImage-1'>
446 <para>
447 <varname>name</varname> specifies the name of the resulting image. This
448 is the only required argument for <varname>buildImage</varname>.
449 </para>
450 </callout>
451 <callout arearefs='ex-dockerTools-buildImage-2'>
452 <para>
453 <varname>tag</varname> specifies the tag of the resulting image. By
454 default it's <literal>latest</literal>.
455 </para>
456 </callout>
457 <callout arearefs='ex-dockerTools-buildImage-3'>
458 <para>
459 <varname>fromImage</varname> is the repository tarball containing the
460 base image. It must be a valid Docker image, such as exported by
461 <command>docker save</command>. By default it's <literal>null</literal>,
462 which can be seen as equivalent to <literal>FROM scratch</literal> of a
463 <filename>Dockerfile</filename>.
464 </para>
465 </callout>
466 <callout arearefs='ex-dockerTools-buildImage-4'>
467 <para>
468 <varname>fromImageName</varname> can be used to further specify the base
469 image within the repository, in case it contains multiple images. By
470 default it's <literal>null</literal>, in which case
471 <varname>buildImage</varname> will peek the first image available in the
472 repository.
473 </para>
474 </callout>
475 <callout arearefs='ex-dockerTools-buildImage-5'>
476 <para>
477 <varname>fromImageTag</varname> can be used to further specify the tag of
478 the base image within the repository, in case an image contains multiple
479 tags. By default it's <literal>null</literal>, in which case
480 <varname>buildImage</varname> will peek the first tag available for the
481 base image.
482 </para>
483 </callout>
484 <callout arearefs='ex-dockerTools-buildImage-6'>
485 <para>
486 <varname>contents</varname> is a derivation that will be copied in the
487 new layer of the resulting image. This can be similarly seen as
488 <command>ADD contents/ /</command> in a <filename>Dockerfile</filename>.
489 By default it's <literal>null</literal>.
490 </para>
491 </callout>
492 <callout arearefs='ex-dockerTools-buildImage-runAsRoot'>
493 <para>
494 <varname>runAsRoot</varname> is a bash script that will run as root in an
495 environment that overlays the existing layers of the base image with the
496 new resulting layer, including the previously copied
497 <varname>contents</varname> derivation. This can be similarly seen as
498 <command>RUN ...</command> in a <filename>Dockerfile</filename>.
499 <note>
500 <para>
501 Using this parameter requires the <literal>kvm</literal> device to be
502 available.
503 </para>
504 </note>
505 </para>
506 </callout>
507 <callout arearefs='ex-dockerTools-buildImage-8'>
508 <para>
509 <varname>config</varname> is used to specify the configuration of the
510 containers that will be started off the built image in Docker. The
511 available options are listed in the
512 <link xlink:href="https://github.com/moby/moby/blob/master/image/spec/v1.2.md#image-json-field-descriptions">
513 Docker Image Specification v1.2.0 </link>.
514 </para>
515 </callout>
516 </calloutlist>
517
518 <para>
519 After the new layer has been created, its closure (to which
520 <varname>contents</varname>, <varname>config</varname> and
521 <varname>runAsRoot</varname> contribute) will be copied in the layer
522 itself. Only new dependencies that are not already in the existing layers
523 will be copied.
524 </para>
525
526 <para>
527 At the end of the process, only one new single layer will be produced and
528 added to the resulting image.
529 </para>
530
531 <para>
532 The resulting repository will only list the single image
533 <varname>image/tag</varname>. In the case of
534 <xref linkend='ex-dockerTools-buildImage'/> it would be
535 <varname>redis/latest</varname>.
536 </para>
537
538 <para>
539 It is possible to inspect the arguments with which an image was built using
540 its <varname>buildArgs</varname> attribute.
541 </para>
542
543 <note>
544 <para>
545 If you see errors similar to <literal>getProtocolByName: does not exist
546 (no such protocol name: tcp)</literal> you may need to add
547 <literal>pkgs.iana-etc</literal> to <varname>contents</varname>.
548 </para>
549 </note>
550
551 <note>
552 <para>
553 If you see errors similar to <literal>Error_Protocol ("certificate has
554 unknown CA",True,UnknownCa)</literal> you may need to add
555 <literal>pkgs.cacert</literal> to <varname>contents</varname>.
556 </para>
557 </note>
558 </section>
559
560 <section xml:id="ssec-pkgs-dockerTools-fetchFromRegistry">
561 <title>pullImage</title>
562
563 <para>
564 This function is analogous to the <command>docker pull</command> command,
565 in that can be used to fetch a Docker image from a Docker registry.
566 Currently only registry <literal>v1</literal> is supported. By default
567 <link xlink:href="https://hub.docker.com/">Docker Hub</link> is used to
568 pull images.
569 </para>
570
571 <para>
572 Its parameters are described in the example below:
573 </para>
574
575 <example xml:id='ex-dockerTools-pullImage'>
576 <title>Docker pull</title>
577<programlisting>
578 pullImage {
579 imageName = "debian"; <co xml:id='ex-dockerTools-pullImage-1' />
580 imageTag = "jessie"; <co xml:id='ex-dockerTools-pullImage-2' />
581 imageId = null; <co xml:id='ex-dockerTools-pullImage-3' />
582 sha256 = "1bhw5hkz6chrnrih0ymjbmn69hyfriza2lr550xyvpdrnbzr4gk2"; <co xml:id='ex-dockerTools-pullImage-4' />
583
584 indexUrl = "https://index.docker.io"; <co xml:id='ex-dockerTools-pullImage-5' />
585 registryVersion = "v1";
586 }
587 </programlisting>
588 </example>
589
590 <calloutlist>
591 <callout arearefs='ex-dockerTools-pullImage-1'>
592 <para>
593 <varname>imageName</varname> specifies the name of the image to be
594 downloaded, which can also include the registry namespace (e.g.
595 <literal>library/debian</literal>). This argument is required.
596 </para>
597 </callout>
598 <callout arearefs='ex-dockerTools-pullImage-2'>
599 <para>
600 <varname>imageTag</varname> specifies the tag of the image to be
601 downloaded. By default it's <literal>latest</literal>.
602 </para>
603 </callout>
604 <callout arearefs='ex-dockerTools-pullImage-3'>
605 <para>
606 <varname>imageId</varname>, if specified this exact image will be
607 fetched, instead of <varname>imageName/imageTag</varname>. However, the
608 resulting repository will still be named
609 <varname>imageName/imageTag</varname>. By default it's
610 <literal>null</literal>.
611 </para>
612 </callout>
613 <callout arearefs='ex-dockerTools-pullImage-4'>
614 <para>
615 <varname>sha256</varname> is the checksum of the whole fetched image.
616 This argument is required.
617 </para>
618 <note>
619 <para>
620 The checksum is computed on the unpacked directory, not on the final
621 tarball.
622 </para>
623 </note>
624 </callout>
625 <callout arearefs='ex-dockerTools-pullImage-5'>
626 <para>
627 In the above example the default values are shown for the variables
628 <varname>indexUrl</varname> and <varname>registryVersion</varname>. Hence
629 by default the Docker.io registry is used to pull the images.
630 </para>
631 </callout>
632 </calloutlist>
633 </section>
634
635 <section xml:id="ssec-pkgs-dockerTools-exportImage">
636 <title>exportImage</title>
637
638 <para>
639 This function is analogous to the <command>docker export</command> command,
640 in that can used to flatten a Docker image that contains multiple layers.
641 It is in fact the result of the merge of all the layers of the image. As
642 such, the result is suitable for being imported in Docker with
643 <command>docker import</command>.
644 </para>
645
646 <note>
647 <para>
648 Using this function requires the <literal>kvm</literal> device to be
649 available.
650 </para>
651 </note>
652
653 <para>
654 The parameters of <varname>exportImage</varname> are the following:
655 </para>
656
657 <example xml:id='ex-dockerTools-exportImage'>
658 <title>Docker export</title>
659<programlisting>
660 exportImage {
661 fromImage = someLayeredImage;
662 fromImageName = null;
663 fromImageTag = null;
664
665 name = someLayeredImage.name;
666 }
667 </programlisting>
668 </example>
669
670 <para>
671 The parameters relative to the base image have the same synopsis as
672 described in <xref linkend='ssec-pkgs-dockerTools-buildImage'/>, except
673 that <varname>fromImage</varname> is the only required argument in this
674 case.
675 </para>
676
677 <para>
678 The <varname>name</varname> argument is the name of the derivation output,
679 which defaults to <varname>fromImage.name</varname>.
680 </para>
681 </section>
682
683 <section xml:id="ssec-pkgs-dockerTools-shadowSetup">
684 <title>shadowSetup</title>
685
686 <para>
687 This constant string is a helper for setting up the base files for managing
688 users and groups, only if such files don't exist already. It is suitable
689 for being used in a <varname>runAsRoot</varname>
690 <xref linkend='ex-dockerTools-buildImage-runAsRoot'/> script for cases like
691 in the example below:
692 </para>
693
694 <example xml:id='ex-dockerTools-shadowSetup'>
695 <title>Shadow base files</title>
696<programlisting>
697 buildImage {
698 name = "shadow-basic";
699
700 runAsRoot = ''
701 #!${stdenv.shell}
702 ${shadowSetup}
703 groupadd -r redis
704 useradd -r -g redis redis
705 mkdir /data
706 chown redis:redis /data
707 '';
708 }
709 </programlisting>
710 </example>
711
712 <para>
713 Creating base files like <literal>/etc/passwd</literal> or
714 <literal>/etc/login.defs</literal> are necessary for shadow-utils to
715 manipulate users and groups.
716 </para>
717 </section>
718 </section>
719</chapter>