nixpkgs mirror (for testing)
github.com/NixOS/nixpkgs
nix
1# Python {#python}
2
3## Reference {#reference}
4
5### Interpreters {#interpreters}
6
7@python-interpreter-table@
8
9The Nix expressions for the interpreters can be found in
10`pkgs/development/interpreters/python`.
11
12All packages depending on any Python interpreter get appended
13`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
14exists.
15
16#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
17
18To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
19
20#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
21
22Each interpreter has the following attributes:
23
24- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
25- `interpreter`. Alias for `${python}/bin/${executable}`.
26- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See [](#python.buildenv-function) for usage and documentation.
27- `withPackages`. Simpler interface to `buildEnv`. See [](#python.withpackages-function) for usage and documentation.
28- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
29- `executable`. Name of the interpreter executable, e.g. `python3.10`.
30- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
31
32### Building packages and applications {#building-packages-and-applications}
33
34Python libraries and applications that use tools to follow PEP 517 (e.g. `setuptools` or `hatchling`, etc.) or
35previous tools such as `distutils` are typically built with respectively the [`buildPythonPackage`](#buildpythonpackage-function) and
36[`buildPythonApplication`](#buildpythonapplication-function) functions. These two functions also support installing a `wheel`.
37
38All Python packages reside in `pkgs/top-level/python-packages.nix` and all
39applications elsewhere. In case a package is used as both a library and an
40application, then the package should be in `pkgs/top-level/python-packages.nix`
41since only those packages are made available for all interpreter versions. The
42preferred location for library expressions is in
43`pkgs/development/python-modules`. It is important that these packages are
44called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
45the right version of the package is built.
46
47Based on the packages defined in `pkgs/top-level/python-packages.nix` an
48attribute set is created for each available Python interpreter. The available
49sets are
50
51* `pkgs.python27Packages`
52* `pkgs.python3Packages`
53* `pkgs.python311Packages`
54* `pkgs.python312Packages`
55* `pkgs.python313Packages`
56* `pkgs.python314Packages`
57* `pkgs.python315Packages`
58* `pkgs.pypy27Packages`
59* `pkgs.pypy310Packages`
60
61and the aliases
62
63* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
64* `pkgs.python3Packages` pointing to `pkgs.python313Packages`
65* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
66* `pkgs.pypy2Packages` pointing to `pkgs.pypy27Packages`
67* `pkgs.pypy3Packages` pointing to `pkgs.pypy310Packages`
68* `pkgs.pypyPackages` pointing to `pkgs.pypy2Packages`
69
70
71#### `buildPythonPackage` function {#buildpythonpackage-function}
72
73The `buildPythonPackage` function has its name binding in
74`pkgs/development/interpreters/python/python-packages-base.nix` and is
75implemented in `pkgs/development/interpreters/python/mk-python-derivation.nix`
76using setup hooks.
77
78The following is an example:
79
80```nix
81{
82 lib,
83 buildPythonPackage,
84 fetchPypi,
85
86 # build-system
87 setuptools,
88 setuptools-scm,
89
90 # dependencies
91 attrs,
92 pluggy,
93 py,
94 setuptools,
95 six,
96
97 # tests
98 hypothesis,
99}:
100
101buildPythonPackage (finalAttrs: {
102 pname = "pytest";
103 version = "3.3.1";
104 pyproject = true;
105
106 src = fetchPypi {
107 inherit (finalAttrs) pname version;
108
109 hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M=";
110 };
111
112 postPatch = ''
113 # don't test bash builtins
114 rm testing/test_argcomplete.py
115 '';
116
117 build-system = [
118 setuptools
119 setuptools-scm
120 ];
121
122 dependencies = [
123 attrs
124 py
125 setuptools
126 six
127 pluggy
128 ];
129
130 nativeCheckInputs = [ hypothesis ];
131
132 meta = {
133 changelog = "https://github.com/pytest-dev/pytest/releases/tag/${finalAttrs.version}";
134 description = "Framework for writing tests";
135 homepage = "https://github.com/pytest-dev/pytest";
136 license = lib.licenses.mit;
137 maintainers = with lib.maintainers; [
138 lovek323
139 madjar
140 lsix
141 ];
142 };
143})
144```
145
146The `buildPythonPackage` mainly does four things:
147
148* In the [`buildPhase`](#build-phase), it calls `${python.pythonOnBuildForHost.interpreter} -m build --wheel` to
149 build a wheel binary zipfile.
150* In the [`installPhase`](#ssec-install-phase), it installs the wheel file using `${python.pythonOnBuildForHost.interpreter} -m installer *.whl`.
151* In the [`postFixup`](#var-stdenv-postFixup) phase, the `wrapPythonPrograms` bash function is called to
152 wrap all programs in the `$out/bin/*` directory to include `$PATH`
153 environment variable and add dependent libraries to script's `sys.path`.
154* In the [`installCheck`](#ssec-installCheck-phase) phase, `${python.interpreter} -m pytest` is run.
155
156By default tests are run because [`doCheck = true`](#var-stdenv-doCheck). Test dependencies, like
157e.g. the test runner, should be added to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs).
158
159By default `meta.platforms` is set to the same value
160as the interpreter unless overridden otherwise.
161
162##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
163
164All parameters from [`stdenv.mkDerivation`](#sec-using-stdenv) function are still supported. The
165following are specific to `buildPythonPackage`:
166
167* `catchConflicts ? true`: If `true`, abort package build if a package name
168 appears more than once in dependency tree. Default is `true`.
169* `disabled ? false`: If `true`, package is not built for the particular Python
170 interpreter version.
171* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
172* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
173 variable in wrapped programs.
174* `pyproject`: Whether the pyproject format should be used. As all other formats
175 are deprecated, you are recommended to set this to `true`. When you do so,
176 `pypaBuildHook` will be used, and you can add the required build dependencies
177 from `build-system.requires` to `build-system`. Note that the pyproject
178 format falls back to using `setuptools`, so you can use `pyproject = true`
179 even if the package only has a `setup.py`. When set to `false`, you can
180 use the existing [hooks](#setup-hooks) or provide your own logic to build the
181 package. This can be useful for packages that don't support the pyproject
182 format. When unset, the legacy `setuptools` hooks are used for backwards
183 compatibility.
184* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
185 [`makeWrapper`](#fun-makeWrapper), which wraps generated binaries. By default, the arguments to
186 [`makeWrapper`](#fun-makeWrapper) set `PATH` and `PYTHONPATH` environment variables before calling
187 the binary. Additional arguments here can allow a developer to set environment
188 variables which will be available when the binary is run. For example,
189 `makeWrapperArgs = ["--set" "FOO" "BAR" "--set" "BAZ" "QUX"]`.
190
191 ::: {.note}
192 When `__structuredAttrs = false`, the attribute `makeWrapperArgs` is passed as a space-separated string to the build script. Developers should use `prependToVar` or `appendToVar` to add arguments to it in build phases, or use `__structuredAttrs = true` to ensure that `makeWrapperArgs` is passed as a Bash array.
193
194 For compatibility purposes,
195 when `makeWrapperArgs` shell variable is specified as a space-separated string (instead of a Bash array) in the build script, the string content is Bash-expanded before concatenated into the `wrapProgram` command. Still, developers should not rely on such behaviours, but use `__structuredAttrs = true` to specify flags containing spaces (e.g. `makeWrapperArgs = [ "--set" "GREETING" "Hello, world!" ]`), or use -pre and -post phases to specify flags with Bash-expansions (e.g. `preFixup = ''makeWrapperArgs+=(--prefix PATH : "$SOME_PATH")''`).
196 :::
197
198* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
199 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`.
200* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
201* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
202 in `pythonPath` are not propagated (contrary to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)).
203* `preShellHook`: Hook to execute commands before `shellHook`.
204* `postShellHook`: Hook to execute commands after `shellHook`.
205* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
206 created when the filenames end with `.py`.
207* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
208* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
209
210The [`stdenv.mkDerivation`](#sec-using-stdenv) function accepts various parameters for describing
211build inputs (see "Specifying dependencies"). The following are of special
212interest for Python packages, either because these are primarily used, or
213because their behaviour is different:
214
215* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables.
216* `build-system ? []`: Build-time only Python dependencies. Items listed in `build-system.requires`/`setup_requires`.
217* `buildInputs ? []`: Build and/or run-time dependencies that need to be
218 compiled for the host machine. Typically non-Python libraries which are being
219 linked.
220* `nativeCheckInputs ? []`: Dependencies needed for running the [`checkPhase`](#ssec-check-phase). These
221 are added to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs) when [`doCheck = true`](#var-stdenv-doCheck). Items listed in
222 `tests_require` go here.
223* `dependencies ? []`: Aside from propagating dependencies,
224 `buildPythonPackage` also injects code into and wraps executables with the
225 paths included in this list. Items listed in `install_requires` go here.
226* `optional-dependencies ? { }`: Optional feature flagged dependencies. Items listed in `extras_require` go here.
227
228
229##### Overriding Python packages {#overriding-python-packages}
230
231The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
232used to override the package. In the following example we create an environment
233where we have the `blaze` package using an older version of `pandas`. We
234override first the Python interpreter and pass `packageOverrides` which contains
235the overrides for packages in the package set.
236
237```nix
238with import <nixpkgs> { };
239
240(
241 let
242 python =
243 let
244 packageOverrides = self: super: {
245 pandas = super.pandas.overridePythonAttrs (old: rec {
246 version = "0.19.1";
247 src = fetchPypi {
248 pname = "pandas";
249 inherit version;
250 hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE=";
251 };
252 });
253 };
254 in
255 pkgs.python3.override {
256 inherit packageOverrides;
257 self = python;
258 };
259
260 in
261 python.withPackages (ps: [ ps.blaze ])
262).env
263```
264
265The next example shows a non trivial overriding of the `blas` implementation to
266be used through out all of the Python package set:
267
268```nix
269{
270 python3MyBlas = pkgs.python3.override {
271 packageOverrides = self: super: {
272 # We need toPythonModule for the package set to evaluate this
273 blas = super.toPythonModule (super.pkgs.blas.override { blasProvider = super.pkgs.mkl; });
274 lapack = super.toPythonModule (super.pkgs.lapack.override { lapackProvider = super.pkgs.mkl; });
275 };
276 };
277}
278```
279
280This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations.
281Note that using `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in
282compilation issues, because scipy dependencies need to use the same blas implementation as well.
283
284#### `buildPythonApplication` function {#buildpythonapplication-function}
285
286The [`buildPythonApplication`](#buildpythonapplication-function) function is practically the same as
287[`buildPythonPackage`](#buildpythonpackage-function). The main purpose of this function is to build a Python
288package where one is interested only in the executables, and not importable
289modules. For that reason, when adding this package to a [`python.buildEnv`](#python.buildenv-function), the
290modules won't be made available.
291
292Another difference is that [`buildPythonPackage`](#buildpythonpackage-function) by default prefixes the names of
293the packages with the version of the interpreter. Because this is irrelevant for
294applications, the prefix is omitted.
295
296When packaging a Python application with [`buildPythonApplication`](#buildpythonapplication-function), it should be
297called with `callPackage` and passed `python3` or `python3Packages` (possibly
298specifying an interpreter version), like this:
299
300```nix
301{
302 lib,
303 python3Packages,
304 fetchPypi,
305}:
306
307python3Packages.buildPythonApplication (finalAttrs: {
308 pname = "luigi";
309 version = "2.7.9";
310 pyproject = true;
311
312 src = fetchPypi {
313 inherit (finalAttrs) pname version;
314 hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw=";
315 };
316
317 build-system = with python3Packages; [ setuptools ];
318
319 dependencies = with python3Packages; [
320 tornado
321 python-daemon
322 ];
323
324 meta = {
325 # ...
326 };
327})
328```
329
330This is then added to `pkgs/by-name` just as any other application would be.
331
332Since the package is an application, a consumer doesn't need to care about
333Python versions or modules, which is why they don't go in `python3Packages`.
334
335#### `toPythonApplication` function {#topythonapplication-function}
336
337A distinction is made between applications and libraries, however, sometimes a
338package is used as both. In this case the package is added as a library to
339`python-packages.nix` and as an application to `pkgs/by-name`. To reduce
340duplication the `toPythonApplication` can be used to convert a library to an
341application.
342
343The Nix expression shall use [`buildPythonPackage`](#buildpythonpackage-function) and be called from
344`python-packages.nix`. A reference shall be created from `pkgs/by-name` to
345the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
346applied to the reference:
347
348```nix
349{ python3Packages }:
350
351python3Packages.toPythonApplication python3Packages.youtube-dl
352```
353
354#### `toPythonModule` function {#topythonmodule-function}
355
356In some cases, such as bindings, a package is created using
357[`stdenv.mkDerivation`](#sec-using-stdenv) and added as attribute in `pkgs/by-name` or in `all-packages.nix`. The Python
358bindings should be made available from `python-packages.nix`. The
359`toPythonModule` function takes a derivation and makes certain Python-specific
360modifications.
361
362```nix
363{
364 opencv = toPythonModule (
365 pkgs.opencv.override {
366 enablePython = true;
367 pythonPackages = self;
368 }
369 );
370}
371```
372
373Do pay attention to passing in the right Python version!
374
375#### `mkPythonMetaPackage` function {#mkpythonmetapackage-function}
376
377This will create a meta package containing [metadata files](https://packaging.python.org/en/latest/specifications/recording-installed-packages/) to satisfy a dependency on a package, without it actually having been installed into the environment.
378In nixpkgs this is used to package Python packages with split binary/source distributions such as [psycopg2](https://pypi.org/project/psycopg2/)/[psycopg2-binary](https://pypi.org/project/psycopg2-binary/).
379
380```nix
381mkPythonMetaPackage {
382 pname = "psycopg2-binary";
383 inherit (psycopg2) optional-dependencies version;
384 dependencies = [ psycopg2 ];
385 meta = { inherit (psycopg2.meta) description homepage; };
386}
387```
388
389#### `mkPythonEditablePackage` function {#mkpythoneditablepackage-function}
390
391When developing Python packages it's common to install packages in [editable mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html).
392Like `mkPythonMetaPackage` this function exists to create an otherwise empty package, but also containing a pointer to an impure location outside the Nix store that can be changed without rebuilding.
393
394The editable root is passed as a string. Normally `.pth` files contains absolute paths to the mutable location. This isn't always ergonomic with Nix, so environment variables are expanded at runtime.
395This means that a shell hook setting up something like a `$REPO_ROOT` variable can be used as the relative package root.
396
397As an implementation detail, the [PEP-518](https://peps.python.org/pep-0518/) `build-system` specified won't be used, but instead the editable package will be built using [hatchling](https://pypi.org/project/hatchling/).
398The `build-system`'s provided will instead become runtime dependencies of the editable package.
399
400Note that overriding packages deeper in the dependency graph _can_ work, but it's not the primary use case and overriding existing packages can make others break in unexpected ways.
401
402```nix
403{
404 pkgs ? import <nixpkgs> { },
405}:
406
407let
408 pyproject = pkgs.lib.importTOML ./pyproject.toml;
409
410 myPython = pkgs.python.override {
411 self = myPython;
412 packageOverrides = pyfinal: pyprev: {
413 # An editable package with a script that loads our mutable location
414 my-editable = pyfinal.mkPythonEditablePackage {
415 # Inherit project metadata from pyproject.toml
416 pname = pyproject.project.name;
417 inherit (pyproject.project) version;
418
419 # The editable root passed as a string
420 root = "$REPO_ROOT/src"; # Use environment variable expansion at runtime
421
422 # Inject a script (other PEP-621 entrypoints are also accepted)
423 inherit (pyproject.project) scripts;
424 };
425 };
426 };
427
428 pythonEnv = myPython.withPackages (ps: [ ps.my-editable ]);
429
430in
431pkgs.mkShell { packages = [ pythonEnv ]; }
432```
433
434#### `python.buildEnv` function {#python.buildenv-function}
435
436Python environments can be created using the low-level `pkgs.buildEnv` function.
437This example shows how to create an environment that has the Pyramid Web Framework.
438Saving the following as `default.nix`
439
440```nix
441with import <nixpkgs> { };
442
443python3.buildEnv.override {
444 extraLibs = [ python3Packages.pyramid ];
445 ignoreCollisions = true;
446}
447```
448
449and running `nix-build` will create
450
451```
452/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
453```
454
455with wrapped binaries in `bin/`.
456
457You can also use the `env` attribute to create local environments with needed
458packages installed. This is somewhat comparable to `virtualenv`. For example,
459running `nix-shell` with the following `shell.nix`
460
461```nix
462with import <nixpkgs> { };
463
464(python3.buildEnv.override {
465 extraLibs = with python3Packages; [
466 numpy
467 requests
468 ];
469}).env
470```
471
472will drop you into a shell where Python will have the
473specified packages in its path.
474
475##### `python.buildEnv` arguments {#python.buildenv-arguments}
476
477
478* `extraLibs`: List of packages installed inside the environment.
479* `postBuild`: Shell command executed after the build of environment.
480* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
481* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
482 wrapped binaries in the environment.
483
484#### `python.withPackages` function {#python.withpackages-function}
485
486The [`python.withPackages`](#python.withpackages-function) function provides a simpler interface to the [`python.buildEnv`](#python.buildenv-function) functionality.
487It takes a function as an argument that is passed the set of python packages and returns the list
488of the packages to be included in the environment. Using the [`withPackages`](#python.withpackages-function) function, the previous
489example for the Pyramid Web Framework environment can be written like this:
490
491```nix
492with import <nixpkgs> { };
493
494python.withPackages (ps: [ ps.pyramid ])
495```
496
497[`withPackages`](#python.withpackages-function) passes the correct package set for the specific interpreter
498version as an argument to the function. In the above example, `ps` equals
499`pythonPackages`. But you can also easily switch to using python3:
500
501```nix
502with import <nixpkgs> { };
503
504python3.withPackages (ps: [ ps.pyramid ])
505```
506
507Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
508
509As [`python.withPackages`](#python.withpackages-function) uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also
510supports the `env` attribute. The `shell.nix` file from the previous section can
511thus be also written like this:
512
513```nix
514with import <nixpkgs> { };
515
516(python3.withPackages (
517 ps: with ps; [
518 numpy
519 requests
520 ]
521)).env
522```
523
524In contrast to [`python.buildEnv`](#python.buildenv-function), [`python.withPackages`](#python.withpackages-function) does not support the
525more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
526need them, you have to use [`python.buildEnv`](#python.buildenv-function).
527
528Python 2 namespace packages may provide `__init__.py` that collide. In that case
529[`python.buildEnv`](#python.buildenv-function) should be used with `ignoreCollisions = true`.
530
531#### Setup hooks {#setup-hooks}
532
533The following are setup hooks specifically for Python packages. Most of these
534are used in [`buildPythonPackage`](#buildpythonpackage-function).
535
536- `eggUnpackhook` to move an egg to the correct folder so it can be installed
537 with the `eggInstallHook`
538- `eggBuildHook` to skip building for eggs.
539- `eggInstallHook` to install eggs.
540- `pypaBuildHook` to build a wheel using
541 [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and
542 PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still
543 be added as `build-system`.
544- `pypaInstallHook` to install wheels.
545- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
546- `pythonCatchConflictsHook` to fail if the package depends on two different versions of the same dependency.
547- `pythonImportsCheckHook` to check whether importing the listed modules works.
548- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package.
549 See [example usage](#using-pythonrelaxdepshook).
550- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
551- `setuptoolsBuildHook` to build a wheel using `setuptools`.
552- `sphinxHook` to build documentation and manpages using Sphinx.
553- `stestrCheckHook` to run tests with `stestr`.
554- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
555 `venv` is created if it does not yet exist. `postVenvCreation` can be used to
556 to run commands only after venv is first created.
557- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
558 with the `pipInstallHook`.
559- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook).
560
561#### Overriding build helpers {#overriding-python-build-helpers}
562
563Like many of the build helpers provided by Nixpkgs, Python build helpers typically provide a `<function>.override` attribute.
564It works like [`<pkg>.override`](#sec-pkg-override), and can be used to override the dependencies of each build helper.
565
566This allows specifying the stdenv to be used by `buildPythonPackage` or `buildPythonApplication`. The default (`python.stdenv`) can be overridden as follows:
567
568```nix
569buildPythonPackage.override { stdenv = customStdenv; } {
570 # package attrs...
571}
572```
573
574## User Guide {#user-guide}
575
576### Using Python {#using-python}
577
578#### Overview {#overview}
579
580Several versions of the Python interpreter are available on Nix, as well as a
581high amount of packages. The attribute `python3` refers to the default
582interpreter, which is currently CPython 3.13. The attribute `python` refers to
583CPython 2.7 for backwards compatibility. It is also possible to refer to
584specific versions, e.g., `python313` refers to CPython 3.13, and `pypy` refers to
585the default PyPy interpreter.
586
587Python is used a lot, and in different ways. This affects also how it is
588packaged. In the case of Python on Nix, an important distinction is made between
589whether the package is considered primarily an application, or whether it should
590be used as a library, i.e., of primary interest are the modules in
591`site-packages` that should be importable.
592
593In the Nixpkgs tree Python applications can be found throughout, depending on
594what they do, and are called from the main package set. Python libraries,
595however, are in separate sets, with one set per interpreter version.
596
597The interpreters have several common attributes. One of these attributes is
598`pkgs`, which is a package set of Python libraries for this specific
599interpreter. E.g., the `toolz` package corresponding to the default interpreter
600is `python3.pkgs.toolz`, and the CPython 3.13 version is `python313.pkgs.toolz`.
601The main package set contains aliases to these package sets, e.g.
602`pythonPackages` refers to `python.pkgs` and `python313Packages` to
603`python313.pkgs`.
604
605#### Installing Python and packages {#installing-python-and-packages}
606
607The Nix and NixOS manuals explain how packages are generally installed. In the
608case of Python and Nix, it is important to make a distinction between whether the
609package is considered an application or a library.
610
611Applications on Nix are typically installed into your user profile imperatively
612using `nix-env -i`, and on NixOS declaratively by adding the package name to
613`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies
614such as libraries are automatically installed and should not be installed
615explicitly.
616
617The same goes for Python applications. Python applications can be installed in
618your profile, and will be wrapped to find their exact library dependencies,
619without impacting other applications or polluting your user environment.
620
621But Python libraries you would like to use for development cannot be installed,
622at least not individually, because they won't be able to find each other
623resulting in import errors. Instead, it is possible to create an environment
624with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#python.withpackages-function) where the interpreter and other
625executables are wrapped to be able to find each other and all of the modules.
626
627In the following examples we will start by creating a simple, ad-hoc environment
628with a nix-shell that has `numpy` and `toolz` in Python 3.13; then we will create
629a re-usable environment in a single-file Python script; then we will create a
630full Python environment for development with this same environment.
631
632Philosophically, this should be familiar to users who are used to a `venv` style
633of development: individual projects create their own Python environments without
634impacting the global environment or each other.
635
636#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell}
637
638The simplest way to start playing with the way nix wraps and sets up Python
639environments is with `nix-shell` at the cmdline. These environments create a
640temporary shell session with a Python and a *precise* list of packages (plus
641their runtime dependencies), with no other Python packages in the Python
642interpreter's scope.
643
644To create a Python 3.13 session with `numpy` and `toolz` available, run:
645
646```sh
647$ nix-shell -p 'python313.withPackages(ps: with ps; [ numpy toolz ])'
648```
649
650By default `nix-shell` will start a `bash` session with this interpreter in our
651`PATH`, so if we then run:
652
653```Python console
654[nix-shell:~/src/nixpkgs]$ python3
655Python 3.13.3 (main, Apr 8 2025, 13:54:08) [GCC 14.2.1 20250322] on linux
656Type "help", "copyright", "credits" or "license" for more information.
657>>> import numpy; import toolz
658```
659
660Note that no other modules are in scope, even if they were imperatively
661installed into our user environment as a dependency of a Python application:
662
663```Python console
664>>> import requests
665Traceback (most recent call last):
666 File "<stdin>", line 1, in <module>
667ModuleNotFoundError: No module named 'requests'
668```
669
670We can add as many additional modules onto the `nix-shell` as we need, and we
671will still get 1 wrapped Python interpreter. We can start the interpreter
672directly like so:
673
674```sh
675$ nix-shell -p "python313.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
676Python 3.13.3 (main, Apr 8 2025, 13:54:08) [GCC 14.2.1 20250322] on linux
677Type "help", "copyright", "credits" or "license" for more information.
678>>> import requests
679>>>
680```
681
682Notice that this time it built a new Python environment, which now includes
683`requests`. Building an environment just creates wrapper scripts that expose the
684selected dependencies to the interpreter while re-using the actual modules. This
685means if any other env has installed `requests` or `numpy` in a different
686context, we don't need to recompile them -- we just recompile the wrapper script
687that sets up an interpreter pointing to them. This matters much more for "big"
688modules like `pytorch` or `tensorflow`.
689
690Module names usually match their names on [pypi.org](https://pypi.org/), but
691normalized according to PEP 503/508. (e.g. Foo__Bar.baz -> foo-bar-baz)
692You can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
693to find them as well (along with non-python packages).
694
695At this point we can create throwaway experimental Python environments with
696arbitrary dependencies. This is a good way to get a feel for how the Python
697interpreter and dependencies work in Nix and NixOS, but to do some actual
698development, we'll want to make it a bit more persistent.
699
700##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang}
701
702Sometimes, we have a script whose header looks like this:
703
704```python
705#!/usr/bin/env python3
706import numpy as np
707a = np.array([1,2])
708b = np.array([3,4])
709print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
710```
711
712Executing this script requires a `python3` that has `numpy`. Using what we learned
713in the previous section, we could startup a shell and just run it like so:
714
715```ShellSession
716$ nix-shell -p 'python313.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
717The dot product of [1 2] and [3 4] is: 11
718```
719
720But if we maintain the script ourselves, and if there are more dependencies, it
721may be nice to encode those dependencies in source to make the script re-usable
722without that bit of knowledge. That can be done by using `nix-shell` as a
723[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:
724
725```python
726#!/usr/bin/env nix-shell
727#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
728import numpy as np
729a = np.array([1,2])
730b = np.array([3,4])
731print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
732```
733
734Then we execute it, without requiring any environment setup at all!
735
736```sh
737$ ./foo.py
738The dot product of [1 2] and [3 4] is: 11
739```
740
741If the dependencies are not available on the host where `foo.py` is executed, it
742will build or download them from a Nix binary cache prior to starting up, prior
743that it is executed on a machine with a multi-user Nix installation.
744
745This provides a way to ship a self bootstrapping Python script, akin to a
746statically linked binary, where it can be run on any machine (provided nix is
747installed) without having to assume that `numpy` is installed globally on the
748system.
749
750By default it is pulling the import checkout of Nixpkgs itself from our nix
751channel, which is nice as it cache-aligns with our other package builds, but we
752can make it fully reproducible by pinning the `nixpkgs` import:
753
754```python
755#!/usr/bin/env nix-shell
756#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.numpy ])"
757#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/e51209796c4262bfb8908e3d6d72302fe4e96f5f.tar.gz
758import numpy as np
759a = np.array([1,2])
760b = np.array([3,4])
761print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
762```
763
764This will execute with the exact same versions of Python 3.10, numpy, and system
765dependencies a year from now as it does today, because it will always use
766exactly git commit `e51209796c4262bfb8908e3d6d72302fe4e96f5f` of Nixpkgs for all
767of the package versions.
768
769This is also a great way to ensure the script executes identically on different
770servers.
771
772##### Load environment from `.nix` expression {#load-environment-from-.nix-expression}
773
774We've now seen how to create an ad-hoc temporary shell session, and how to
775create a single script with Python dependencies, but in the course of normal
776development we're usually working in an entire package repository.
777
778As explained [in the `nix-shell` section](https://nixos.org/manual/nix/stable/command-ref/nix-shell) of the Nix manual, `nix-shell` can also load an expression from a `.nix` file.
779Say we want to have Python 3.13, `numpy` and `toolz`, like before,
780in an environment. We can add a `shell.nix` file describing our dependencies:
781
782```nix
783with import <nixpkgs> { };
784(python313.withPackages (
785 ps: with ps; [
786 numpy
787 toolz
788 ]
789)).env
790```
791
792And then at the command line, just typing `nix-shell` produces the same
793environment as before. In a normal project, we'll likely have many more
794dependencies; this can provide a way for developers to share the environments
795with each other and with CI builders.
796
797What's happening here?
798
7991. We begin with importing the Nix Packages collections. `import <nixpkgs>`
800 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
801 brings all attributes of `nixpkgs` in the local scope. These attributes form
802 the main package set.
8032. Then we create a Python 3.13 environment with the [`withPackages`](#python.withpackages-function) function, as before.
8043. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument
805 that takes the set of all Python packages and returns a list of packages to
806 include in the environment. Here, we select the packages `numpy` and `toolz`
807 from the package set.
808
809To combine this with `mkShell` you can:
810
811```nix
812with import <nixpkgs> { };
813let
814 pythonEnv = python313.withPackages (ps: [
815 ps.numpy
816 ps.toolz
817 ]);
818in
819mkShell {
820 packages = [
821 pythonEnv
822
823 black
824 mypy
825
826 libffi
827 openssl
828 ];
829}
830```
831
832This will create a unified environment that has not just our Python interpreter
833and its Python dependencies, but also tools like `black` or `mypy` and libraries
834like `libffi` and `openssl` in scope. This is generic and can span any number of
835tools or languages across the Nixpkgs ecosystem.
836
837##### Installing environments globally on the system {#installing-environments-globally-on-the-system}
838
839Up to now, we've been creating environments scoped to an ad-hoc shell session,
840or a single script, or a single project. This is generally advisable, as it
841avoids pollution across contexts.
842
843However, sometimes we know we will often want a Python with some basic packages,
844and want this available without having to enter into a shell or build context.
845This can be useful to have things like vim/emacs editors and plugins or shell
846tools "just work" without having to set them up, or when running other software
847that expects packages to be installed globally.
848
849To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/`
850that looks like this:
851
852```nix
853# ~/.config/nixpkgs/overlays/myEnv.nix
854self: super: {
855 myEnv = super.buildEnv {
856 name = "myEnv";
857 paths = [
858 # A Python 3 interpreter with some packages
859 (self.python3.withPackages (
860 ps: with ps; [
861 pyflakes
862 pytest
863 black
864 ]
865 ))
866
867 # Some other packages we'd like as part of this env
868 self.mypy
869 self.black
870 self.ripgrep
871 self.tmux
872 ];
873 };
874}
875```
876
877You can then build and install this to your profile with:
878
879```sh
880nix-env -iA myEnv
881```
882
883One limitation of this is that you can only have 1 Python env installed
884globally, since they conflict on the `python` to load out of your `PATH`.
885
886If you get a conflict or prefer to keep the setup clean, you can have `nix-env`
887atomically *uninstall* all other imperatively installed packages and replace
888your profile with just `myEnv` by using the `--replace` flag.
889
890##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix}
891
892For the sake of completeness, here's how to install the environment system-wide
893on NixOS.
894
895```nix
896{
897 # ...
898
899 environment.systemPackages = with pkgs; [
900 (python314.withPackages (
901 ps: with ps; [
902 numpy
903 toolz
904 ]
905 ))
906 ];
907}
908```
909
910### Developing with Python {#developing-with-python}
911
912Above, we were mostly just focused on use cases and what to do to get started
913creating working Python environments in nix.
914
915Now that you know the basics to be up and running, it is time to take a step
916back and take a deeper look at how Python packages are packaged on Nix.
917
918#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
919
920With Nix all packages are built by functions. The main function in Nix for
921building Python libraries is [`buildPythonPackage`](#buildpythonpackage-function). Let's see how we can build the
922`toolz` package.
923
924```nix
925{
926 lib,
927 buildPythonPackage,
928 fetchPypi,
929 setuptools,
930}:
931
932buildPythonPackage (finalAttrs: {
933 pname = "toolz";
934 version = "0.10.0";
935 pyproject = true;
936
937 src = fetchPypi {
938 inherit (finalAttrs) pname version;
939 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
940 };
941
942 build-system = [ setuptools ];
943
944 # has no tests
945 doCheck = false;
946
947 pythonImportsCheck = [
948 "toolz.itertoolz"
949 "toolz.functoolz"
950 "toolz.dicttoolz"
951 ];
952
953 meta = {
954 changelog = "https://github.com/pytoolz/toolz/releases/tag/${finalAttrs.version}";
955 homepage = "https://github.com/pytoolz/toolz";
956 description = "List processing tools and functional utilities";
957 license = lib.licenses.bsd3;
958 };
959})
960```
961
962What happens here? The function [`buildPythonPackage`](#buildpythonpackage-function) is called and as argument
963it accepts a set. In this case the set is a recursive set, `rec`. One of the
964arguments is the name of the package, which consists of a basename (generally
965following the name on PyPI) and a version. Another argument, `src` specifies the
966source, which in this case is fetched from PyPI using the helper function
967`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
968when building the package. Since there are no tests, we rely on [`pythonImportsCheck`](#using-pythonimportscheck)
969to test whether the package can be imported. Furthermore, we specify some meta
970information. The output of the function is a derivation.
971
972An expression for `toolz` can be found in the Nixpkgs repository. As explained
973in the introduction of this Python section, a derivation of `toolz` is available
974for each interpreter version, e.g. `python313.pkgs.toolz` refers to the `toolz`
975derivation corresponding to the CPython 3.13 interpreter.
976
977The above example works when you're directly working on
978`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
979you will want to test a Nix expression outside of the Nixpkgs tree.
980
981The following expression creates a derivation for the `toolz` package,
982and adds it along with a `numpy` package to a Python environment.
983
984```nix
985with import <nixpkgs> { };
986
987(
988 let
989 my_toolz = python313.pkgs.buildPythonPackage (finalAttrs: {
990 pname = "toolz";
991 version = "0.10.0";
992 pyproject = true;
993
994 src = fetchPypi {
995 inherit (finalAttrs) pname version;
996 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
997 };
998
999 build-system = [ python313.pkgs.setuptools ];
1000
1001 # has no tests
1002 doCheck = false;
1003
1004 meta = {
1005 homepage = "https://github.com/pytoolz/toolz/";
1006 description = "List processing tools and functional utilities";
1007 # [...]
1008 };
1009 });
1010
1011 in
1012 python313.withPackages (
1013 ps: with ps; [
1014 numpy
1015 my_toolz
1016 ]
1017 )
1018).env
1019```
1020
1021Executing `nix-shell` will result in an environment in which you can use
1022Python 3.13 and the `toolz` package. As you can see we had to explicitly mention
1023for which Python version we want to build a package.
1024
1025So, what did we do here? Well, we took the Nix expression that we used earlier
1026to build a Python environment, and said that we wanted to include our own
1027version of `toolz`, named `my_toolz`. To introduce our own package in the scope
1028of [`withPackages`](#python.withpackages-function) we used a `let` expression. You can see that we used
1029`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
1030`toolz` from the Nixpkgs package set this time, but instead took our own version
1031that we introduced with the `let` expression.
1032
1033#### Handling dependencies {#handling-dependencies}
1034
1035Our example, `toolz`, does not have any dependencies on other Python packages or system libraries.
1036[`buildPythonPackage`](#buildpythonpackage-function) uses the following arguments in the following circumstances:
1037
1038- `dependencies` - For Python runtime dependencies.
1039- `build-system` - For Python build-time requirements.
1040- [`buildInputs`](#var-stdenv-buildInputs) - For non-Python build-time requirements.
1041- [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) - For test dependencies
1042
1043Dependencies can belong to multiple arguments, for example if something is both a build time requirement & a runtime dependency.
1044
1045The following example shows which arguments are given to [`buildPythonPackage`](#buildpythonpackage-function) in
1046order to build [`datashape`](https://github.com/blaze/datashape).
1047
1048```nix
1049{
1050 lib,
1051 buildPythonPackage,
1052 fetchPypi,
1053
1054 # build dependencies
1055 setuptools,
1056
1057 # dependencies
1058 numpy,
1059 multipledispatch,
1060 python-dateutil,
1061
1062 # tests
1063 pytestCheckHook,
1064}:
1065
1066buildPythonPackage (finalAttrs: {
1067 pname = "datashape";
1068 version = "0.4.7";
1069 pyproject = true;
1070
1071 src = fetchPypi {
1072 inherit (finalAttrs) pname version;
1073 hash = "sha256-FLLvdm1MllKrgTGC6Gb0k0deZeVYvtCCLji/B7uhong=";
1074 };
1075
1076 build-system = [ setuptools ];
1077
1078 dependencies = [
1079 multipledispatch
1080 numpy
1081 python-dateutil
1082 ];
1083
1084 nativeCheckInputs = [ pytestCheckHook ];
1085
1086 meta = {
1087 changelog = "https://github.com/blaze/datashape/releases/tag/${finalAttrs.version}";
1088 homepage = "https://github.com/ContinuumIO/datashape";
1089 description = "Data description language";
1090 license = lib.licenses.bsd2;
1091 };
1092})
1093```
1094
1095We can see several runtime dependencies, `numpy`, `multipledispatch`, and
1096`python-dateutil`. Furthermore, we have [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) with `pytestCheckHook`.
1097`pytestCheckHook` is a test runner hook and is only used during the [`checkPhase`](#ssec-check-phase) and is
1098therefore not added to `dependencies`.
1099
1100In the previous case we had only dependencies on other Python packages to consider.
1101Occasionally you have also system libraries to consider. E.g., `lxml` provides
1102Python bindings to `libxml2` and `libxslt`. These libraries are only required
1103when building the bindings and are therefore added as [`buildInputs`](#var-stdenv-buildInputs).
1104
1105```nix
1106{
1107 lib,
1108 buildPythonPackage,
1109 fetchPypi,
1110 setuptools,
1111 libxml2,
1112 libxslt,
1113}:
1114
1115buildPythonPackage (finalAttrs: {
1116 pname = "lxml";
1117 version = "3.4.4";
1118 pyproject = true;
1119
1120 src = fetchPypi {
1121 inherit (finalAttrs) pname version;
1122 hash = "sha256-s9NiusRxFydHzaNRMjjxFcvWxfi45jGb9ql6eJJyQJk=";
1123 };
1124
1125 build-system = [ setuptools ];
1126
1127 buildInputs = [
1128 libxml2
1129 libxslt
1130 ];
1131
1132 # tests are meant to be ran "in-place" in the same directory as src
1133 doCheck = false;
1134
1135 pythonImportsCheck = [
1136 "lxml"
1137 "lxml.etree"
1138 ];
1139
1140 meta = {
1141 changelog = "https://github.com/lxml/lxml/releases/tag/lxml-${finalAttrs.version}";
1142 description = "Pythonic binding for the libxml2 and libxslt libraries";
1143 homepage = "https://lxml.de";
1144 license = lib.licenses.bsd3;
1145 maintainers = with lib.maintainers; [ sjourdois ];
1146 };
1147})
1148```
1149
1150In this example `lxml` and Nix are able to work out exactly where the relevant
1151files of the dependencies are. This is not always the case.
1152
1153The example below shows bindings to The Fastest Fourier Transform in the West,
1154commonly known as FFTW. On Nix we have separate packages of FFTW for the
1155different types of floats (`"single"`, `"double"`, `"long-double"`). The
1156bindings need all three types, and therefore we add all three as [`buildInputs`](#var-stdenv-buildInputs).
1157The bindings don't expect to find each of them in a different folder, and
1158therefore we have to set `LDFLAGS` and `CFLAGS`.
1159
1160```nix
1161{
1162 lib,
1163 buildPythonPackage,
1164 fetchPypi,
1165
1166 # build dependencies
1167 setuptools,
1168
1169 # dependencies
1170 fftw,
1171 fftwFloat,
1172 fftwLongDouble,
1173 numpy,
1174 scipy,
1175}:
1176
1177buildPythonPackage (finalAttrs: {
1178 pname = "pyfftw";
1179 version = "0.9.2";
1180 pyproject = true;
1181
1182 src = fetchPypi {
1183 inherit (finalAttrs) pname version;
1184 hash = "sha256-9ru2r6kwhUCaskiFoaPNuJCfCVoUL01J40byvRt4kHQ=";
1185 };
1186
1187 build-system = [ setuptools ];
1188
1189 buildInputs = [
1190 fftw
1191 fftwFloat
1192 fftwLongDouble
1193 ];
1194
1195 dependencies = [
1196 numpy
1197 scipy
1198 ];
1199
1200 preConfigure = ''
1201 export LDFLAGS="-L${fftw.dev}/lib -L${fftwFloat.out}/lib -L${fftwLongDouble.out}/lib"
1202 export CFLAGS="-I${fftw.dev}/include -I${fftwFloat.dev}/include -I${fftwLongDouble.dev}/include"
1203 '';
1204
1205 # Tests cannot import pyfftw. pyfftw works fine though.
1206 doCheck = false;
1207
1208 pythonImportsCheck = [ "pyfftw" ];
1209
1210 meta = {
1211 changelog = "https://github.com/pyFFTW/pyFFTW/releases/tag/v${finalAttrs.version}";
1212 description = "Pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
1213 homepage = "http://hgomersall.github.com/pyFFTW";
1214 license = with lib.licenses; [
1215 bsd2
1216 bsd3
1217 ];
1218 };
1219})
1220```
1221
1222Note also the line [`doCheck = false;`](#var-stdenv-doCheck), we explicitly disabled running the test-suite.
1223
1224#### Testing Python Packages {#testing-python-packages}
1225
1226It is highly encouraged to have testing as part of the package build. This
1227helps to avoid situations where the package was able to build and install,
1228but is not usable at runtime.
1229Your package should provide its own [`checkPhase`](#ssec-check-phase).
1230
1231::: {.note}
1232The [`checkPhase`](#ssec-check-phase) for python maps to the `installCheckPhase` on a
1233normal derivation. This is due to many python packages not behaving well
1234to the pre-installed version of the package. Version info, and natively
1235compiled extensions generally only exist in the install directory, and
1236thus can cause issues when a test suite asserts on that behavior.
1237:::
1238
1239::: {.note}
1240Tests should only be disabled if they don't agree with nix
1241(e.g. external dependencies, network access, flakey tests), however,
1242as many tests should be enabled as possible. Failing tests can still be
1243a good indication that the package is not in a valid state.
1244:::
1245
1246::: {.note}
1247We only want to test the functionality of a package. In particular, we are not
1248interested in coverage, formatting, and type checking. If pytest fails with
1249`unrecognized arguments: --cov`, add `pytest-cov-stub` to `nativeCheckInputs`
1250rather than `pytest-cov`.
1251:::
1252
1253#### Using pytest {#using-pytest}
1254
1255Pytest is the most common test runner for python repositories. A trivial
1256test run would be:
1257
1258```nix
1259{
1260 nativeCheckInputs = [ pytest ];
1261 checkPhase = ''
1262 runHook preCheck
1263
1264 pytest
1265
1266 runHook postCheck
1267 '';
1268}
1269```
1270
1271However, many repositories' test suites do not translate well to nix's build
1272sandbox, and will generally need many tests to be disabled.
1273
1274This is achievable by
1275- Including paths or test items (`path/to/file.py::MyClass` or `path/to/file.py::MyClass::test_method`) with positional arguments.
1276- Excluding paths with `--ignore` or globbed paths with `--ignore-glob`.
1277- Excluding test items using the `--deselect` flag.
1278- Including or excluding classes or test methods by their name using the `-k` flag.
1279- Including or excluding test by their marks using the `-m` flag.
1280
1281We highly recommend `pytestCheckHook` for an easier and more structural setup.
1282
1283#### Using pytestCheckHook {#using-pytestcheckhook}
1284
1285`pytestCheckHook` is a convenient hook which will set up (or configure)
1286a [`checkPhase`](#ssec-check-phase) to run `pytest`. This is also beneficial
1287when a package may need many items disabled to run the test suite.
1288Most packages use `pytest` or `unittest`, which is compatible with `pytest`,
1289so you will most likely use `pytestCheckHook`.
1290
1291To use `pytestCheckHook`, add it to `nativeCheckInputs`.
1292Adding `pytest` is not required, since it is included with `pytestCheckHook`.
1293
1294```nix
1295{ nativeCheckInputs = [ pytestCheckHook ]; }
1296```
1297
1298`pytestCheckHook` recognizes the following attributes:
1299
1300`enabledTestPaths` and `disabledTestPaths`
1301
1302: To specify path globs (files or directories) or test items.
1303
1304`enabledTests` and `disabledTests`
1305
1306: To specify keywords for class names or test method names.
1307
1308`enabledTestMarks` and `disabledTestMarks`
1309
1310: To specify test marks.
1311
1312`pytestFlags`
1313
1314: To append additional command-line arguments to `pytest`.
1315
1316By default, `pytest` automatically discovers which tests to run.
1317If tests are explicitly enabled, only those tests will run.
1318A test, that is both enabled and disabled, will not run.
1319
1320The following example demonstrates usage of various `pytestCheckHook` attributes:
1321
1322```nix
1323{
1324 nativeCheckInputs = [ pytestCheckHook ];
1325
1326 # Allow running the following test paths and test objects.
1327 enabledTestPaths = [
1328 # Find tests under the tests directory.
1329 # The trailing slash is not necessary.
1330 "tests/"
1331
1332 # Additionally run test_foo
1333 "other-tests/test_foo.py::Foo::test_foo"
1334 ];
1335
1336 # Override the above-enabled test paths and test objects.
1337 disabledTestPaths = [
1338 # Tests under tests/integration requires additional data.
1339 "tests/integration"
1340 ];
1341
1342 # Allow tests by keywords matching their class names or method names.
1343 enabledTests = [
1344 # pytest by default only runs test methods begin with "test_" or end with "_test".
1345 # This includes all functions whose name contains "test".
1346 "test"
1347 ];
1348
1349 # Override the above-enabled tests by keywords matching their class names or method names.
1350 disabledTests = [
1351 # Tests touching networks.
1352 "upload"
1353 "download"
1354 ];
1355
1356 # Additional pytest flags
1357 pytestFlags = [
1358 # Disable benchmarks and run benchmarking tests only once.
1359 "--benchmark-disable"
1360 ];
1361}
1362```
1363
1364These attributes are all passed into the derivation directly
1365and added to the `pytest` command without additional Bash expansion.
1366It requires `__structuredAttrs = true` to pass list elements containing spaces.
1367
1368The `<enabled/disabled>TestsPaths` attributes expand Unix-style globs.
1369If a test path contains characters like `*`, `?`, `[`, or `]`, you can
1370quote them with square brackets (`[*]`, `[?]`, `[[]`, and `[]]`) to match literally.
1371
1372The `<enabled/disabled>Tests` and `<enabled/disabled>TestMarks` attribute pairs
1373form a logical expression `((included_element1) or (included_element2)) and not (excluded_element1) and not (excluded_element2)`
1374which will be passed to pytest's `-k` and `-m` flags respectively.
1375With `__structuredAttrs = true` enabled, they additionally support sub-expressions.
1376
1377For example, you could disable test items like `TestFoo::test_bar_functionality`
1378by disabling tests that match both `"Foo"` **and** `"bar"`:
1379
1380```nix
1381{
1382 __structuredAttrs = true;
1383
1384 disabledTests = [ "Foo and bar" ];
1385}
1386```
1387
1388The main benefits of using `pytestCheckHook` to construct `pytest` commands
1389is structuralization and eval-time accessibility.
1390This is especially helpful to select tests or specify flags conditionally:
1391
1392```nix
1393{
1394 disabledTests = [
1395 # touches network
1396 "download"
1397 "update"
1398 ]
1399 ++ lib.optionals (pythonAtLeast "3.8") [
1400 # broken due to python3.8 async changes
1401 "async"
1402 ]
1403 ++ lib.optionals stdenv.buildPlatform.isDarwin [
1404 # can fail when building with other packages
1405 "socket"
1406 ];
1407}
1408```
1409
1410#### Using pythonImportsCheck {#using-pythonimportscheck}
1411
1412Although unit tests are highly preferred to validate correctness of a package, not
1413all packages have test suites that can be run easily, and some have none at all.
1414To help ensure the package still works, [`pythonImportsCheck`](#using-pythonimportscheck) can attempt to import
1415the listed modules.
1416
1417```nix
1418{
1419 pythonImportsCheck = [
1420 "requests"
1421 "urllib"
1422 ];
1423}
1424```
1425
1426roughly translates to:
1427
1428```nix
1429{
1430 postCheck = ''
1431 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH
1432 python -c "import requests; import urllib"
1433 '';
1434}
1435```
1436
1437However, this is done in its own phase, and not dependent on whether [`doCheck = true;`](#var-stdenv-doCheck).
1438
1439This can also be useful in verifying that the package doesn't assume commonly
1440present packages (e.g. `setuptools`).
1441
1442#### Using pythonRelaxDepsHook {#using-pythonrelaxdepshook}
1443
1444It is common for upstream to specify a range of versions for its package
1445dependencies. This makes sense, since it ensures that the package will be built
1446with a subset of packages that is well tested. However, this commonly causes
1447issues when packaging in Nixpkgs, because the dependencies that this package
1448may need are too new or old for the package to build correctly. We also cannot
1449package multiple versions of the same package since this may cause conflicts
1450in `PYTHONPATH`.
1451
1452One way to side step this issue is to relax the dependencies. This can be done
1453by either removing the package version range or by removing the package
1454declaration entirely. This can be done using the `pythonRelaxDepsHook` hook. For
1455example, given the following `requirements.txt` file:
1456
1457```
1458pkg1<1.0
1459pkg2
1460pkg3>=1.0,<=2.0
1461```
1462
1463we can do:
1464
1465```nix
1466{
1467 pythonRelaxDeps = [
1468 "pkg1"
1469 "pkg3"
1470 ];
1471 pythonRemoveDeps = [ "pkg2" ];
1472}
1473```
1474
1475which would result in the following `requirements.txt` file:
1476
1477```
1478pkg1
1479pkg3
1480```
1481
1482Another option is to pass `true`, that will relax/remove all dependencies, for
1483example:
1484
1485```nix
1486{ pythonRelaxDeps = true; }
1487```
1488
1489which would result in the following `requirements.txt` file:
1490
1491```
1492pkg1
1493pkg2
1494pkg3
1495```
1496
1497In general you should always use `pythonRelaxDeps`, because `pythonRemoveDeps`
1498will convert build errors into runtime errors. However `pythonRemoveDeps` may
1499still be useful in exceptional cases, and also to remove dependencies wrongly
1500declared by upstream (for example, declaring `black` as a runtime dependency
1501instead of a dev dependency).
1502
1503Keep in mind that while the examples above are done with `requirements.txt`,
1504`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should
1505work with any of the [existing hooks](#setup-hooks).
1506
1507The `pythonRelaxDepsHook` has no effect on build time dependencies, such as
1508those specified in `build-system`. If a package requires incompatible build
1509time dependencies, they should be removed in `postPatch` through
1510`substituteInPlace` or similar.
1511
1512For ease of use, both `buildPythonPackage` and `buildPythonApplication` will
1513automatically add `pythonRelaxDepsHook` if either `pythonRelaxDeps` or
1514`pythonRemoveDeps` is specified.
1515
1516#### Using unittestCheckHook {#using-unittestcheckhook}
1517
1518`unittestCheckHook` is a hook which will set up (or configure) a [`checkPhase`](#ssec-check-phase) to run `python -m unittest discover`:
1519
1520```nix
1521{
1522 nativeCheckInputs = [ unittestCheckHook ];
1523
1524 unittestFlags = [
1525 "-s"
1526 "tests"
1527 "-v"
1528 ];
1529}
1530```
1531
1532`pytest` is compatible with `unittest`, so in most cases you can use `pytestCheckHook` instead.
1533
1534#### Using sphinxHook {#using-sphinxhook}
1535
1536The `sphinxHook` is a helpful tool to build documentation and manpages
1537using the popular Sphinx documentation generator.
1538It is setup to automatically find common documentation source paths and
1539render them using the default `html` style.
1540
1541```nix
1542{
1543 outputs = [
1544 "out"
1545 "doc"
1546 ];
1547
1548 nativeBuildInputs = [ sphinxHook ];
1549}
1550```
1551
1552The hook will automatically build and install the artifact into the
1553`doc` output, if it exists. It also provides an automatic diversion
1554for the artifacts of the `man` builder into the `man` target.
1555
1556```nix
1557{
1558 outputs = [
1559 "out"
1560 "doc"
1561 "man"
1562 ];
1563
1564 # Use multiple builders
1565 sphinxBuilders = [
1566 "singlehtml"
1567 "man"
1568 ];
1569}
1570```
1571
1572Overwrite `sphinxRoot` when the hook is unable to find your
1573documentation source root.
1574
1575```nix
1576{
1577 # Configure sphinxRoot for uncommon paths
1578 sphinxRoot = "weird/docs/path";
1579}
1580```
1581
1582The hook is also available to packages outside the python ecosystem by
1583referencing it using `sphinxHook` from top-level.
1584
1585### Organising your packages {#organising-your-packages}
1586
1587So far we discussed how you can use Python on Nix, and how you can develop with
1588it. We've looked at how you write expressions to package Python packages, and we
1589looked at how you can create environments in which specified packages are
1590available.
1591
1592At some point you'll likely have multiple packages which you would
1593like to be able to use in different projects. In order to minimise unnecessary
1594duplication we now look at how you can maintain a repository with your
1595own packages. The important functions here are `import` and `callPackage`.
1596
1597### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
1598
1599Earlier we created a Python environment using [`withPackages`](#python.withpackages-function), and included the
1600`toolz` package via a `let` expression.
1601Let's split the package definition from the environment definition.
1602
1603We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
1604
1605```nix
1606{
1607 lib,
1608 buildPythonPackage,
1609 fetchPypi,
1610 setuptools,
1611}:
1612
1613buildPythonPackage (finalAttrs: {
1614 pname = "toolz";
1615 version = "0.10.0";
1616 pyproject = true;
1617
1618 src = fetchPypi {
1619 inherit (finalAttrs) pname version;
1620 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA=";
1621 };
1622
1623 build-system = [ setuptools ];
1624
1625 meta = {
1626 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}";
1627 homepage = "https://github.com/pytoolz/toolz/";
1628 description = "List processing tools and functional utilities";
1629 license = lib.licenses.bsd3;
1630 };
1631})
1632```
1633
1634It takes an argument [`buildPythonPackage`](#buildpythonpackage-function). We now call this function using
1635`callPackage` in the definition of our environment
1636
1637```nix
1638with import <nixpkgs> { };
1639
1640(
1641 let
1642 toolz = callPackage /path/to/toolz/release.nix {
1643 buildPythonPackage = python3Packages.buildPythonPackage;
1644 };
1645 in
1646 python3.withPackages (ps: [
1647 ps.numpy
1648 toolz
1649 ])
1650).env
1651```
1652
1653Important to remember is that the Python version for which the package is made
1654depends on the `python` derivation that is passed to [`buildPythonPackage`](#buildpythonpackage-function). Nix
1655tries to automatically pass arguments when possible, which is why generally you
1656don't explicitly define which `python` derivation should be used. In the above
1657example we use [`buildPythonPackage`](#buildpythonpackage-function) that is part of the set `python3Packages`,
1658and in this case the `python3` interpreter is automatically used.
1659
1660## FAQ {#faq}
1661
1662### How to solve circular dependencies? {#how-to-solve-circular-dependencies}
1663
1664Consider the packages `A` and `B` that depend on each other. When packaging `B`,
1665a solution is to override package `A` not to depend on `B` as an input. The same
1666should also be done when packaging `A`.
1667
1668### How to override a Python package? {#how-to-override-a-python-package}
1669
1670We can override the interpreter and pass `packageOverrides`. In the following
1671example we rename the `pandas` package and build it.
1672
1673```nix
1674with import <nixpkgs> { };
1675
1676(
1677 let
1678 python =
1679 let
1680 packageOverrides = self: super: {
1681 pandas = super.pandas.overridePythonAttrs (old: {
1682 name = "foo";
1683 });
1684 };
1685 in
1686 pkgs.python313.override { inherit packageOverrides; };
1687
1688 in
1689 python.withPackages (ps: [ ps.pandas ])
1690).env
1691```
1692
1693Using `nix-build` on this expression will build an environment that contains the
1694package `pandas` but with the new name `foo`.
1695
1696All packages in the package set will use the renamed package. A typical use case
1697is to switch to another version of a certain package. For example, in the
1698Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
1699following example we use a different version of `scipy` and create an
1700environment that uses it. All packages in the Python package set will now use
1701the updated `scipy` version.
1702
1703```nix
1704with import <nixpkgs> { };
1705
1706(
1707 let
1708 packageOverrides = self: super: { scipy = super.scipy_0_17; };
1709 in
1710 (pkgs.python313.override { inherit packageOverrides; }).withPackages (ps: [ ps.blaze ])
1711).env
1712```
1713
1714The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
1715
1716If you want the whole of Nixpkgs to use your modifications, then you can use
1717`overlays` as explained in this manual. In the following example we build a
1718`inkscape` using a different version of `numpy`.
1719
1720```nix
1721let
1722 pkgs = import <nixpkgs> { };
1723 newpkgs = import pkgs.path {
1724 overlays = [
1725 (self: super: {
1726 python313 =
1727 let
1728 packageOverrides = python-self: python-super: {
1729 numpy = python-super.numpy_1_18;
1730 };
1731 in
1732 super.python313.override { inherit packageOverrides; };
1733 })
1734 ];
1735 };
1736in
1737newpkgs.inkscape
1738```
1739
1740### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl}
1741
1742Executing `python setup.py bdist_wheel` in a `nix-shell`fails with
1743
1744```
1745ValueError: ZIP does not support timestamps before 1980
1746```
1747
1748This is because files from the Nix store (which have a timestamp of the UNIX
1749epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
1750DOS convention of counting timestamps from 1980.
1751
1752The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
1753which `nix-shell` sets to 1. Unsetting this variable or giving it a value
1754corresponding to 1980 or later enables building wheels.
1755
1756Use 1980 as timestamp:
1757
1758```shell
1759nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"
1760```
1761
1762or the current time:
1763
1764```shell
1765nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"
1766```
1767
1768or unset `SOURCE_DATE_EPOCH`:
1769
1770```shell
1771nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"
1772```
1773
1774### `install_data` / `data_files` problems {#install_data-data_files-problems}
1775
1776If you get the following error:
1777
1778```
1779could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
1780Permission denied
1781```
1782
1783This is a [known bug](https://github.com/pypa/setuptools/issues/130) in
1784`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example
1785of such package using the feature is `pkgs/tools/X11/xpra/default.nix`.
1786
1787As workaround install it as an extra `preInstall` step:
1788
1789```shell
1790${python.pythonOnBuildForHost.interpreter} setup.py install_data --install-dir=$out --root=$out
1791sed -i '/ = data\_files/d' setup.py
1792```
1793
1794### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages}
1795
1796On most operating systems a global `site-packages` is maintained. This however
1797becomes problematic if you want to run multiple Python versions or have multiple
1798versions of certain libraries for your projects. Generally, you would solve such
1799issues by creating virtual environments using `virtualenv`.
1800
1801On Nix each package has an isolated dependency tree which, in the case of
1802Python, guarantees the right versions of the interpreter and libraries or
1803packages are available. There is therefore no need to maintain a global `site-packages`.
1804
1805If you want to create a Python environment for development, then the recommended
1806method is to use `nix-shell`, either with or without the [`python.buildEnv`](#python.buildenv-function)
1807function.
1808
1809### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
1810
1811While this approach is not very idiomatic from Nix perspective, it can still be
1812useful when dealing with pre-existing projects or in situations where it's not
1813feasible or desired to write derivations for all required dependencies.
1814
1815This is an example of a `default.nix` for a `nix-shell`, which allows to consume
1816a virtual environment created by `venv`, and install Python modules through
1817`pip` the traditional way.
1818
1819Create this `default.nix` file, together with a `requirements.txt` and
1820execute `nix-shell`.
1821
1822```nix
1823with import <nixpkgs> { };
1824
1825let
1826 pythonPackages = python3Packages;
1827in
1828pkgs.mkShell rec {
1829 name = "impurePythonEnv";
1830 venvDir = "./.venv";
1831 buildInputs = [
1832 # A Python interpreter including the 'venv' module is required to bootstrap
1833 # the environment.
1834 pythonPackages.python
1835
1836 # This executes some shell code to initialize a venv in $venvDir before
1837 # dropping into the shell
1838 pythonPackages.venvShellHook
1839
1840 # Those are dependencies that we would like to use from nixpkgs, which will
1841 # add them to PYTHONPATH and thus make them accessible from within the venv.
1842 pythonPackages.numpy
1843 pythonPackages.requests
1844
1845 # In this particular example, in order to compile any binary extensions they may
1846 # require, the Python modules listed in the hypothetical requirements.txt need
1847 # the following packages to be installed locally:
1848 taglib
1849 openssl
1850 git
1851 libxml2
1852 libxslt
1853 libzip
1854 zlib
1855 ];
1856
1857 # Run this command, only after creating the virtual environment
1858 postVenvCreation = ''
1859 unset SOURCE_DATE_EPOCH
1860 pip install -r requirements.txt
1861 '';
1862
1863 # Now we can execute any commands within the virtual environment.
1864 # This is optional and can be left out to run pip manually.
1865 postShellHook = ''
1866 # allow pip to install wheels
1867 unset SOURCE_DATE_EPOCH
1868 '';
1869
1870}
1871```
1872
1873In case the supplied venvShellHook is insufficient, or when Python 2 support is
1874needed, you can define your own shell hook and adapt to your needs like in the
1875following example:
1876
1877```nix
1878with import <nixpkgs> { };
1879
1880let
1881 venvDir = "./.venv";
1882 pythonPackages = python3Packages;
1883in
1884pkgs.mkShell rec {
1885 name = "impurePythonEnv";
1886 buildInputs = [
1887 pythonPackages.python
1888 # Needed when using python 2.7
1889 # pythonPackages.virtualenv
1890 # ...
1891 ];
1892
1893 # This is very close to how venvShellHook is implemented, but
1894 # adapted to use 'virtualenv'
1895 shellHook = ''
1896 SOURCE_DATE_EPOCH=$(date +%s)
1897
1898 if [ -d "${venvDir}" ]; then
1899 echo "Skipping venv creation, '${venvDir}' already exists"
1900 else
1901 echo "Creating new venv environment in path: '${venvDir}'"
1902 # Note that the module venv was only introduced in python 3, so for 2.7
1903 # this needs to be replaced with a call to virtualenv
1904 ${pythonPackages.python.interpreter} -m venv "${venvDir}"
1905 fi
1906
1907 # Under some circumstances it might be necessary to add your virtual
1908 # environment to PYTHONPATH, which you can do here too;
1909 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH
1910
1911 source "${venvDir}/bin/activate"
1912
1913 # As in the previous example, this is optional.
1914 pip install -r requirements.txt
1915 '';
1916}
1917```
1918
1919Note that the `pip install` is an imperative action. So every time `nix-shell`
1920is executed it will attempt to download the Python modules listed in
1921requirements.txt. However these will be cached locally within the `virtualenv`
1922folder and not downloaded again.
1923
1924### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix}
1925
1926If you need to change a package's attribute(s) from `configuration.nix` you could do:
1927
1928```nix
1929{
1930 nixpkgs.config.packageOverrides = super: {
1931 python3 = super.python3.override {
1932 packageOverrides = python-self: python-super: {
1933 twisted = python-super.twisted.overridePythonAttrs (oldAttrs: {
1934 src = super.fetchPypi {
1935 pname = "Twisted";
1936 version = "19.10.0";
1937 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1938 extension = "tar.bz2";
1939 };
1940 });
1941 };
1942 };
1943 };
1944}
1945```
1946
1947`python3Packages.twisted` is now globally overridden.
1948All packages and also all NixOS services that reference `twisted`
1949(such as `services.buildbot-worker`) now use the new definition.
1950Note that `python-super` refers to the old package set and `python-self`
1951to the new, overridden version.
1952
1953To modify only a Python package set instead of a whole Python derivation, use
1954this snippet:
1955
1956```nix
1957{
1958 myPythonPackages = python3Packages.override { overrides = self: super: { twisted = <...>; }; };
1959}
1960```
1961
1962### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays}
1963
1964Use the following overlay template:
1965
1966```nix
1967self: super: {
1968 python = super.python.override {
1969 packageOverrides = python-self: python-super: {
1970 twisted = python-super.twisted.overrideAttrs (oldAttrs: {
1971 src = super.fetchPypi {
1972 pname = "Twisted";
1973 version = "19.10.0";
1974 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
1975 extension = "tar.bz2";
1976 };
1977 });
1978 };
1979 };
1980}
1981```
1982
1983### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions}
1984
1985The following overlay overrides the call to [`buildPythonPackage`](#buildpythonpackage-function) for the
1986`foo` package for all interpreters by appending a Python extension to the
1987`pythonPackagesExtensions` list of extensions.
1988
1989```nix
1990final: prev: {
1991 pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
1992 (python-final: python-prev: {
1993 foo = python-prev.foo.overridePythonAttrs (oldAttrs: {
1994 # ...
1995 });
1996 })
1997 ];
1998}
1999```
2000
2001### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy}
2002
2003MKL can be configured using an overlay. See the section "[Using overlays to
2004configure alternatives](#sec-overlays-alternatives-blas-lapack)".
2005
2006### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to}
2007
2008In a `setup.py` or `setup.cfg` it is common to declare dependencies:
2009
2010* `setup_requires` corresponds to `build-system`
2011* `install_requires` corresponds to `dependencies`
2012* `tests_require` corresponds to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs)
2013
2014### How to enable interpreter optimizations? {#optimizations}
2015
2016The Python interpreters are by default not built with optimizations enabled, because
2017the builds are in that case not reproducible. To enable optimizations, override the
2018interpreter of interest, e.g using
2019
2020```nix
2021let
2022 pkgs = import ./. { };
2023 mypython = pkgs.python3.override {
2024 enableOptimizations = true;
2025 reproducibleBuild = false;
2026 self = mypython;
2027 };
2028in
2029mypython
2030```
2031
2032### How to add optional dependencies? {#python-optional-dependencies}
2033
2034Some packages define optional dependencies for additional features. With
2035`setuptools` this is called `extras_require` and `flit` calls it
2036`extras-require`, while PEP 621 calls these `optional-dependencies`.
2037
2038```nix
2039{
2040 optional-dependencies = {
2041 complete = [ distributed ];
2042 };
2043}
2044```
2045
2046and letting the package requiring the extra add the list to its dependencies
2047
2048```nix
2049{
2050 dependencies = [
2051 # ...
2052 ]
2053 ++ dask.optional-dependencies.complete;
2054}
2055```
2056
2057This method is using `passthru`, meaning that changing `optional-dependencies` of a package won't cause it to rebuild.
2058
2059Note this method is preferred over adding parameters to builders, as that can
2060result in packages depending on different variants and thereby causing
2061collisions.
2062
2063::: {.note}
2064The `optional-dependencies` attribute should only be used for dependency groups
2065as defined in package metadata. If a package gracefully handles missing
2066dependencies in runtime but doesn't advertise it through package metadata, then
2067these dependencies should not be listed at all. (One may still have to list
2068them in `nativeCheckInputs` to pass test suite.)
2069:::
2070
2071### How to contribute a Python package to nixpkgs? {#tools}
2072
2073Packages inside nixpkgs must use the [`buildPythonPackage`](#buildpythonpackage-function) or [`buildPythonApplication`](#buildpythonapplication-function) function directly,
2074because we can only provide security support for non-vendored dependencies.
2075
2076We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs,
2077as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`.
2078When using the tool, pull from the original source repository instead of PyPI, if possible.
2079
2080See also [contributing section](#contributing).
2081
2082### Are Python interpreters built deterministically? {#deterministic-builds}
2083
2084The Python interpreters are now built deterministically. Minor modifications had
2085to be made to the interpreters in order to generate deterministic bytecode. This
2086has security implications and is relevant for those using Python in a
2087`nix-shell`.
2088
2089When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
2090have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1`
2091and [PYTHONHASHSEED=0](https://docs.python.org/3.13/using/cmdline.html#envvar-PYTHONHASHSEED).
2092Both are also exported in `nix-shell`.
2093
2094### How to provide automatic tests to Python packages? {#automatic-tests}
2095
2096It is recommended to test packages as part of the build process.
2097Source distributions (`sdist`) often include test files, but not always.
2098
2099The best practice today is to pass a test hook (e.g. pytestCheckHook, unittestCheckHook) into nativeCheckInputs.
2100This will reconfigure the checkPhase to make use of that particular test framework.
2101Occasionally packages don't make use of a common test framework, which may then require a custom checkPhase.
2102
2103#### Common issues {#common-issues}
2104
2105* Tests that attempt to access `$HOME` can be fixed by using `writableTmpDirAsHomeHook` in
2106 `nativeCheckInputs`, which sets up a writable temporary directory as the home directory. Alternatively,
2107 you can achieve the same effect manually (e.g. in `preCheck`) with: `export HOME=$(mktemp -d)`.
2108* Compiling with Cython causes tests to fail with a `ModuleNotLoadedError`.
2109 This can be fixed with two changes in the derivation: 1) replacing `pytest` with
2110 `pytestCheckHook` and 2) adding a `preCheck` containing `cd $out` to run
2111 tests within the built output.
2112
2113## Contributing {#contributing}
2114
2115### Contributing guidelines {#contributing-guidelines}
2116
2117The following rules are desired to be respected:
2118
2119* Python libraries are called from `python-packages.nix` and packaged with
2120 [`buildPythonPackage`](#buildpythonpackage-function). The expression of a library should be in
2121 `pkgs/development/python-modules/<name>/default.nix`.
2122* Python applications live outside of `python-packages.nix` and are packaged
2123 with [`buildPythonApplication`](#buildpythonapplication-function).
2124* Make sure libraries build for all Python interpreters.
2125 If it fails to build on some Python versions, consider disabling them by setting `disable = pythonAtLeast "3.x"` along with a comment.
2126* The two parameters, `pyproject` and `build-system` are set to avoid the legacy setuptools/distutils build.
2127* Only unversioned attributes (e.g. `pydantic`, but not `pypdantic_1`) can be included in `dependencies`,
2128 since due to `PYTHONPATH` limitations we can only ever support a single version for libraries
2129 without running into duplicate module name conflicts.
2130* The version restrictions of `dependencies` can be relaxed by [`pythonRelaxDepsHook`](#using-pythonrelaxdepshook).
2131* Make sure the tests are enabled using for example [`pytestCheckHook`](#using-pytestcheckhook) and, in the case of
2132 libraries, are passing for all interpreters. If certain tests fail they can be
2133 disabled individually. Try to avoid disabling the tests altogether. In any
2134 case, when you disable tests, leave a comment explaining not only _what_ the failure
2135 is but _why_ the test failure can be ignored for safe distribution with nixpkgs.
2136* `pythonImportsCheck` is set. This is still a good smoke test even if `pytestCheckHook` is set.
2137* `meta.platforms` takes the default value in many cases.
2138 It does not need to be set explicitly unless the package requires a specific platform.
2139* The file is formatted correctly (e.g., `nix-shell --run treefmt`).
2140* Commit names of Python libraries must reflect that they are Python
2141 libraries (e.g. `python3Packages.numpy: 1.11 -> 1.12` rather than `numpy: 1.11 -> 1.12`).
2142 See also [`pkgs/README.md`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#commit-conventions).
2143* Attribute names in `python-packages.nix` as well as `pname`s should match the
2144 library's name on PyPI, but be normalized according to [PEP
2145 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means
2146 that characters should be converted to lowercase and `.` and `_` should be
2147 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz).
2148 If necessary, `pname` has to be given a different value within `fetchPypi`.
2149* It's generally preferable to fetch `src` directly from the repo and not from
2150 PyPI. Use `fetchPypi` when there's a clear technical reason to do so.
2151* Packages from sources such as GitHub and GitLab that do not exist on PyPI
2152 should not use a name that is already used on PyPI. When possible, they should
2153 use the package repository name prefixed with the owner (e.g. organization) name
2154 and using a `-` as delimiter.
2155* Attribute names in `python-packages.nix` should be sorted alphanumerically to
2156 avoid merge conflicts and ease locating attributes.
2157* Non-python runtime dependencies should be added via explicit wrapping or
2158 patching (using e.g. `substituteInPlace`), rather than through propagation via
2159 `dependencies`/`propagatedBuildInputs`, to reduce clutter in `$PATH`.
2160
2161This list is useful for reviewers as well as for self-checking when submitting packages.
2162
2163## Package set maintenance {#python-package-set-maintenance}
2164
2165The whole Python package set has a lot of packages that do not see regular
2166updates, because they either are a very fragile component in the Python
2167ecosystem, like for example the `hypothesis` package, or packages that have
2168no maintainer, so maintenance falls back to the package set maintainers.
2169
2170### Updating packages in bulk {#python-package-bulk-updates}
2171
2172A tool to bulk-update numerous Python libraries is available in the
2173repository at `maintainers/scripts/update-python-libraries`.
2174
2175It can quickly update minor or major versions for all packages selected
2176and create update commits, and supports the `fetchPypi`, `fetchurl` and
2177`fetchFromGitHub` fetchers. When updating lots of packages that are
2178hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended.
2179
2180Updating packages in bulk leads to lots of breakages, which is why a
2181stabilization period on the `python-updates` branch is required.
2182
2183If a package is fragile and often breaks during these bulks updates, it
2184may be reasonable to set `passthru.skipBulkUpdate = true` in the
2185derivation. This decision should not be made on a whim and should
2186always be supported by a qualifying comment.
2187
2188Once the branch is sufficiently stable it should normally be merged
2189into the `staging` branch.
2190
2191An exemplary call to update all python libraries between minor versions
2192would be:
2193
2194```ShellSession
2195$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix
2196```
2197
2198## CPython Update Schedule {#python-cpython-update-schedule}
2199
2200With [PEP 602](https://www.python.org/dev/peps/pep-0602/), CPython now
2201follows a yearly release cadence. In nixpkgs, all supported interpreters
2202are made available, but only the most recent two
2203interpreters package sets are built; this is a compromise between being
2204the latest interpreter, and what the majority of the Python packages support.
2205
2206New CPython interpreters are released in October. Generally, it takes some
2207time for the majority of active Python projects to support the latest stable
2208interpreter. To help ease the migration for Nixpkgs users
2209between Python interpreters the schedule below will be used:
2210
2211| When | Event |
2212| --- | --- |
2213| After YY.11 Release | Bump CPython package set window. The latest and previous latest stable should now be built. |
2214| After YY.05 Release | Bump default CPython interpreter to latest stable. |
2215
2216In practice, this means that the Python community will have had a stable interpreter
2217for ~2 months before attempting to update the package set. And this will
2218allow for ~7 months for Python applications to support the latest interpreter.