···11# Python {#python}
2233+## Reference {#reference}
44+55+### Interpreters {#interpreters}
66+77+| Package | Aliases | Interpreter |
88+|------------|-----------------|-------------|
99+| python27 | python2, python | CPython 2.7 |
1010+| python38 | | CPython 3.8 |
1111+| python39 | | CPython 3.9 |
1212+| python310 | python3 | CPython 3.10 |
1313+| python311 | | CPython 3.11 |
1414+| python312 | | CPython 3.12 |
1515+| pypy27 | pypy2, pypy | PyPy2.7 |
1616+| pypy39 | pypy3 | PyPy 3.9 |
1717+1818+The Nix expressions for the interpreters can be found in
1919+`pkgs/development/interpreters/python`.
2020+2121+All packages depending on any Python interpreter get appended
2222+`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
2323+exists.
2424+2525+#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
2626+2727+To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
2828+2929+#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
3030+3131+Each interpreter has the following attributes:
3232+3333+- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
3434+- `interpreter`. Alias for `${python}/bin/${executable}`.
3535+- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
3636+- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
3737+- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
3838+- `executable`. Name of the interpreter executable, e.g. `python3.10`.
3939+- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
4040+4141+### Building packages and applications {#building-packages-and-applications}
4242+4343+Python libraries and applications that use `setuptools` or
4444+`distutils` are typically built with respectively the `buildPythonPackage` and
4545+`buildPythonApplication` functions. These two functions also support installing a `wheel`.
4646+4747+All Python packages reside in `pkgs/top-level/python-packages.nix` and all
4848+applications elsewhere. In case a package is used as both a library and an
4949+application, then the package should be in `pkgs/top-level/python-packages.nix`
5050+since only those packages are made available for all interpreter versions. The
5151+preferred location for library expressions is in
5252+`pkgs/development/python-modules`. It is important that these packages are
5353+called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
5454+the right version of the package is built.
5555+5656+Based on the packages defined in `pkgs/top-level/python-packages.nix` an
5757+attribute set is created for each available Python interpreter. The available
5858+sets are
5959+6060+* `pkgs.python27Packages`
6161+* `pkgs.python3Packages`
6262+* `pkgs.python38Packages`
6363+* `pkgs.python39Packages`
6464+* `pkgs.python310Packages`
6565+* `pkgs.python311Packages`
6666+* `pkgs.pypyPackages`
6767+6868+and the aliases
6969+7070+* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
7171+* `pkgs.python3Packages` pointing to `pkgs.python310Packages`
7272+* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
7373+7474+#### `buildPythonPackage` function {#buildpythonpackage-function}
7575+7676+The `buildPythonPackage` function is implemented in
7777+`pkgs/development/interpreters/python/mk-python-derivation.nix`
7878+using setup hooks.
7979+8080+The following is an example:
8181+8282+```nix
8383+{ lib
8484+, buildPythonPackage
8585+, fetchPypi
8686+8787+# build-system
8888+, setuptools-scm
8989+9090+# dependencies
9191+, attrs
9292+, pluggy
9393+, py
9494+, setuptools
9595+, six
9696+9797+# tests
9898+, hypothesis
9999+ }:
100100+101101+buildPythonPackage rec {
102102+ pname = "pytest";
103103+ version = "3.3.1";
104104+ format = "setuptools";
105105+106106+ src = fetchPypi {
107107+ inherit pname version;
108108+ hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M=";
109109+ };
110110+111111+ postPatch = ''
112112+ # don't test bash builtins
113113+ rm testing/test_argcomplete.py
114114+ '';
115115+116116+ nativeBuildInputs = [
117117+ setuptools-scm
118118+ ];
119119+120120+ propagatedBuildInputs = [
121121+ attrs
122122+ py
123123+ setuptools
124124+ six
125125+ pluggy
126126+ ];
127127+128128+ nativeCheckInputs = [
129129+ hypothesis
130130+ ];
131131+132132+ meta = with lib; {
133133+ changelog = "https://github.com/pytest-dev/pytest/releases/tag/${version}";
134134+ description = "Framework for writing tests";
135135+ homepage = "https://github.com/pytest-dev/pytest";
136136+ license = licenses.mit;
137137+ maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ];
138138+ };
139139+}
140140+```
141141+142142+The `buildPythonPackage` mainly does four things:
143143+144144+* In the `buildPhase`, it calls `${python.pythonForBuild.interpreter} setup.py bdist_wheel` to
145145+ build a wheel binary zipfile.
146146+* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
147147+* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
148148+ wrap all programs in the `$out/bin/*` directory to include `$PATH`
149149+ environment variable and add dependent libraries to script's `sys.path`.
150150+* In the `installCheck` phase, `${python.interpreter} setup.py test` is run.
151151+152152+By default tests are run because `doCheck = true`. Test dependencies, like
153153+e.g. the test runner, should be added to `nativeCheckInputs`.
154154+155155+By default `meta.platforms` is set to the same value
156156+as the interpreter unless overridden otherwise.
157157+158158+##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
159159+160160+All parameters from `stdenv.mkDerivation` function are still supported. The
161161+following are specific to `buildPythonPackage`:
162162+163163+* `catchConflicts ? true`: If `true`, abort package build if a package name
164164+ appears more than once in dependency tree. Default is `true`.
165165+* `disabled ? false`: If `true`, package is not built for the particular Python
166166+ interpreter version.
167167+* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
168168+* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
169169+ variable in wrapped programs.
170170+* `format ? "setuptools"`: Format of the source. Valid options are
171171+ `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`.
172172+ `"setuptools"` is for when the source has a `setup.py` and `setuptools` is
173173+ used to build a wheel, `flit`, in case `flit` should be used to build a wheel,
174174+ and `wheel` in case a wheel is provided. Use `other` when a custom
175175+ `buildPhase` and/or `installPhase` is needed.
176176+* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
177177+ `makeWrapper`, which wraps generated binaries. By default, the arguments to
178178+ `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling
179179+ the binary. Additional arguments here can allow a developer to set environment
180180+ variables which will be available when the binary is run. For example,
181181+ `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
182182+* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
183183+ defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`.
184184+* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip
185185+ install`. To pass options to `python setup.py install`, use
186186+ `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`.
187187+* `pipBuildFlags ? []`: A list of strings. Arguments to be passed to `pip wheel`.
188188+* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
189189+* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
190190+ in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
191191+* `preShellHook`: Hook to execute commands before `shellHook`.
192192+* `postShellHook`: Hook to execute commands after `shellHook`.
193193+* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
194194+ created when the filenames end with `.py`.
195195+* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
196196+* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
197197+198198+The `stdenv.mkDerivation` function accepts various parameters for describing
199199+build inputs (see "Specifying dependencies"). The following are of special
200200+interest for Python packages, either because these are primarily used, or
201201+because their behaviour is different:
202202+203203+* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables
204204+ as well as the items listed in `setup_requires`.
205205+* `buildInputs ? []`: Build and/or run-time dependencies that need to be
206206+ compiled for the host machine. Typically non-Python libraries which are being
207207+ linked.
208208+* `nativeCheckInputs ? []`: Dependencies needed for running the `checkPhase`. These
209209+ are added to `nativeBuildInputs` when `doCheck = true`. Items listed in
210210+ `tests_require` go here.
211211+* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
212212+ `buildPythonPackage` also injects code into and wraps executables with the
213213+ paths included in this list. Items listed in `install_requires` go here.
214214+215215+##### Overriding Python packages {#overriding-python-packages}
216216+217217+The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
218218+used to override the package. In the following example we create an environment
219219+where we have the `blaze` package using an older version of `pandas`. We
220220+override first the Python interpreter and pass `packageOverrides` which contains
221221+the overrides for packages in the package set.
222222+223223+```nix
224224+with import <nixpkgs> {};
225225+226226+(let
227227+ python = let
228228+ packageOverrides = self: super: {
229229+ pandas = super.pandas.overridePythonAttrs(old: rec {
230230+ version = "0.19.1";
231231+ src = fetchPypi {
232232+ pname = "pandas";
233233+ inherit version;
234234+ hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE=";
235235+ };
236236+ });
237237+ };
238238+ in pkgs.python3.override {inherit packageOverrides; self = python;};
239239+240240+in python.withPackages(ps: [ ps.blaze ])).env
241241+```
242242+243243+The next example shows a non trivial overriding of the `blas` implementation to
244244+be used through out all of the Python package set:
245245+246246+```nix
247247+python3MyBlas = pkgs.python3.override {
248248+ packageOverrides = self: super: {
249249+ # We need toPythonModule for the package set to evaluate this
250250+ blas = super.toPythonModule(super.pkgs.blas.override {
251251+ blasProvider = super.pkgs.mkl;
252252+ });
253253+ lapack = super.toPythonModule(super.pkgs.lapack.override {
254254+ lapackProvider = super.pkgs.mkl;
255255+ });
256256+ };
257257+};
258258+```
259259+260260+This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations.
261261+Note that using simply `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in
262262+compilation issues, because scipy dependencies need to use the same blas implementation as well.
263263+264264+#### `buildPythonApplication` function {#buildpythonapplication-function}
265265+266266+The `buildPythonApplication` function is practically the same as
267267+`buildPythonPackage`. The main purpose of this function is to build a Python
268268+package where one is interested only in the executables, and not importable
269269+modules. For that reason, when adding this package to a `python.buildEnv`, the
270270+modules won't be made available.
271271+272272+Another difference is that `buildPythonPackage` by default prefixes the names of
273273+the packages with the version of the interpreter. Because this is irrelevant for
274274+applications, the prefix is omitted.
275275+276276+When packaging a Python application with `buildPythonApplication`, it should be
277277+called with `callPackage` and passed `python` or `pythonPackages` (possibly
278278+specifying an interpreter version), like this:
279279+280280+```nix
281281+{ lib
282282+, python3
283283+, fetchPypi
284284+}:
285285+286286+python3.pkgs.buildPythonApplication rec {
287287+ pname = "luigi";
288288+ version = "2.7.9";
289289+ format = "setuptools";
290290+291291+ src = fetchPypi {
292292+ inherit pname version;
293293+ hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw=";
294294+ };
295295+296296+ propagatedBuildInputs = with python3.pkgs; [
297297+ tornado
298298+ python-daemon
299299+ ];
300300+301301+ meta = with lib; {
302302+ ...
303303+ };
304304+}
305305+```
306306+307307+This is then added to `all-packages.nix` just as any other application would be.
308308+309309+```nix
310310+luigi = callPackage ../applications/networking/cluster/luigi { };
311311+```
312312+313313+Since the package is an application, a consumer doesn't need to care about
314314+Python versions or modules, which is why they don't go in `pythonPackages`.
315315+316316+#### `toPythonApplication` function {#topythonapplication-function}
317317+318318+A distinction is made between applications and libraries, however, sometimes a
319319+package is used as both. In this case the package is added as a library to
320320+`python-packages.nix` and as an application to `all-packages.nix`. To reduce
321321+duplication the `toPythonApplication` can be used to convert a library to an
322322+application.
323323+324324+The Nix expression shall use `buildPythonPackage` and be called from
325325+`python-packages.nix`. A reference shall be created from `all-packages.nix` to
326326+the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
327327+applied to the reference:
328328+329329+```nix
330330+youtube-dl = with pythonPackages; toPythonApplication youtube-dl;
331331+```
332332+333333+#### `toPythonModule` function {#topythonmodule-function}
334334+335335+In some cases, such as bindings, a package is created using
336336+`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python
337337+bindings should be made available from `python-packages.nix`. The
338338+`toPythonModule` function takes a derivation and makes certain Python-specific
339339+modifications.
340340+341341+```nix
342342+opencv = toPythonModule (pkgs.opencv.override {
343343+ enablePython = true;
344344+ pythonPackages = self;
345345+});
346346+```
347347+348348+Do pay attention to passing in the right Python version!
349349+350350+#### `python.buildEnv` function {#python.buildenv-function}
351351+352352+Python environments can be created using the low-level `pkgs.buildEnv` function.
353353+This example shows how to create an environment that has the Pyramid Web Framework.
354354+Saving the following as `default.nix`
355355+356356+```nix
357357+with import <nixpkgs> {};
358358+359359+python.buildEnv.override {
360360+ extraLibs = [ pythonPackages.pyramid ];
361361+ ignoreCollisions = true;
362362+}
363363+```
364364+365365+and running `nix-build` will create
366366+367367+```
368368+/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
369369+```
370370+371371+with wrapped binaries in `bin/`.
372372+373373+You can also use the `env` attribute to create local environments with needed
374374+packages installed. This is somewhat comparable to `virtualenv`. For example,
375375+running `nix-shell` with the following `shell.nix`
376376+377377+```nix
378378+with import <nixpkgs> {};
379379+380380+(python3.buildEnv.override {
381381+ extraLibs = with python3Packages; [
382382+ numpy
383383+ requests
384384+ ];
385385+}).env
386386+```
387387+388388+will drop you into a shell where Python will have the
389389+specified packages in its path.
390390+391391+##### `python.buildEnv` arguments {#python.buildenv-arguments}
392392+393393+394394+* `extraLibs`: List of packages installed inside the environment.
395395+* `postBuild`: Shell command executed after the build of environment.
396396+* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
397397+* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
398398+ wrapped binaries in the environment.
399399+400400+#### `python.withPackages` function {#python.withpackages-function}
401401+402402+The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
403403+It takes a function as an argument that is passed the set of python packages and returns the list
404404+of the packages to be included in the environment. Using the `withPackages` function, the previous
405405+example for the Pyramid Web Framework environment can be written like this:
406406+407407+```nix
408408+with import <nixpkgs> {};
409409+410410+python.withPackages (ps: [ ps.pyramid ])
411411+```
412412+413413+`withPackages` passes the correct package set for the specific interpreter
414414+version as an argument to the function. In the above example, `ps` equals
415415+`pythonPackages`. But you can also easily switch to using python3:
416416+417417+```nix
418418+with import <nixpkgs> {};
419419+420420+python3.withPackages (ps: [ ps.pyramid ])
421421+```
422422+423423+Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
424424+425425+As `python.withPackages` simply uses `python.buildEnv` under the hood, it also
426426+supports the `env` attribute. The `shell.nix` file from the previous section can
427427+thus be also written like this:
428428+429429+```nix
430430+with import <nixpkgs> {};
431431+432432+(python3.withPackages (ps: with ps; [
433433+ numpy
434434+ requests
435435+])).env
436436+```
437437+438438+In contrast to `python.buildEnv`, `python.withPackages` does not support the
439439+more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
440440+need them, you have to use `python.buildEnv`.
441441+442442+Python 2 namespace packages may provide `__init__.py` that collide. In that case
443443+`python.buildEnv` should be used with `ignoreCollisions = true`.
444444+445445+#### Setup hooks {#setup-hooks}
446446+447447+The following are setup hooks specifically for Python packages. Most of these
448448+are used in `buildPythonPackage`.
449449+450450+- `eggUnpackhook` to move an egg to the correct folder so it can be installed
451451+ with the `eggInstallHook`
452452+- `eggBuildHook` to skip building for eggs.
453453+- `eggInstallHook` to install eggs.
454454+- `flitBuildHook` to build a wheel using `flit`.
455455+- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
456456+ (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
457457+- `pypaBuildHook` to build a wheel using
458458+ [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and
459459+ PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still
460460+ be added as `nativeBuildInput`.
461461+- `pipInstallHook` to install wheels.
462462+- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
463463+- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
464464+- `pythonImportsCheckHook` to check whether importing the listed modules works.
465465+- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package.
466466+ See [example usage](#using-pythonrelaxdepshook).
467467+- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
468468+- `setuptoolsBuildHook` to build a wheel using `setuptools`.
469469+- `setuptoolsCheckHook` to run tests with `python setup.py test`.
470470+- `sphinxHook` to build documentation and manpages using Sphinx.
471471+- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
472472+ `venv` is created if it does not yet exist. `postVenvCreation` can be used to
473473+ to run commands only after venv is first created.
474474+- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
475475+ with the `pipInstallHook`.
476476+- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook).
477477+478478+### Development mode {#development-mode}
479479+480480+Development or editable mode is supported. To develop Python packages
481481+`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
482482+install -e . --prefix $TMPDIR/`for the package.
483483+484484+Warning: `shellPhase` is executed only if `setup.py` exists.
485485+486486+Given a `default.nix`:
487487+488488+```nix
489489+with import <nixpkgs> {};
490490+491491+pythonPackages.buildPythonPackage {
492492+ name = "myproject";
493493+ buildInputs = with pythonPackages; [ pyramid ];
494494+495495+ src = ./.;
496496+}
497497+```
498498+499499+Running `nix-shell` with no arguments should give you the environment in which
500500+the package would be built with `nix-build`.
501501+502502+Shortcut to setup environments with C headers/libraries and Python packages:
503503+504504+```shell
505505+nix-shell -p pythonPackages.pyramid zlib libjpeg git
506506+```
507507+508508+Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
509509+3510## User Guide {#user-guide}
45115512### Using Python {#using-python}
···9931500example we use `buildPythonPackage` that is part of the set `python3Packages`,
9941501and in this case the `python3` interpreter is automatically used.
9951502996996-## Reference {#reference}
997997-998998-### Interpreters {#interpreters}
999999-10001000-| Package | Aliases | Interpreter |
10011001-|------------|-----------------|-------------|
10021002-| python27 | python2, python | CPython 2.7 |
10031003-| python38 | | CPython 3.8 |
10041004-| python39 | | CPython 3.9 |
10051005-| python310 | python3 | CPython 3.10 |
10061006-| python311 | | CPython 3.11 |
10071007-| python312 | | CPython 3.12 |
10081008-| pypy27 | pypy2, pypy | PyPy2.7 |
10091009-| pypy39 | pypy3 | PyPy 3.9 |
10101010-10111011-The Nix expressions for the interpreters can be found in
10121012-`pkgs/development/interpreters/python`.
10131013-10141014-All packages depending on any Python interpreter get appended
10151015-`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
10161016-exists.
10171017-10181018-#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
10191019-10201020-To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
10211021-10221022-#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
10231023-10241024-Each interpreter has the following attributes:
10251025-10261026-- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
10271027-- `interpreter`. Alias for `${python}/bin/${executable}`.
10281028-- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
10291029-- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
10301030-- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
10311031-- `executable`. Name of the interpreter executable, e.g. `python3.10`.
10321032-- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
10331033-10341034-### Optimizations {#optimizations}
10351035-10361036-The Python interpreters are by default not built with optimizations enabled, because
10371037-the builds are in that case not reproducible. To enable optimizations, override the
10381038-interpreter of interest, e.g using
10391039-10401040-```
10411041-let
10421042- pkgs = import ./. {};
10431043- mypython = pkgs.python3.override {
10441044- enableOptimizations = true;
10451045- reproducibleBuild = false;
10461046- self = mypython;
10471047- };
10481048-in mypython
10491049-```
10501050-10511051-### Building packages and applications {#building-packages-and-applications}
10521052-10531053-Python libraries and applications that use `setuptools` or
10541054-`distutils` are typically built with respectively the `buildPythonPackage` and
10551055-`buildPythonApplication` functions. These two functions also support installing a `wheel`.
10561056-10571057-All Python packages reside in `pkgs/top-level/python-packages.nix` and all
10581058-applications elsewhere. In case a package is used as both a library and an
10591059-application, then the package should be in `pkgs/top-level/python-packages.nix`
10601060-since only those packages are made available for all interpreter versions. The
10611061-preferred location for library expressions is in
10621062-`pkgs/development/python-modules`. It is important that these packages are
10631063-called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
10641064-the right version of the package is built.
10651065-10661066-Based on the packages defined in `pkgs/top-level/python-packages.nix` an
10671067-attribute set is created for each available Python interpreter. The available
10681068-sets are
10691069-10701070-* `pkgs.python27Packages`
10711071-* `pkgs.python3Packages`
10721072-* `pkgs.python38Packages`
10731073-* `pkgs.python39Packages`
10741074-* `pkgs.python310Packages`
10751075-* `pkgs.python311Packages`
10761076-* `pkgs.pypyPackages`
10771077-10781078-and the aliases
10791079-10801080-* `pkgs.python2Packages` pointing to `pkgs.python27Packages`
10811081-* `pkgs.python3Packages` pointing to `pkgs.python310Packages`
10821082-* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
10831083-10841084-#### `buildPythonPackage` function {#buildpythonpackage-function}
10851085-10861086-The `buildPythonPackage` function is implemented in
10871087-`pkgs/development/interpreters/python/mk-python-derivation.nix`
10881088-using setup hooks.
10891089-10901090-The following is an example:
10911091-10921092-```nix
10931093-{ lib
10941094-, buildPythonPackage
10951095-, fetchPypi
10961096-10971097-# build-system
10981098-, setuptools-scm
10991099-11001100-# dependencies
11011101-, attrs
11021102-, pluggy
11031103-, py
11041104-, setuptools
11051105-, six
11061106-11071107-# tests
11081108-, hypothesis
11091109- }:
11101110-11111111-buildPythonPackage rec {
11121112- pname = "pytest";
11131113- version = "3.3.1";
11141114- format = "setuptools";
11151115-11161116- src = fetchPypi {
11171117- inherit pname version;
11181118- hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M=";
11191119- };
11201120-11211121- postPatch = ''
11221122- # don't test bash builtins
11231123- rm testing/test_argcomplete.py
11241124- '';
11251125-11261126- nativeBuildInputs = [
11271127- setuptools-scm
11281128- ];
11291129-11301130- propagatedBuildInputs = [
11311131- attrs
11321132- py
11331133- setuptools
11341134- six
11351135- pluggy
11361136- ];
11371137-11381138- nativeCheckInputs = [
11391139- hypothesis
11401140- ];
11411141-11421142- meta = with lib; {
11431143- changelog = "https://github.com/pytest-dev/pytest/releases/tag/${version}";
11441144- description = "Framework for writing tests";
11451145- homepage = "https://github.com/pytest-dev/pytest";
11461146- license = licenses.mit;
11471147- maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ];
11481148- };
11491149-}
11501150-```
11511151-11521152-The `buildPythonPackage` mainly does four things:
11531153-11541154-* In the `buildPhase`, it calls `${python.pythonForBuild.interpreter} setup.py bdist_wheel` to
11551155- build a wheel binary zipfile.
11561156-* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
11571157-* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
11581158- wrap all programs in the `$out/bin/*` directory to include `$PATH`
11591159- environment variable and add dependent libraries to script's `sys.path`.
11601160-* In the `installCheck` phase, `${python.interpreter} setup.py test` is run.
11611161-11621162-By default tests are run because `doCheck = true`. Test dependencies, like
11631163-e.g. the test runner, should be added to `nativeCheckInputs`.
11641164-11651165-By default `meta.platforms` is set to the same value
11661166-as the interpreter unless overridden otherwise.
11671167-11681168-##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
11691169-11701170-All parameters from `stdenv.mkDerivation` function are still supported. The
11711171-following are specific to `buildPythonPackage`:
11721172-11731173-* `catchConflicts ? true`: If `true`, abort package build if a package name
11741174- appears more than once in dependency tree. Default is `true`.
11751175-* `disabled ? false`: If `true`, package is not built for the particular Python
11761176- interpreter version.
11771177-* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
11781178-* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
11791179- variable in wrapped programs.
11801180-* `format ? "setuptools"`: Format of the source. Valid options are
11811181- `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`.
11821182- `"setuptools"` is for when the source has a `setup.py` and `setuptools` is
11831183- used to build a wheel, `flit`, in case `flit` should be used to build a wheel,
11841184- and `wheel` in case a wheel is provided. Use `other` when a custom
11851185- `buildPhase` and/or `installPhase` is needed.
11861186-* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
11871187- `makeWrapper`, which wraps generated binaries. By default, the arguments to
11881188- `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling
11891189- the binary. Additional arguments here can allow a developer to set environment
11901190- variables which will be available when the binary is run. For example,
11911191- `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
11921192-* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
11931193- defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`.
11941194-* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip
11951195- install`. To pass options to `python setup.py install`, use
11961196- `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`.
11971197-* `pipBuildFlags ? []`: A list of strings. Arguments to be passed to `pip wheel`.
11981198-* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
11991199-* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
12001200- in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
12011201-* `preShellHook`: Hook to execute commands before `shellHook`.
12021202-* `postShellHook`: Hook to execute commands after `shellHook`.
12031203-* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
12041204- created when the filenames end with `.py`.
12051205-* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
12061206-* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
12071207-12081208-The `stdenv.mkDerivation` function accepts various parameters for describing
12091209-build inputs (see "Specifying dependencies"). The following are of special
12101210-interest for Python packages, either because these are primarily used, or
12111211-because their behaviour is different:
12121212-12131213-* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables
12141214- as well as the items listed in `setup_requires`.
12151215-* `buildInputs ? []`: Build and/or run-time dependencies that need to be
12161216- compiled for the host machine. Typically non-Python libraries which are being
12171217- linked.
12181218-* `nativeCheckInputs ? []`: Dependencies needed for running the `checkPhase`. These
12191219- are added to `nativeBuildInputs` when `doCheck = true`. Items listed in
12201220- `tests_require` go here.
12211221-* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
12221222- `buildPythonPackage` also injects code into and wraps executables with the
12231223- paths included in this list. Items listed in `install_requires` go here.
12241224-12251225-##### Overriding Python packages {#overriding-python-packages}
12261226-12271227-The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
12281228-used to override the package. In the following example we create an environment
12291229-where we have the `blaze` package using an older version of `pandas`. We
12301230-override first the Python interpreter and pass `packageOverrides` which contains
12311231-the overrides for packages in the package set.
12321232-12331233-```nix
12341234-with import <nixpkgs> {};
12351235-12361236-(let
12371237- python = let
12381238- packageOverrides = self: super: {
12391239- pandas = super.pandas.overridePythonAttrs(old: rec {
12401240- version = "0.19.1";
12411241- src = fetchPypi {
12421242- pname = "pandas";
12431243- inherit version;
12441244- hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE=";
12451245- };
12461246- });
12471247- };
12481248- in pkgs.python3.override {inherit packageOverrides; self = python;};
12491249-12501250-in python.withPackages(ps: [ ps.blaze ])).env
12511251-```
12521252-12531253-The next example shows a non trivial overriding of the `blas` implementation to
12541254-be used through out all of the Python package set:
12551255-12561256-```nix
12571257-python3MyBlas = pkgs.python3.override {
12581258- packageOverrides = self: super: {
12591259- # We need toPythonModule for the package set to evaluate this
12601260- blas = super.toPythonModule(super.pkgs.blas.override {
12611261- blasProvider = super.pkgs.mkl;
12621262- });
12631263- lapack = super.toPythonModule(super.pkgs.lapack.override {
12641264- lapackProvider = super.pkgs.mkl;
12651265- });
12661266- };
12671267-};
12681268-```
12691269-12701270-This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations.
12711271-Note that using simply `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in
12721272-compilation issues, because scipy dependencies need to use the same blas implementation as well.
12731273-12741274-#### Optional extra dependencies {#python-optional-dependencies}
12751275-12761276-Some packages define optional dependencies for additional features. With
12771277-`setuptools` this is called `extras_require` and `flit` calls it
12781278-`extras-require`, while PEP 621 calls these `optional-dependencies`. A
12791279-method for supporting this is by declaring the extras of a package in its
12801280-`passthru`, e.g. in case of the package `dask`
12811281-12821282-```nix
12831283-passthru.optional-dependencies = {
12841284- complete = [ distributed ];
12851285-};
12861286-```
12871287-12881288-and letting the package requiring the extra add the list to its dependencies
12891289-12901290-```nix
12911291-propagatedBuildInputs = [
12921292- ...
12931293-] ++ dask.optional-dependencies.complete;
12941294-```
12951295-12961296-Note this method is preferred over adding parameters to builders, as that can
12971297-result in packages depending on different variants and thereby causing
12981298-collisions.
12991299-13001300-#### `buildPythonApplication` function {#buildpythonapplication-function}
13011301-13021302-The `buildPythonApplication` function is practically the same as
13031303-`buildPythonPackage`. The main purpose of this function is to build a Python
13041304-package where one is interested only in the executables, and not importable
13051305-modules. For that reason, when adding this package to a `python.buildEnv`, the
13061306-modules won't be made available.
13071307-13081308-Another difference is that `buildPythonPackage` by default prefixes the names of
13091309-the packages with the version of the interpreter. Because this is irrelevant for
13101310-applications, the prefix is omitted.
13111311-13121312-When packaging a Python application with `buildPythonApplication`, it should be
13131313-called with `callPackage` and passed `python` or `pythonPackages` (possibly
13141314-specifying an interpreter version), like this:
13151315-13161316-```nix
13171317-{ lib
13181318-, python3
13191319-, fetchPypi
13201320-}:
13211321-13221322-python3.pkgs.buildPythonApplication rec {
13231323- pname = "luigi";
13241324- version = "2.7.9";
13251325- format = "setuptools";
13261326-13271327- src = fetchPypi {
13281328- inherit pname version;
13291329- hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw=";
13301330- };
13311331-13321332- propagatedBuildInputs = with python3.pkgs; [
13331333- tornado
13341334- python-daemon
13351335- ];
13361336-13371337- meta = with lib; {
13381338- ...
13391339- };
13401340-}
13411341-```
13421342-13431343-This is then added to `all-packages.nix` just as any other application would be.
13441344-13451345-```nix
13461346-luigi = callPackage ../applications/networking/cluster/luigi { };
13471347-```
13481348-13491349-Since the package is an application, a consumer doesn't need to care about
13501350-Python versions or modules, which is why they don't go in `pythonPackages`.
13511351-13521352-#### `toPythonApplication` function {#topythonapplication-function}
13531353-13541354-A distinction is made between applications and libraries, however, sometimes a
13551355-package is used as both. In this case the package is added as a library to
13561356-`python-packages.nix` and as an application to `all-packages.nix`. To reduce
13571357-duplication the `toPythonApplication` can be used to convert a library to an
13581358-application.
13591359-13601360-The Nix expression shall use `buildPythonPackage` and be called from
13611361-`python-packages.nix`. A reference shall be created from `all-packages.nix` to
13621362-the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
13631363-applied to the reference:
13641364-13651365-```nix
13661366-youtube-dl = with pythonPackages; toPythonApplication youtube-dl;
13671367-```
13681368-13691369-#### `toPythonModule` function {#topythonmodule-function}
13701370-13711371-In some cases, such as bindings, a package is created using
13721372-`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python
13731373-bindings should be made available from `python-packages.nix`. The
13741374-`toPythonModule` function takes a derivation and makes certain Python-specific
13751375-modifications.
13761376-13771377-```nix
13781378-opencv = toPythonModule (pkgs.opencv.override {
13791379- enablePython = true;
13801380- pythonPackages = self;
13811381-});
13821382-```
13831383-13841384-Do pay attention to passing in the right Python version!
13851385-13861386-#### `python.buildEnv` function {#python.buildenv-function}
13871387-13881388-Python environments can be created using the low-level `pkgs.buildEnv` function.
13891389-This example shows how to create an environment that has the Pyramid Web Framework.
13901390-Saving the following as `default.nix`
13911391-13921392-```nix
13931393-with import <nixpkgs> {};
13941394-13951395-python.buildEnv.override {
13961396- extraLibs = [ pythonPackages.pyramid ];
13971397- ignoreCollisions = true;
13981398-}
13991399-```
14001400-14011401-and running `nix-build` will create
14021402-14031403-```
14041404-/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env
14051405-```
14061406-14071407-with wrapped binaries in `bin/`.
14081408-14091409-You can also use the `env` attribute to create local environments with needed
14101410-packages installed. This is somewhat comparable to `virtualenv`. For example,
14111411-running `nix-shell` with the following `shell.nix`
14121412-14131413-```nix
14141414-with import <nixpkgs> {};
14151415-14161416-(python3.buildEnv.override {
14171417- extraLibs = with python3Packages; [
14181418- numpy
14191419- requests
14201420- ];
14211421-}).env
14221422-```
14231423-14241424-will drop you into a shell where Python will have the
14251425-specified packages in its path.
14261426-14271427-##### `python.buildEnv` arguments {#python.buildenv-arguments}
14281428-14291429-14301430-* `extraLibs`: List of packages installed inside the environment.
14311431-* `postBuild`: Shell command executed after the build of environment.
14321432-* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`).
14331433-* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in
14341434- wrapped binaries in the environment.
14351435-14361436-#### `python.withPackages` function {#python.withpackages-function}
14371437-14381438-The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
14391439-It takes a function as an argument that is passed the set of python packages and returns the list
14401440-of the packages to be included in the environment. Using the `withPackages` function, the previous
14411441-example for the Pyramid Web Framework environment can be written like this:
14421442-14431443-```nix
14441444-with import <nixpkgs> {};
14451445-14461446-python.withPackages (ps: [ ps.pyramid ])
14471447-```
14481448-14491449-`withPackages` passes the correct package set for the specific interpreter
14501450-version as an argument to the function. In the above example, `ps` equals
14511451-`pythonPackages`. But you can also easily switch to using python3:
14521452-14531453-```nix
14541454-with import <nixpkgs> {};
14551455-14561456-python3.withPackages (ps: [ ps.pyramid ])
14571457-```
14581458-14591459-Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
14601460-14611461-As `python.withPackages` simply uses `python.buildEnv` under the hood, it also
14621462-supports the `env` attribute. The `shell.nix` file from the previous section can
14631463-thus be also written like this:
14641464-14651465-```nix
14661466-with import <nixpkgs> {};
14671467-14681468-(python3.withPackages (ps: with ps; [
14691469- numpy
14701470- requests
14711471-])).env
14721472-```
14731473-14741474-In contrast to `python.buildEnv`, `python.withPackages` does not support the
14751475-more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
14761476-need them, you have to use `python.buildEnv`.
14771477-14781478-Python 2 namespace packages may provide `__init__.py` that collide. In that case
14791479-`python.buildEnv` should be used with `ignoreCollisions = true`.
14801480-14811481-#### Setup hooks {#setup-hooks}
14821482-14831483-The following are setup hooks specifically for Python packages. Most of these
14841484-are used in `buildPythonPackage`.
14851485-14861486-- `eggUnpackhook` to move an egg to the correct folder so it can be installed
14871487- with the `eggInstallHook`
14881488-- `eggBuildHook` to skip building for eggs.
14891489-- `eggInstallHook` to install eggs.
14901490-- `flitBuildHook` to build a wheel using `flit`.
14911491-- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
14921492- (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
14931493-- `pypaBuildHook` to build a wheel using
14941494- [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and
14951495- PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still
14961496- be added as `nativeBuildInput`.
14971497-- `pipInstallHook` to install wheels.
14981498-- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
14991499-- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
15001500-- `pythonImportsCheckHook` to check whether importing the listed modules works.
15011501-- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package.
15021502- See [example usage](#using-pythonrelaxdepshook).
15031503-- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
15041504-- `setuptoolsBuildHook` to build a wheel using `setuptools`.
15051505-- `setuptoolsCheckHook` to run tests with `python setup.py test`.
15061506-- `sphinxHook` to build documentation and manpages using Sphinx.
15071507-- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
15081508- `venv` is created if it does not yet exist. `postVenvCreation` can be used to
15091509- to run commands only after venv is first created.
15101510-- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
15111511- with the `pipInstallHook`.
15121512-- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook).
15131513-15141514-### Development mode {#development-mode}
15151515-15161516-Development or editable mode is supported. To develop Python packages
15171517-`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
15181518-install -e . --prefix $TMPDIR/`for the package.
15191519-15201520-Warning: `shellPhase` is executed only if `setup.py` exists.
15211521-15221522-Given a `default.nix`:
15231523-15241524-```nix
15251525-with import <nixpkgs> {};
15261526-15271527-pythonPackages.buildPythonPackage {
15281528- name = "myproject";
15291529- buildInputs = with pythonPackages; [ pyramid ];
15301530-15311531- src = ./.;
15321532-}
15331533-```
15341534-15351535-Running `nix-shell` with no arguments should give you the environment in which
15361536-the package would be built with `nix-build`.
15371537-15381538-Shortcut to setup environments with C headers/libraries and Python packages:
15391539-15401540-```shell
15411541-nix-shell -p pythonPackages.pyramid zlib libjpeg git
15421542-```
15431543-15441544-Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked.
15451545-15461546-### Tools {#tools}
15471547-15481548-Packages inside nixpkgs must use the `buildPythonPackage` or `buildPythonApplication` function directly,
15491549-because we can only provide security support for non-vendored dependencies.
15501550-15511551-We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs,
15521552-as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`.
15531553-15541554-### Deterministic builds {#deterministic-builds}
15551555-15561556-The Python interpreters are now built deterministically. Minor modifications had
15571557-to be made to the interpreters in order to generate deterministic bytecode. This
15581558-has security implications and is relevant for those using Python in a
15591559-`nix-shell`.
15601560-15611561-When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
15621562-have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1`
15631563-and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED).
15641564-Both are also exported in `nix-shell`.
15651565-15661566-### Automatic tests {#automatic-tests}
15671567-15681568-It is recommended to test packages as part of the build process.
15691569-Source distributions (`sdist`) often include test files, but not always.
15701570-15711571-By default the command `python setup.py test` is run as part of the
15721572-`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An
15731573-example of such a situation is when `py.test` is used.
15741574-15751575-#### Common issues {#common-issues}
15761576-15771577-* Non-working tests can often be deselected. By default `buildPythonPackage`
15781578- runs `python setup.py test`. which is deprecated. Most Python modules however
15791579- do follow the standard test protocol where the pytest runner can be used
15801580- instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test
15811581- methods or classes as well as whole files. For `pytestCheckHook` these are
15821582- conveniently exposed as `disabledTests` and `disabledTestPaths` respectively.
15831583-15841584- ```nix
15851585- buildPythonPackage {
15861586- # ...
15871587- nativeCheckInputs = [
15881588- pytestCheckHook
15891589- ];
15901590-15911591- disabledTests = [
15921592- "function_name"
15931593- "other_function"
15941594- ];
15951595-15961596- disabledTestPaths = [
15971597- "this/file.py"
15981598- ];
15991599- }
16001600- ```
16011601-16021602-* Tests that attempt to access `$HOME` can be fixed by using the following
16031603- work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
16041604-16051503## FAQ {#faq}
1606150416071505### How to solve circular dependencies? {#how-to-solve-circular-dependencies}
···19491847* `setup_requires` corresponds to `nativeBuildInputs`
19501848* `install_requires` corresponds to `propagatedBuildInputs`
19511849* `tests_require` corresponds to `nativeCheckInputs`
18501850+18511851+### How to enable interpreter optimizations? {#optimizations}
18521852+18531853+The Python interpreters are by default not built with optimizations enabled, because
18541854+the builds are in that case not reproducible. To enable optimizations, override the
18551855+interpreter of interest, e.g using
18561856+18571857+```
18581858+let
18591859+ pkgs = import ./. {};
18601860+ mypython = pkgs.python3.override {
18611861+ enableOptimizations = true;
18621862+ reproducibleBuild = false;
18631863+ self = mypython;
18641864+ };
18651865+in mypython
18661866+```
18671867+18681868+### How to add optional dependencies? {#python-optional-dependencies}
18691869+18701870+Some packages define optional dependencies for additional features. With
18711871+`setuptools` this is called `extras_require` and `flit` calls it
18721872+`extras-require`, while PEP 621 calls these `optional-dependencies`. A
18731873+method for supporting this is by declaring the extras of a package in its
18741874+`passthru`, e.g. in case of the package `dask`
18751875+18761876+```nix
18771877+passthru.optional-dependencies = {
18781878+ complete = [ distributed ];
18791879+};
18801880+```
18811881+18821882+and letting the package requiring the extra add the list to its dependencies
18831883+18841884+```nix
18851885+propagatedBuildInputs = [
18861886+ ...
18871887+] ++ dask.optional-dependencies.complete;
18881888+```
18891889+18901890+Note this method is preferred over adding parameters to builders, as that can
18911891+result in packages depending on different variants and thereby causing
18921892+collisions.
18931893+18941894+### How to contribute a Python package to nixpkgs? {#tools}
18951895+18961896+Packages inside nixpkgs must use the `buildPythonPackage` or `buildPythonApplication` function directly,
18971897+because we can only provide security support for non-vendored dependencies.
18981898+18991899+We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs,
19001900+as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`.
19011901+19021902+### Are Python interpreters built deterministically? {#deterministic-builds}
19031903+19041904+The Python interpreters are now built deterministically. Minor modifications had
19051905+to be made to the interpreters in order to generate deterministic bytecode. This
19061906+has security implications and is relevant for those using Python in a
19071907+`nix-shell`.
19081908+19091909+When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
19101910+have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1`
19111911+and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED).
19121912+Both are also exported in `nix-shell`.
19131913+19141914+### How to provide automatic tests to Python packages? {#automatic-tests}
19151915+19161916+It is recommended to test packages as part of the build process.
19171917+Source distributions (`sdist`) often include test files, but not always.
19181918+19191919+By default the command `python setup.py test` is run as part of the
19201920+`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An
19211921+example of such a situation is when `py.test` is used.
19221922+19231923+#### Common issues {#common-issues}
19241924+19251925+* Non-working tests can often be deselected. By default `buildPythonPackage`
19261926+ runs `python setup.py test`. which is deprecated. Most Python modules however
19271927+ do follow the standard test protocol where the pytest runner can be used
19281928+ instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test
19291929+ methods or classes as well as whole files. For `pytestCheckHook` these are
19301930+ conveniently exposed as `disabledTests` and `disabledTestPaths` respectively.
19311931+19321932+ ```nix
19331933+ buildPythonPackage {
19341934+ # ...
19351935+ nativeCheckInputs = [
19361936+ pytestCheckHook
19371937+ ];
19381938+19391939+ disabledTests = [
19401940+ "function_name"
19411941+ "other_function"
19421942+ ];
19431943+19441944+ disabledTestPaths = [
19451945+ "this/file.py"
19461946+ ];
19471947+ }
19481948+ ```
19491949+19501950+* Tests that attempt to access `$HOME` can be fixed by using the following
19511951+ work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
1952195219531953## Contributing {#contributing}
19541954
···361361362362 # Optional features:
363363 use_gio = true;
364364- use_gnome_keyring = false; # Superseded by libsecret
365364 use_cups = cupsSupport;
366365367366 # Feature overrides:
···384383 # We do intentionally not set rustc_version as nixpkgs will never do incremental
385384 # rebuilds, thus leaving this empty is fine.
386385 rust_sysroot_absolute = "${rustc}";
386386+ # Building with rust is disabled for now - this matches the flags in other major distributions.
387387+ enable_rust = false;
387388 } // lib.optionalAttrs (!(stdenv.buildPlatform.canExecute stdenv.hostPlatform)) {
388389 # https://www.mail-archive.com/v8-users@googlegroups.com/msg14528.html
389390 arm_control_flow_integrity = "none";