···3233- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
34- `interpreter`. Alias for `${python}/bin/${executable}`.
35-- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
36-- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
37- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
38- `executable`. Name of the interpreter executable, e.g. `python3.10`.
39- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
···41### Building packages and applications {#building-packages-and-applications}
4243Python libraries and applications that use `setuptools` or
44-`distutils` are typically built with respectively the `buildPythonPackage` and
45-`buildPythonApplication` functions. These two functions also support installing a `wheel`.
4647All Python packages reside in `pkgs/top-level/python-packages.nix` and all
48applications elsewhere. In case a package is used as both a library and an
···141142The `buildPythonPackage` mainly does four things:
143144-* In the `buildPhase`, it calls `${python.pythonForBuild.interpreter} setup.py bdist_wheel` to
145 build a wheel binary zipfile.
146-* In the `installPhase`, it installs the wheel file using `pip install *.whl`.
147-* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to
148 wrap all programs in the `$out/bin/*` directory to include `$PATH`
149 environment variable and add dependent libraries to script's `sys.path`.
150-* In the `installCheck` phase, `${python.interpreter} setup.py test` is run.
151152-By default tests are run because `doCheck = true`. Test dependencies, like
153-e.g. the test runner, should be added to `nativeCheckInputs`.
154155By default `meta.platforms` is set to the same value
156as the interpreter unless overridden otherwise.
157158##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
159160-All parameters from `stdenv.mkDerivation` function are still supported. The
161following are specific to `buildPythonPackage`:
162163* `catchConflicts ? true`: If `true`, abort package build if a package name
···177 format. When unset, the legacy `setuptools` hooks are used for backwards
178 compatibility.
179* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
180- `makeWrapper`, which wraps generated binaries. By default, the arguments to
181- `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling
182 the binary. Additional arguments here can allow a developer to set environment
183 variables which will be available when the binary is run. For example,
184 `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
···190* `pipBuildFlags ? []`: A list of strings. Arguments to be passed to `pip wheel`.
191* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
192* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
193- in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
194* `preShellHook`: Hook to execute commands before `shellHook`.
195* `postShellHook`: Hook to execute commands after `shellHook`.
196* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
···198* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
199* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
200201-The `stdenv.mkDerivation` function accepts various parameters for describing
202build inputs (see "Specifying dependencies"). The following are of special
203interest for Python packages, either because these are primarily used, or
204because their behaviour is different:
···208* `buildInputs ? []`: Build and/or run-time dependencies that need to be
209 compiled for the host machine. Typically non-Python libraries which are being
210 linked.
211-* `nativeCheckInputs ? []`: Dependencies needed for running the `checkPhase`. These
212- are added to `nativeBuildInputs` when `doCheck = true`. Items listed in
213 `tests_require` go here.
214* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
215 `buildPythonPackage` also injects code into and wraps executables with the
···266267#### `buildPythonApplication` function {#buildpythonapplication-function}
268269-The `buildPythonApplication` function is practically the same as
270-`buildPythonPackage`. The main purpose of this function is to build a Python
271package where one is interested only in the executables, and not importable
272-modules. For that reason, when adding this package to a `python.buildEnv`, the
273modules won't be made available.
274275-Another difference is that `buildPythonPackage` by default prefixes the names of
276the packages with the version of the interpreter. Because this is irrelevant for
277applications, the prefix is omitted.
278279-When packaging a Python application with `buildPythonApplication`, it should be
280called with `callPackage` and passed `python` or `pythonPackages` (possibly
281specifying an interpreter version), like this:
282···329duplication the `toPythonApplication` can be used to convert a library to an
330application.
331332-The Nix expression shall use `buildPythonPackage` and be called from
333`python-packages.nix`. A reference shall be created from `all-packages.nix` to
334the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
335applied to the reference:
···341#### `toPythonModule` function {#topythonmodule-function}
342343In some cases, such as bindings, a package is created using
344-`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python
345bindings should be made available from `python-packages.nix`. The
346`toPythonModule` function takes a derivation and makes certain Python-specific
347modifications.
···407408#### `python.withPackages` function {#python.withpackages-function}
409410-The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality.
411It takes a function as an argument that is passed the set of python packages and returns the list
412-of the packages to be included in the environment. Using the `withPackages` function, the previous
413example for the Pyramid Web Framework environment can be written like this:
414415```nix
···418python.withPackages (ps: [ ps.pyramid ])
419```
420421-`withPackages` passes the correct package set for the specific interpreter
422version as an argument to the function. In the above example, `ps` equals
423`pythonPackages`. But you can also easily switch to using python3:
424···430431Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
432433-As `python.withPackages` simply uses `python.buildEnv` under the hood, it also
434supports the `env` attribute. The `shell.nix` file from the previous section can
435thus be also written like this:
436···443])).env
444```
445446-In contrast to `python.buildEnv`, `python.withPackages` does not support the
447more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
448-need them, you have to use `python.buildEnv`.
449450Python 2 namespace packages may provide `__init__.py` that collide. In that case
451-`python.buildEnv` should be used with `ignoreCollisions = true`.
452453#### Setup hooks {#setup-hooks}
454455The following are setup hooks specifically for Python packages. Most of these
456-are used in `buildPythonPackage`.
457458- `eggUnpackhook` to move an egg to the correct folder so it can be installed
459 with the `eggInstallHook`
···486### Development mode {#development-mode}
487488Development or editable mode is supported. To develop Python packages
489-`buildPythonPackage` has additional logic inside `shellPhase` to run `pip
490install -e . --prefix $TMPDIR/`for the package.
491492Warning: `shellPhase` is executed only if `setup.py` exists.
···567But Python libraries you would like to use for development cannot be installed,
568at least not individually, because they won't be able to find each other
569resulting in import errors. Instead, it is possible to create an environment
570-with `python.buildEnv` or `python.withPackages` where the interpreter and other
571executables are wrapped to be able to find each other and all of the modules.
572573In the following examples we will start by creating a simple, ad-hoc environment
···747 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
748 brings all attributes of `nixpkgs` in the local scope. These attributes form
749 the main package set.
750-2. Then we create a Python 3.11 environment with the `withPackages` function, as before.
751-3. The `withPackages` function expects us to provide a function as an argument
752 that takes the set of all Python packages and returns a list of packages to
753 include in the environment. Here, we select the packages `numpy` and `toolz`
754 from the package set.
···859#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
860861With Nix all packages are built by functions. The main function in Nix for
862-building Python libraries is `buildPythonPackage`. Let's see how we can build the
863`toolz` package.
864865```nix
···904}
905```
906907-What happens here? The function `buildPythonPackage` is called and as argument
908it accepts a set. In this case the set is a recursive set, `rec`. One of the
909arguments is the name of the package, which consists of a basename (generally
910following the name on PyPi) and a version. Another argument, `src` specifies the
911source, which in this case is fetched from PyPI using the helper function
912`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
913-when building the package. Since there are no tests, we rely on `pythonImportsCheck`
914to test whether the package can be imported. Furthermore, we specify some meta
915information. The output of the function is a derivation.
916···969So, what did we do here? Well, we took the Nix expression that we used earlier
970to build a Python environment, and said that we wanted to include our own
971version of `toolz`, named `my_toolz`. To introduce our own package in the scope
972-of `withPackages` we used a `let` expression. You can see that we used
973`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
974`toolz` from the Nixpkgs package set this time, but instead took our own version
975that we introduced with the `let` expression.
···977#### Handling dependencies {#handling-dependencies}
978979Our example, `toolz`, does not have any dependencies on other Python packages or
980-system libraries. According to the manual, `buildPythonPackage` uses the
981-arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If
982something is exclusively a build-time dependency, then the dependency should be
983-included in `buildInputs`, but if it is (also) a runtime dependency, then it
984-should be added to `propagatedBuildInputs`. Test dependencies are considered
985-build-time dependencies and passed to `nativeCheckInputs`.
986987-The following example shows which arguments are given to `buildPythonPackage` in
988order to build [`datashape`](https://github.com/blaze/datashape).
989990```nix
···1038```
10391040We can see several runtime dependencies, `numpy`, `multipledispatch`, and
1041-`python-dateutil`. Furthermore, we have `nativeCheckInputs` with `pytest`.
1042-`pytest` is a test runner and is only used during the `checkPhase` and is
1043-therefore not added to `propagatedBuildInputs`.
10441045In the previous case we had only dependencies on other Python packages to consider.
1046Occasionally you have also system libraries to consider. E.g., `lxml` provides
1047Python bindings to `libxml2` and `libxslt`. These libraries are only required
1048-when building the bindings and are therefore added as `buildInputs`.
10491050```nix
1051{ lib
···1093The example below shows bindings to The Fastest Fourier Transform in the West,
1094commonly known as FFTW. On Nix we have separate packages of FFTW for the
1095different types of floats (`"single"`, `"double"`, `"long-double"`). The
1096-bindings need all three types, and therefore we add all three as `buildInputs`.
1097The bindings don't expect to find each of them in a different folder, and
1098therefore we have to set `LDFLAGS` and `CFLAGS`.
1099···1158}
1159```
11601161-Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
11621163#### Testing Python Packages {#testing-python-packages}
1164···1167but is not usable at runtime. Currently, all packages will use the `test`
1168command provided by the setup.py (i.e. `python setup.py test`). However,
1169this is currently deprecated https://github.com/pypa/setuptools/pull/1878
1170-and your package should provide its own checkPhase.
11711172::: {.note}
1173-The `checkPhase` for python maps to the `installCheckPhase` on a
1174normal derivation. This is due to many python packages not behaving well
1175to the pre-installed version of the package. Version info, and natively
1176compiled extensions generally only exist in the install directory, and
···1235#### Using pytestCheckHook {#using-pytestcheckhook}
12361237`pytestCheckHook` is a convenient hook which will substitute the setuptools
1238-`test` command for a `checkPhase` which runs `pytest`. This is also beneficial
1239when a package may need many items disabled to run the test suite.
12401241Using the example above, the analogous `pytestCheckHook` usage would be:
···1280```
12811282Trying to concatenate the related strings to disable tests in a regular
1283-`checkPhase` would be much harder to read. This also enables us to comment on
1284why specific tests are disabled.
12851286#### Using pythonImportsCheck {#using-pythonimportscheck}
12871288Although unit tests are highly preferred to validate correctness of a package, not
1289all packages have test suites that can be run easily, and some have none at all.
1290-To help ensure the package still works, `pythonImportsCheck` can attempt to import
1291the listed modules.
12921293```
···1306 '';
1307```
13081309-However, this is done in its own phase, and not dependent on whether `doCheck = true;`.
13101311This can also be useful in verifying that the package doesn't assume commonly
1312present packages (e.g. `setuptools`).
···13781379Keep in mind that while the examples above are done with `requirements.txt`,
1380`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should
1381-work with any of the existing [hooks](#setup-hooks).
13821383#### Using unittestCheckHook {#using-unittestcheckhook}
13841385-`unittestCheckHook` is a hook which will substitute the setuptools `test` command for a `checkPhase` which runs `python -m unittest discover`:
13861387```
1388 nativeCheckInputs = [
···1452In the previous Nix expression the source was fetched from a url. We can also
1453refer to a local source instead using `src = ./path/to/source/tree;`
14541455-If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src`
1456is a local source, and if the local source has a `setup.py`, then development
1457mode is activated.
14581459In the following example, we create a simple environment that has a Python 3.11
1460version of our package in it, as well as its dependencies and other packages we
1461-like to have in the environment, all specified with `propagatedBuildInputs`.
1462Indeed, we can just add any package we like to have in our environment to
1463-`propagatedBuildInputs`.
14641465```nix
1466with import <nixpkgs> {};
···14941495### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
14961497-Earlier we created a Python environment using `withPackages`, and included the
1498`toolz` package via a `let` expression.
1499Let's split the package definition from the environment definition.
1500···1533}
1534```
15351536-It takes an argument `buildPythonPackage`. We now call this function using
1537`callPackage` in the definition of our environment
15381539```nix
···1552```
15531554Important to remember is that the Python version for which the package is made
1555-depends on the `python` derivation that is passed to `buildPythonPackage`. Nix
1556tries to automatically pass arguments when possible, which is why generally you
1557don't explicitly define which `python` derivation should be used. In the above
1558-example we use `buildPythonPackage` that is part of the set `python3Packages`,
1559and in this case the `python3` interpreter is automatically used.
15601561## FAQ {#faq}
···1698packages are available. There is therefore no need to maintain a global `site-packages`.
16991700If you want to create a Python environment for development, then the recommended
1701-method is to use `nix-shell`, either with or without the `python.buildEnv`
1702function.
17031704### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
···18751876### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions}
18771878-The following overlay overrides the call to `buildPythonPackage` for the
1879`foo` package for all interpreters by appending a Python extension to the
1880`pythonPackagesExtensions` list of extensions.
1881···19021903In a `setup.py` or `setup.cfg` it is common to declare dependencies:
19041905-* `setup_requires` corresponds to `nativeBuildInputs`
1906-* `install_requires` corresponds to `propagatedBuildInputs`
1907-* `tests_require` corresponds to `nativeCheckInputs`
19081909### How to enable interpreter optimizations? {#optimizations}
1910···19511952### How to contribute a Python package to nixpkgs? {#tools}
19531954-Packages inside nixpkgs must use the `buildPythonPackage` or `buildPythonApplication` function directly,
1955because we can only provide security support for non-vendored dependencies.
19561957We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs,
···1965`nix-shell`.
19661967When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
1968-have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1`
1969and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED).
1970Both are also exported in `nix-shell`.
1971···1975Source distributions (`sdist`) often include test files, but not always.
19761977By default the command `python setup.py test` is run as part of the
1978-`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An
1979example of such a situation is when `py.test` is used.
19801981#### Common issues {#common-issues}
19821983-* Non-working tests can often be deselected. By default `buildPythonPackage`
1984 runs `python setup.py test`. which is deprecated. Most Python modules however
1985 do follow the standard test protocol where the pytest runner can be used
1986 instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test
···2015The following rules are desired to be respected:
20162017* Python libraries are called from `python-packages.nix` and packaged with
2018- `buildPythonPackage`. The expression of a library should be in
2019 `pkgs/development/python-modules/<name>/default.nix`.
2020* Python applications live outside of `python-packages.nix` and are packaged
2021- with `buildPythonApplication`.
2022* Make sure libraries build for all Python interpreters.
2023* By default we enable tests. Make sure the tests are found and, in the case of
2024 libraries, are passing for all interpreters. If certain tests fail they can be
···3233- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter.
34- `interpreter`. Alias for `${python}/bin/${executable}`.
35+- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See [](#python.buildenv-function) for usage and documentation.
36+- `withPackages`. Simpler interface to `buildEnv`. See [](#python.withpackages-function) for usage and documentation.
37- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
38- `executable`. Name of the interpreter executable, e.g. `python3.10`.
39- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
···41### Building packages and applications {#building-packages-and-applications}
4243Python libraries and applications that use `setuptools` or
44+`distutils` are typically built with respectively the [`buildPythonPackage`](#buildpythonpackage-function) and
45+[`buildPythonApplication`](#buildpythonapplication-function) functions. These two functions also support installing a `wheel`.
4647All Python packages reside in `pkgs/top-level/python-packages.nix` and all
48applications elsewhere. In case a package is used as both a library and an
···141142The `buildPythonPackage` mainly does four things:
143144+* In the [`buildPhase`](#build-phase), it calls `${python.pythonForBuild.interpreter} setup.py bdist_wheel` to
145 build a wheel binary zipfile.
146+* In the [`installPhase`](#ssec-install-phase), it installs the wheel file using `pip install *.whl`.
147+* In the [`postFixup`](#var-stdenv-postFixup) phase, the `wrapPythonPrograms` bash function is called to
148 wrap all programs in the `$out/bin/*` directory to include `$PATH`
149 environment variable and add dependent libraries to script's `sys.path`.
150+* In the [`installCheck`](#ssec-installCheck-phase) phase, `${python.interpreter} setup.py test` is run.
151152+By default tests are run because [`doCheck = true`](#var-stdenv-doCheck). Test dependencies, like
153+e.g. the test runner, should be added to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs).
154155By default `meta.platforms` is set to the same value
156as the interpreter unless overridden otherwise.
157158##### `buildPythonPackage` parameters {#buildpythonpackage-parameters}
159160+All parameters from [`stdenv.mkDerivation`](#sec-using-stdenv) function are still supported. The
161following are specific to `buildPythonPackage`:
162163* `catchConflicts ? true`: If `true`, abort package build if a package name
···177 format. When unset, the legacy `setuptools` hooks are used for backwards
178 compatibility.
179* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to
180+ [`makeWrapper`](#fun-makeWrapper), which wraps generated binaries. By default, the arguments to
181+ [`makeWrapper`](#fun-makeWrapper) set `PATH` and `PYTHONPATH` environment variables before calling
182 the binary. Additional arguments here can allow a developer to set environment
183 variables which will be available when the binary is run. For example,
184 `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
···190* `pipBuildFlags ? []`: A list of strings. Arguments to be passed to `pip wheel`.
191* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`.
192* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
193+ in `pythonPath` are not propagated (contrary to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)).
194* `preShellHook`: Hook to execute commands before `shellHook`.
195* `postShellHook`: Hook to execute commands after `shellHook`.
196* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
···198* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
199* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
200201+The [`stdenv.mkDerivation`](#sec-using-stdenv) function accepts various parameters for describing
202build inputs (see "Specifying dependencies"). The following are of special
203interest for Python packages, either because these are primarily used, or
204because their behaviour is different:
···208* `buildInputs ? []`: Build and/or run-time dependencies that need to be
209 compiled for the host machine. Typically non-Python libraries which are being
210 linked.
211+* `nativeCheckInputs ? []`: Dependencies needed for running the [`checkPhase`](#ssec-check-phase). These
212+ are added to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs) when [`doCheck = true`](#var-stdenv-doCheck). Items listed in
213 `tests_require` go here.
214* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
215 `buildPythonPackage` also injects code into and wraps executables with the
···266267#### `buildPythonApplication` function {#buildpythonapplication-function}
268269+The [`buildPythonApplication`](#buildpythonapplication-function) function is practically the same as
270+[`buildPythonPackage`](#buildpythonpackage-function). The main purpose of this function is to build a Python
271package where one is interested only in the executables, and not importable
272+modules. For that reason, when adding this package to a [`python.buildEnv`](#python.buildenv-function), the
273modules won't be made available.
274275+Another difference is that [`buildPythonPackage`](#buildpythonpackage-function) by default prefixes the names of
276the packages with the version of the interpreter. Because this is irrelevant for
277applications, the prefix is omitted.
278279+When packaging a Python application with [`buildPythonApplication`](#buildpythonapplication-function), it should be
280called with `callPackage` and passed `python` or `pythonPackages` (possibly
281specifying an interpreter version), like this:
282···329duplication the `toPythonApplication` can be used to convert a library to an
330application.
331332+The Nix expression shall use [`buildPythonPackage`](#buildpythonpackage-function) and be called from
333`python-packages.nix`. A reference shall be created from `all-packages.nix` to
334the attribute in `python-packages.nix`, and the `toPythonApplication` shall be
335applied to the reference:
···341#### `toPythonModule` function {#topythonmodule-function}
342343In some cases, such as bindings, a package is created using
344+[`stdenv.mkDerivation`](#sec-using-stdenv) and added as attribute in `all-packages.nix`. The Python
345bindings should be made available from `python-packages.nix`. The
346`toPythonModule` function takes a derivation and makes certain Python-specific
347modifications.
···407408#### `python.withPackages` function {#python.withpackages-function}
409410+The [`python.withPackages`](#python.withpackages-function) function provides a simpler interface to the [`python.buildEnv`](#python.buildenv-function) functionality.
411It takes a function as an argument that is passed the set of python packages and returns the list
412+of the packages to be included in the environment. Using the [`withPackages`](#python.withpackages-function) function, the previous
413example for the Pyramid Web Framework environment can be written like this:
414415```nix
···418python.withPackages (ps: [ ps.pyramid ])
419```
420421+[`withPackages`](#python.withpackages-function) passes the correct package set for the specific interpreter
422version as an argument to the function. In the above example, `ps` equals
423`pythonPackages`. But you can also easily switch to using python3:
424···430431Now, `ps` is set to `python3Packages`, matching the version of the interpreter.
432433+As [`python.withPackages`](#python.withpackages-function) simply uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also
434supports the `env` attribute. The `shell.nix` file from the previous section can
435thus be also written like this:
436···443])).env
444```
445446+In contrast to [`python.buildEnv`](#python.buildenv-function), [`python.withPackages`](#python.withpackages-function) does not support the
447more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
448+need them, you have to use [`python.buildEnv`](#python.buildenv-function).
449450Python 2 namespace packages may provide `__init__.py` that collide. In that case
451+[`python.buildEnv`](#python.buildenv-function) should be used with `ignoreCollisions = true`.
452453#### Setup hooks {#setup-hooks}
454455The following are setup hooks specifically for Python packages. Most of these
456+are used in [`buildPythonPackage`](#buildpythonpackage-function).
457458- `eggUnpackhook` to move an egg to the correct folder so it can be installed
459 with the `eggInstallHook`
···486### Development mode {#development-mode}
487488Development or editable mode is supported. To develop Python packages
489+[`buildPythonPackage`](#buildpythonpackage-function) has additional logic inside `shellPhase` to run `pip
490install -e . --prefix $TMPDIR/`for the package.
491492Warning: `shellPhase` is executed only if `setup.py` exists.
···567But Python libraries you would like to use for development cannot be installed,
568at least not individually, because they won't be able to find each other
569resulting in import errors. Instead, it is possible to create an environment
570+with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#python.withpackages-function) where the interpreter and other
571executables are wrapped to be able to find each other and all of the modules.
572573In the following examples we will start by creating a simple, ad-hoc environment
···747 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
748 brings all attributes of `nixpkgs` in the local scope. These attributes form
749 the main package set.
750+2. Then we create a Python 3.11 environment with the [`withPackages`](#python.withpackages-function) function, as before.
751+3. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument
752 that takes the set of all Python packages and returns a list of packages to
753 include in the environment. Here, we select the packages `numpy` and `toolz`
754 from the package set.
···859#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs}
860861With Nix all packages are built by functions. The main function in Nix for
862+building Python libraries is [`buildPythonPackage`](#buildpythonpackage-function). Let's see how we can build the
863`toolz` package.
864865```nix
···904}
905```
906907+What happens here? The function [`buildPythonPackage`](#buildpythonpackage-function) is called and as argument
908it accepts a set. In this case the set is a recursive set, `rec`. One of the
909arguments is the name of the package, which consists of a basename (generally
910following the name on PyPi) and a version. Another argument, `src` specifies the
911source, which in this case is fetched from PyPI using the helper function
912`fetchPypi`. The argument `doCheck` is used to set whether tests should be run
913+when building the package. Since there are no tests, we rely on [`pythonImportsCheck`](#using-pythonimportscheck)
914to test whether the package can be imported. Furthermore, we specify some meta
915information. The output of the function is a derivation.
916···969So, what did we do here? Well, we took the Nix expression that we used earlier
970to build a Python environment, and said that we wanted to include our own
971version of `toolz`, named `my_toolz`. To introduce our own package in the scope
972+of [`withPackages`](#python.withpackages-function) we used a `let` expression. You can see that we used
973`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take
974`toolz` from the Nixpkgs package set this time, but instead took our own version
975that we introduced with the `let` expression.
···977#### Handling dependencies {#handling-dependencies}
978979Our example, `toolz`, does not have any dependencies on other Python packages or
980+system libraries. According to the manual, [`buildPythonPackage`](#buildpythonpackage-function) uses the
981+arguments [`buildInputs`](#var-stdenv-buildInputs) and [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs) to specify dependencies. If
982something is exclusively a build-time dependency, then the dependency should be
983+included in [`buildInputs`](#var-stdenv-buildInputs), but if it is (also) a runtime dependency, then it
984+should be added to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs). Test dependencies are considered
985+build-time dependencies and passed to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs).
986987+The following example shows which arguments are given to [`buildPythonPackage`](#buildpythonpackage-function) in
988order to build [`datashape`](https://github.com/blaze/datashape).
989990```nix
···1038```
10391040We can see several runtime dependencies, `numpy`, `multipledispatch`, and
1041+`python-dateutil`. Furthermore, we have [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) with `pytest`.
1042+`pytest` is a test runner and is only used during the [`checkPhase`](#ssec-check-phase) and is
1043+therefore not added to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs).
10441045In the previous case we had only dependencies on other Python packages to consider.
1046Occasionally you have also system libraries to consider. E.g., `lxml` provides
1047Python bindings to `libxml2` and `libxslt`. These libraries are only required
1048+when building the bindings and are therefore added as [`buildInputs`](#var-stdenv-buildInputs).
10491050```nix
1051{ lib
···1093The example below shows bindings to The Fastest Fourier Transform in the West,
1094commonly known as FFTW. On Nix we have separate packages of FFTW for the
1095different types of floats (`"single"`, `"double"`, `"long-double"`). The
1096+bindings need all three types, and therefore we add all three as [`buildInputs`](#var-stdenv-buildInputs).
1097The bindings don't expect to find each of them in a different folder, and
1098therefore we have to set `LDFLAGS` and `CFLAGS`.
1099···1158}
1159```
11601161+Note also the line [`doCheck = false;`](#var-stdenv-doCheck), we explicitly disabled running the test-suite.
11621163#### Testing Python Packages {#testing-python-packages}
1164···1167but is not usable at runtime. Currently, all packages will use the `test`
1168command provided by the setup.py (i.e. `python setup.py test`). However,
1169this is currently deprecated https://github.com/pypa/setuptools/pull/1878
1170+and your package should provide its own [`checkPhase`](#ssec-check-phase).
11711172::: {.note}
1173+The [`checkPhase`](#ssec-check-phase) for python maps to the `installCheckPhase` on a
1174normal derivation. This is due to many python packages not behaving well
1175to the pre-installed version of the package. Version info, and natively
1176compiled extensions generally only exist in the install directory, and
···1235#### Using pytestCheckHook {#using-pytestcheckhook}
12361237`pytestCheckHook` is a convenient hook which will substitute the setuptools
1238+`test` command for a [`checkPhase`](#ssec-check-phase) which runs `pytest`. This is also beneficial
1239when a package may need many items disabled to run the test suite.
12401241Using the example above, the analogous `pytestCheckHook` usage would be:
···1280```
12811282Trying to concatenate the related strings to disable tests in a regular
1283+[`checkPhase`](#ssec-check-phase) would be much harder to read. This also enables us to comment on
1284why specific tests are disabled.
12851286#### Using pythonImportsCheck {#using-pythonimportscheck}
12871288Although unit tests are highly preferred to validate correctness of a package, not
1289all packages have test suites that can be run easily, and some have none at all.
1290+To help ensure the package still works, [`pythonImportsCheck`](#using-pythonimportscheck) can attempt to import
1291the listed modules.
12921293```
···1306 '';
1307```
13081309+However, this is done in its own phase, and not dependent on whether [`doCheck = true;`](#var-stdenv-doCheck).
13101311This can also be useful in verifying that the package doesn't assume commonly
1312present packages (e.g. `setuptools`).
···13781379Keep in mind that while the examples above are done with `requirements.txt`,
1380`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should
1381+work with any of the [existing hooks](#setup-hooks).
13821383#### Using unittestCheckHook {#using-unittestcheckhook}
13841385+`unittestCheckHook` is a hook which will substitute the setuptools `test` command for a [`checkPhase`](#ssec-check-phase) which runs `python -m unittest discover`:
13861387```
1388 nativeCheckInputs = [
···1452In the previous Nix expression the source was fetched from a url. We can also
1453refer to a local source instead using `src = ./path/to/source/tree;`
14541455+If we create a `shell.nix` file which calls [`buildPythonPackage`](#buildpythonpackage-function), and if `src`
1456is a local source, and if the local source has a `setup.py`, then development
1457mode is activated.
14581459In the following example, we create a simple environment that has a Python 3.11
1460version of our package in it, as well as its dependencies and other packages we
1461+like to have in the environment, all specified with [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs).
1462Indeed, we can just add any package we like to have in our environment to
1463+[`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs).
14641465```nix
1466with import <nixpkgs> {};
···14941495### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
14961497+Earlier we created a Python environment using [`withPackages`](#python.withpackages-function), and included the
1498`toolz` package via a `let` expression.
1499Let's split the package definition from the environment definition.
1500···1533}
1534```
15351536+It takes an argument [`buildPythonPackage`](#buildpythonpackage-function). We now call this function using
1537`callPackage` in the definition of our environment
15381539```nix
···1552```
15531554Important to remember is that the Python version for which the package is made
1555+depends on the `python` derivation that is passed to [`buildPythonPackage`](#buildpythonpackage-function). Nix
1556tries to automatically pass arguments when possible, which is why generally you
1557don't explicitly define which `python` derivation should be used. In the above
1558+example we use [`buildPythonPackage`](#buildpythonpackage-function) that is part of the set `python3Packages`,
1559and in this case the `python3` interpreter is automatically used.
15601561## FAQ {#faq}
···1698packages are available. There is therefore no need to maintain a global `site-packages`.
16991700If you want to create a Python environment for development, then the recommended
1701+method is to use `nix-shell`, either with or without the [`python.buildEnv`](#python.buildenv-function)
1702function.
17031704### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
···18751876### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions}
18771878+The following overlay overrides the call to [`buildPythonPackage`](#buildpythonpackage-function) for the
1879`foo` package for all interpreters by appending a Python extension to the
1880`pythonPackagesExtensions` list of extensions.
1881···19021903In a `setup.py` or `setup.cfg` it is common to declare dependencies:
19041905+* `setup_requires` corresponds to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs)
1906+* `install_requires` corresponds to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)
1907+* `tests_require` corresponds to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs)
19081909### How to enable interpreter optimizations? {#optimizations}
1910···19511952### How to contribute a Python package to nixpkgs? {#tools}
19531954+Packages inside nixpkgs must use the [`buildPythonPackage`](#buildpythonpackage-function) or [`buildPythonApplication`](#buildpythonapplication-function) function directly,
1955because we can only provide security support for non-vendored dependencies.
19561957We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs,
···1965`nix-shell`.
19661967When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
1968+have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1`
1969and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED).
1970Both are also exported in `nix-shell`.
1971···1975Source distributions (`sdist`) often include test files, but not always.
19761977By default the command `python setup.py test` is run as part of the
1978+[`checkPhase`](#ssec-check-phase), but often it is necessary to pass a custom [`checkPhase`](#ssec-check-phase). An
1979example of such a situation is when `py.test` is used.
19801981#### Common issues {#common-issues}
19821983+* Non-working tests can often be deselected. By default [`buildPythonPackage`](#buildpythonpackage-function)
1984 runs `python setup.py test`. which is deprecated. Most Python modules however
1985 do follow the standard test protocol where the pytest runner can be used
1986 instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test
···2015The following rules are desired to be respected:
20162017* Python libraries are called from `python-packages.nix` and packaged with
2018+ [`buildPythonPackage`](#buildpythonpackage-function). The expression of a library should be in
2019 `pkgs/development/python-modules/<name>/default.nix`.
2020* Python applications live outside of `python-packages.nix` and are packaged
2021+ with [`buildPythonApplication`](#buildpythonapplication-function).
2022* Make sure libraries build for all Python interpreters.
2023* By default we enable tests. Make sure the tests are found and, in the case of
2024 libraries, are passing for all interpreters. If certain tests fail they can be
+2
nixos/doc/manual/release-notes/rl-2311.section.md
···229230- `rome` was removed because it is no longer maintained and is succeeded by `biome`.
23100232## Other Notable Changes {#sec-release-23.11-notable-changes}
233234- The Cinnamon module now enables XDG desktop integration by default. If you are experiencing collisions related to xdg-desktop-portal-gtk you can safely remove `xdg.portal.extraPortals = [ pkgs.xdg-desktop-portal-gtk ];` from your NixOS configuration.
···229230- `rome` was removed because it is no longer maintained and is succeeded by `biome`.
231232+- The `services.mtr-exporter.target` has been removed in favor of `services.mtr-exporter.jobs` which allows specifying multiple targets.
233+234## Other Notable Changes {#sec-release-23.11-notable-changes}
235236- The Cinnamon module now enables XDG desktop integration by default. If you are experiencing collisions related to xdg-desktop-portal-gtk you can safely remove `xdg.portal.extraPortals = [ pkgs.xdg-desktop-portal-gtk ];` from your NixOS configuration.
···14 # it is known to cause problems. Search online for "rpl_malloc" and
15 # "rpl_realloc" to find out more.
16 ./fix-rpl_malloc.patch
0017 ];
1819 nativeBuildInputs = [ autoreconfHook pkg-config ];
···14 # it is known to cause problems. Search online for "rpl_malloc" and
15 # "rpl_realloc" to find out more.
16 ./fix-rpl_malloc.patch
17+ ./suse-bug-580609.patch
18+ ./suse-bug-955687.patch
19 ];
2021 nativeBuildInputs = [ autoreconfHook pkg-config ];
···1314buildGoModule rec {
15 pname = "gopass";
16- version = "1.15.7";
1718 nativeBuildInputs = [ installShellFiles makeWrapper ];
19···21 owner = "gopasspw";
22 repo = "gopass";
23 rev = "v${version}";
24- hash = "sha256-Q3EX5giteIsH5+fXb7n2qpd9kBjaZZ/A5VuCljc72C8=";
25 };
2627- vendorHash = "sha256-crnr5qXlYrhNT3nLlA7U13CaYAmAqcV+MBs/hee9ixU=";
2829 subPackages = [ "." ];
30···62 homepage = "https://www.gopass.pw/";
63 license = licenses.mit;
64 maintainers = with maintainers; [ rvolosatovs sikmir ];
65- changelog = "https://github.com/gopasspw/gopass/raw/v${version}/CHANGELOG.md";
6667 longDescription = ''
68 gopass is a rewrite of the pass password manager in Go with the aim of
···73 users. We go by the UNIX philosophy and try to do one thing and do it
74 well, providing a stellar user experience and a sane, simple interface.
75 '';
076 };
77}
···1314buildGoModule rec {
15 pname = "gopass";
16+ version = "1.15.8";
1718 nativeBuildInputs = [ installShellFiles makeWrapper ];
19···21 owner = "gopasspw";
22 repo = "gopass";
23 rev = "v${version}";
24+ hash = "sha256-l8Ce0ioMnSlet+PMrQCMvyH3IvmQaE1MQSJR9myyLB8=";
25 };
2627+ vendorHash = "sha256-xyQTlbTPAC2iG8XQ4oEHBXjfXauwuBhaTbsew23nlVw=";
2829 subPackages = [ "." ];
30···62 homepage = "https://www.gopass.pw/";
63 license = licenses.mit;
64 maintainers = with maintainers; [ rvolosatovs sikmir ];
65+ changelog = "https://github.com/gopasspw/gopass/blob/v${version}/CHANGELOG.md";
6667 longDescription = ''
68 gopass is a rewrite of the pass password manager in Go with the aim of
···73 users. We go by the UNIX philosophy and try to do one thing and do it
74 well, providing a stellar user experience and a sane, simple interface.
75 '';
76+ mainProgram = "gopass";
77 };
78}