nixpkgs mirror (for testing) github.com/NixOS/nixpkgs
nix
at r-updates 2218 lines 81 kB view raw view rendered
1# Python {#python} 2 3## Reference {#reference} 4 5### Interpreters {#interpreters} 6 7@python-interpreter-table@ 8 9The Nix expressions for the interpreters can be found in 10`pkgs/development/interpreters/python`. 11 12All packages depending on any Python interpreter get appended 13`out/{python.sitePackages}` to `$PYTHONPATH` if such directory 14exists. 15 16#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library} 17 18To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`. 19 20#### Attributes on interpreters packages {#attributes-on-interpreters-packages} 21 22Each interpreter has the following attributes: 23 24- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter. 25- `interpreter`. Alias for `${python}/bin/${executable}`. 26- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See [](#python.buildenv-function) for usage and documentation. 27- `withPackages`. Simpler interface to `buildEnv`. See [](#python.withpackages-function) for usage and documentation. 28- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`. 29- `executable`. Name of the interpreter executable, e.g. `python3.10`. 30- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`. 31 32### Building packages and applications {#building-packages-and-applications} 33 34Python libraries and applications that use tools to follow PEP 517 (e.g. `setuptools` or `hatchling`, etc.) or 35previous tools such as `distutils` are typically built with respectively the [`buildPythonPackage`](#buildpythonpackage-function) and 36[`buildPythonApplication`](#buildpythonapplication-function) functions. These two functions also support installing a `wheel`. 37 38All Python packages reside in `pkgs/top-level/python-packages.nix` and all 39applications elsewhere. In case a package is used as both a library and an 40application, then the package should be in `pkgs/top-level/python-packages.nix` 41since only those packages are made available for all interpreter versions. The 42preferred location for library expressions is in 43`pkgs/development/python-modules`. It is important that these packages are 44called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee 45the right version of the package is built. 46 47Based on the packages defined in `pkgs/top-level/python-packages.nix` an 48attribute set is created for each available Python interpreter. The available 49sets are 50 51* `pkgs.python27Packages` 52* `pkgs.python3Packages` 53* `pkgs.python311Packages` 54* `pkgs.python312Packages` 55* `pkgs.python313Packages` 56* `pkgs.python314Packages` 57* `pkgs.python315Packages` 58* `pkgs.pypy27Packages` 59* `pkgs.pypy310Packages` 60 61and the aliases 62 63* `pkgs.python2Packages` pointing to `pkgs.python27Packages` 64* `pkgs.python3Packages` pointing to `pkgs.python313Packages` 65* `pkgs.pythonPackages` pointing to `pkgs.python2Packages` 66* `pkgs.pypy2Packages` pointing to `pkgs.pypy27Packages` 67* `pkgs.pypy3Packages` pointing to `pkgs.pypy310Packages` 68* `pkgs.pypyPackages` pointing to `pkgs.pypy2Packages` 69 70 71#### `buildPythonPackage` function {#buildpythonpackage-function} 72 73The `buildPythonPackage` function has its name binding in 74`pkgs/development/interpreters/python/python-packages-base.nix` and is 75implemented in `pkgs/development/interpreters/python/mk-python-derivation.nix` 76using setup hooks. 77 78The following is an example: 79 80```nix 81{ 82 lib, 83 buildPythonPackage, 84 fetchPypi, 85 86 # build-system 87 setuptools, 88 setuptools-scm, 89 90 # dependencies 91 attrs, 92 pluggy, 93 py, 94 setuptools, 95 six, 96 97 # tests 98 hypothesis, 99}: 100 101buildPythonPackage (finalAttrs: { 102 pname = "pytest"; 103 version = "3.3.1"; 104 pyproject = true; 105 106 src = fetchPypi { 107 inherit (finalAttrs) pname version; 108 109 hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M="; 110 }; 111 112 postPatch = '' 113 # don't test bash builtins 114 rm testing/test_argcomplete.py 115 ''; 116 117 build-system = [ 118 setuptools 119 setuptools-scm 120 ]; 121 122 dependencies = [ 123 attrs 124 py 125 setuptools 126 six 127 pluggy 128 ]; 129 130 nativeCheckInputs = [ hypothesis ]; 131 132 meta = { 133 changelog = "https://github.com/pytest-dev/pytest/releases/tag/${finalAttrs.version}"; 134 description = "Framework for writing tests"; 135 homepage = "https://github.com/pytest-dev/pytest"; 136 license = lib.licenses.mit; 137 maintainers = with lib.maintainers; [ 138 lovek323 139 madjar 140 lsix 141 ]; 142 }; 143}) 144``` 145 146The `buildPythonPackage` mainly does four things: 147 148* In the [`buildPhase`](#build-phase), it calls `${python.pythonOnBuildForHost.interpreter} -m build --wheel` to 149 build a wheel binary zipfile. 150* In the [`installPhase`](#ssec-install-phase), it installs the wheel file using `${python.pythonOnBuildForHost.interpreter} -m installer *.whl`. 151* In the [`postFixup`](#var-stdenv-postFixup) phase, the `wrapPythonPrograms` bash function is called to 152 wrap all programs in the `$out/bin/*` directory to include `$PATH` 153 environment variable and add dependent libraries to script's `sys.path`. 154* In the [`installCheck`](#ssec-installCheck-phase) phase, `${python.interpreter} -m pytest` is run. 155 156By default tests are run because [`doCheck = true`](#var-stdenv-doCheck). Test dependencies, like 157e.g. the test runner, should be added to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs). 158 159By default `meta.platforms` is set to the same value 160as the interpreter unless overridden otherwise. 161 162##### `buildPythonPackage` parameters {#buildpythonpackage-parameters} 163 164All parameters from [`stdenv.mkDerivation`](#sec-using-stdenv) function are still supported. The 165following are specific to `buildPythonPackage`: 166 167* `catchConflicts ? true`: If `true`, abort package build if a package name 168 appears more than once in dependency tree. Default is `true`. 169* `disabled ? false`: If `true`, package is not built for the particular Python 170 interpreter version. 171* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs. 172* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment 173 variable in wrapped programs. 174* `pyproject`: Whether the pyproject format should be used. As all other formats 175 are deprecated, you are recommended to set this to `true`. When you do so, 176 `pypaBuildHook` will be used, and you can add the required build dependencies 177 from `build-system.requires` to `build-system`. Note that the pyproject 178 format falls back to using `setuptools`, so you can use `pyproject = true` 179 even if the package only has a `setup.py`. When set to `false`, you can 180 use the existing [hooks](#setup-hooks) or provide your own logic to build the 181 package. This can be useful for packages that don't support the pyproject 182 format. When unset, the legacy `setuptools` hooks are used for backwards 183 compatibility. 184* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to 185 [`makeWrapper`](#fun-makeWrapper), which wraps generated binaries. By default, the arguments to 186 [`makeWrapper`](#fun-makeWrapper) set `PATH` and `PYTHONPATH` environment variables before calling 187 the binary. Additional arguments here can allow a developer to set environment 188 variables which will be available when the binary is run. For example, 189 `makeWrapperArgs = ["--set" "FOO" "BAR" "--set" "BAZ" "QUX"]`. 190 191 ::: {.note} 192 When `__structuredAttrs = false`, the attribute `makeWrapperArgs` is passed as a space-separated string to the build script. Developers should use `prependToVar` or `appendToVar` to add arguments to it in build phases, or use `__structuredAttrs = true` to ensure that `makeWrapperArgs` is passed as a Bash array. 193 194 For compatibility purposes, 195 when `makeWrapperArgs` shell variable is specified as a space-separated string (instead of a Bash array) in the build script, the string content is Bash-expanded before concatenated into the `wrapProgram` command. Still, developers should not rely on such behaviours, but use `__structuredAttrs = true` to specify flags containing spaces (e.g. `makeWrapperArgs = [ "--set" "GREETING" "Hello, world!" ]`), or use -pre and -post phases to specify flags with Bash-expansions (e.g. `preFixup = ''makeWrapperArgs+=(--prefix PATH : "$SOME_PATH")''`). 196 ::: 197 198* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this 199 defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`. 200* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`. 201* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages 202 in `pythonPath` are not propagated (contrary to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)). 203* `preShellHook`: Hook to execute commands before `shellHook`. 204* `postShellHook`: Hook to execute commands after `shellHook`. 205* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only 206 created when the filenames end with `.py`. 207* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command. 208* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command. 209 210The [`stdenv.mkDerivation`](#sec-using-stdenv) function accepts various parameters for describing 211build inputs (see "Specifying dependencies"). The following are of special 212interest for Python packages, either because these are primarily used, or 213because their behaviour is different: 214 215* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables. 216* `build-system ? []`: Build-time only Python dependencies. Items listed in `build-system.requires`/`setup_requires`. 217* `buildInputs ? []`: Build and/or run-time dependencies that need to be 218 compiled for the host machine. Typically non-Python libraries which are being 219 linked. 220* `nativeCheckInputs ? []`: Dependencies needed for running the [`checkPhase`](#ssec-check-phase). These 221 are added to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs) when [`doCheck = true`](#var-stdenv-doCheck). Items listed in 222 `tests_require` go here. 223* `dependencies ? []`: Aside from propagating dependencies, 224 `buildPythonPackage` also injects code into and wraps executables with the 225 paths included in this list. Items listed in `install_requires` go here. 226* `optional-dependencies ? { }`: Optional feature flagged dependencies. Items listed in `extras_require` go here. 227 228 229##### Overriding Python packages {#overriding-python-packages} 230 231The `buildPythonPackage` function has a `overridePythonAttrs` method that can be 232used to override the package. In the following example we create an environment 233where we have the `blaze` package using an older version of `pandas`. We 234override first the Python interpreter and pass `packageOverrides` which contains 235the overrides for packages in the package set. 236 237```nix 238with import <nixpkgs> { }; 239 240( 241 let 242 python = 243 let 244 packageOverrides = self: super: { 245 pandas = super.pandas.overridePythonAttrs (old: rec { 246 version = "0.19.1"; 247 src = fetchPypi { 248 pname = "pandas"; 249 inherit version; 250 hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE="; 251 }; 252 }); 253 }; 254 in 255 pkgs.python3.override { 256 inherit packageOverrides; 257 self = python; 258 }; 259 260 in 261 python.withPackages (ps: [ ps.blaze ]) 262).env 263``` 264 265The next example shows a non trivial overriding of the `blas` implementation to 266be used through out all of the Python package set: 267 268```nix 269{ 270 python3MyBlas = pkgs.python3.override { 271 packageOverrides = self: super: { 272 # We need toPythonModule for the package set to evaluate this 273 blas = super.toPythonModule (super.pkgs.blas.override { blasProvider = super.pkgs.mkl; }); 274 lapack = super.toPythonModule (super.pkgs.lapack.override { lapackProvider = super.pkgs.mkl; }); 275 }; 276 }; 277} 278``` 279 280This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations. 281Note that using `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in 282compilation issues, because scipy dependencies need to use the same blas implementation as well. 283 284#### `buildPythonApplication` function {#buildpythonapplication-function} 285 286The [`buildPythonApplication`](#buildpythonapplication-function) function is practically the same as 287[`buildPythonPackage`](#buildpythonpackage-function). The main purpose of this function is to build a Python 288package where one is interested only in the executables, and not importable 289modules. For that reason, when adding this package to a [`python.buildEnv`](#python.buildenv-function), the 290modules won't be made available. 291 292Another difference is that [`buildPythonPackage`](#buildpythonpackage-function) by default prefixes the names of 293the packages with the version of the interpreter. Because this is irrelevant for 294applications, the prefix is omitted. 295 296When packaging a Python application with [`buildPythonApplication`](#buildpythonapplication-function), it should be 297called with `callPackage` and passed `python3` or `python3Packages` (possibly 298specifying an interpreter version), like this: 299 300```nix 301{ 302 lib, 303 python3Packages, 304 fetchPypi, 305}: 306 307python3Packages.buildPythonApplication (finalAttrs: { 308 pname = "luigi"; 309 version = "2.7.9"; 310 pyproject = true; 311 312 src = fetchPypi { 313 inherit (finalAttrs) pname version; 314 hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw="; 315 }; 316 317 build-system = with python3Packages; [ setuptools ]; 318 319 dependencies = with python3Packages; [ 320 tornado 321 python-daemon 322 ]; 323 324 meta = { 325 # ... 326 }; 327}) 328``` 329 330This is then added to `pkgs/by-name` just as any other application would be. 331 332Since the package is an application, a consumer doesn't need to care about 333Python versions or modules, which is why they don't go in `python3Packages`. 334 335#### `toPythonApplication` function {#topythonapplication-function} 336 337A distinction is made between applications and libraries, however, sometimes a 338package is used as both. In this case the package is added as a library to 339`python-packages.nix` and as an application to `pkgs/by-name`. To reduce 340duplication the `toPythonApplication` can be used to convert a library to an 341application. 342 343The Nix expression shall use [`buildPythonPackage`](#buildpythonpackage-function) and be called from 344`python-packages.nix`. A reference shall be created from `pkgs/by-name` to 345the attribute in `python-packages.nix`, and the `toPythonApplication` shall be 346applied to the reference: 347 348```nix 349{ python3Packages }: 350 351python3Packages.toPythonApplication python3Packages.youtube-dl 352``` 353 354#### `toPythonModule` function {#topythonmodule-function} 355 356In some cases, such as bindings, a package is created using 357[`stdenv.mkDerivation`](#sec-using-stdenv) and added as attribute in `pkgs/by-name` or in `all-packages.nix`. The Python 358bindings should be made available from `python-packages.nix`. The 359`toPythonModule` function takes a derivation and makes certain Python-specific 360modifications. 361 362```nix 363{ 364 opencv = toPythonModule ( 365 pkgs.opencv.override { 366 enablePython = true; 367 pythonPackages = self; 368 } 369 ); 370} 371``` 372 373Do pay attention to passing in the right Python version! 374 375#### `mkPythonMetaPackage` function {#mkpythonmetapackage-function} 376 377This will create a meta package containing [metadata files](https://packaging.python.org/en/latest/specifications/recording-installed-packages/) to satisfy a dependency on a package, without it actually having been installed into the environment. 378In nixpkgs this is used to package Python packages with split binary/source distributions such as [psycopg2](https://pypi.org/project/psycopg2/)/[psycopg2-binary](https://pypi.org/project/psycopg2-binary/). 379 380```nix 381mkPythonMetaPackage { 382 pname = "psycopg2-binary"; 383 inherit (psycopg2) optional-dependencies version; 384 dependencies = [ psycopg2 ]; 385 meta = { inherit (psycopg2.meta) description homepage; }; 386} 387``` 388 389#### `mkPythonEditablePackage` function {#mkpythoneditablepackage-function} 390 391When developing Python packages it's common to install packages in [editable mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html). 392Like `mkPythonMetaPackage` this function exists to create an otherwise empty package, but also containing a pointer to an impure location outside the Nix store that can be changed without rebuilding. 393 394The editable root is passed as a string. Normally `.pth` files contains absolute paths to the mutable location. This isn't always ergonomic with Nix, so environment variables are expanded at runtime. 395This means that a shell hook setting up something like a `$REPO_ROOT` variable can be used as the relative package root. 396 397As an implementation detail, the [PEP-518](https://peps.python.org/pep-0518/) `build-system` specified won't be used, but instead the editable package will be built using [hatchling](https://pypi.org/project/hatchling/). 398The `build-system`'s provided will instead become runtime dependencies of the editable package. 399 400Note that overriding packages deeper in the dependency graph _can_ work, but it's not the primary use case and overriding existing packages can make others break in unexpected ways. 401 402```nix 403{ 404 pkgs ? import <nixpkgs> { }, 405}: 406 407let 408 pyproject = pkgs.lib.importTOML ./pyproject.toml; 409 410 myPython = pkgs.python.override { 411 self = myPython; 412 packageOverrides = pyfinal: pyprev: { 413 # An editable package with a script that loads our mutable location 414 my-editable = pyfinal.mkPythonEditablePackage { 415 # Inherit project metadata from pyproject.toml 416 pname = pyproject.project.name; 417 inherit (pyproject.project) version; 418 419 # The editable root passed as a string 420 root = "$REPO_ROOT/src"; # Use environment variable expansion at runtime 421 422 # Inject a script (other PEP-621 entrypoints are also accepted) 423 inherit (pyproject.project) scripts; 424 }; 425 }; 426 }; 427 428 pythonEnv = myPython.withPackages (ps: [ ps.my-editable ]); 429 430in 431pkgs.mkShell { packages = [ pythonEnv ]; } 432``` 433 434#### `python.buildEnv` function {#python.buildenv-function} 435 436Python environments can be created using the low-level `pkgs.buildEnv` function. 437This example shows how to create an environment that has the Pyramid Web Framework. 438Saving the following as `default.nix` 439 440```nix 441with import <nixpkgs> { }; 442 443python3.buildEnv.override { 444 extraLibs = [ python3Packages.pyramid ]; 445 ignoreCollisions = true; 446} 447``` 448 449and running `nix-build` will create 450 451``` 452/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env 453``` 454 455with wrapped binaries in `bin/`. 456 457You can also use the `env` attribute to create local environments with needed 458packages installed. This is somewhat comparable to `virtualenv`. For example, 459running `nix-shell` with the following `shell.nix` 460 461```nix 462with import <nixpkgs> { }; 463 464(python3.buildEnv.override { 465 extraLibs = with python3Packages; [ 466 numpy 467 requests 468 ]; 469}).env 470``` 471 472will drop you into a shell where Python will have the 473specified packages in its path. 474 475##### `python.buildEnv` arguments {#python.buildenv-arguments} 476 477 478* `extraLibs`: List of packages installed inside the environment. 479* `postBuild`: Shell command executed after the build of environment. 480* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`). 481* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in 482 wrapped binaries in the environment. 483 484#### `python.withPackages` function {#python.withpackages-function} 485 486The [`python.withPackages`](#python.withpackages-function) function provides a simpler interface to the [`python.buildEnv`](#python.buildenv-function) functionality. 487It takes a function as an argument that is passed the set of python packages and returns the list 488of the packages to be included in the environment. Using the [`withPackages`](#python.withpackages-function) function, the previous 489example for the Pyramid Web Framework environment can be written like this: 490 491```nix 492with import <nixpkgs> { }; 493 494python.withPackages (ps: [ ps.pyramid ]) 495``` 496 497[`withPackages`](#python.withpackages-function) passes the correct package set for the specific interpreter 498version as an argument to the function. In the above example, `ps` equals 499`pythonPackages`. But you can also easily switch to using python3: 500 501```nix 502with import <nixpkgs> { }; 503 504python3.withPackages (ps: [ ps.pyramid ]) 505``` 506 507Now, `ps` is set to `python3Packages`, matching the version of the interpreter. 508 509As [`python.withPackages`](#python.withpackages-function) uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also 510supports the `env` attribute. The `shell.nix` file from the previous section can 511thus be also written like this: 512 513```nix 514with import <nixpkgs> { }; 515 516(python3.withPackages ( 517 ps: with ps; [ 518 numpy 519 requests 520 ] 521)).env 522``` 523 524In contrast to [`python.buildEnv`](#python.buildenv-function), [`python.withPackages`](#python.withpackages-function) does not support the 525more advanced options such as `ignoreCollisions = true` or `postBuild`. If you 526need them, you have to use [`python.buildEnv`](#python.buildenv-function). 527 528Python 2 namespace packages may provide `__init__.py` that collide. In that case 529[`python.buildEnv`](#python.buildenv-function) should be used with `ignoreCollisions = true`. 530 531#### Setup hooks {#setup-hooks} 532 533The following are setup hooks specifically for Python packages. Most of these 534are used in [`buildPythonPackage`](#buildpythonpackage-function). 535 536- `eggUnpackhook` to move an egg to the correct folder so it can be installed 537 with the `eggInstallHook` 538- `eggBuildHook` to skip building for eggs. 539- `eggInstallHook` to install eggs. 540- `pypaBuildHook` to build a wheel using 541 [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and 542 PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still 543 be added as `build-system`. 544- `pypaInstallHook` to install wheels. 545- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook). 546- `pythonCatchConflictsHook` to fail if the package depends on two different versions of the same dependency. 547- `pythonImportsCheckHook` to check whether importing the listed modules works. 548- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package. 549 See [example usage](#using-pythonrelaxdepshook). 550- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder. 551- `setuptoolsBuildHook` to build a wheel using `setuptools`. 552- `sphinxHook` to build documentation and manpages using Sphinx. 553- `stestrCheckHook` to run tests with `stestr`. 554- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A 555 `venv` is created if it does not yet exist. `postVenvCreation` can be used to 556 to run commands only after venv is first created. 557- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed 558 with the `pipInstallHook`. 559- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook). 560 561#### Overriding build helpers {#overriding-python-build-helpers} 562 563Like many of the build helpers provided by Nixpkgs, Python build helpers typically provide a `<function>.override` attribute. 564It works like [`<pkg>.override`](#sec-pkg-override), and can be used to override the dependencies of each build helper. 565 566This allows specifying the stdenv to be used by `buildPythonPackage` or `buildPythonApplication`. The default (`python.stdenv`) can be overridden as follows: 567 568```nix 569buildPythonPackage.override { stdenv = customStdenv; } { 570 # package attrs... 571} 572``` 573 574## User Guide {#user-guide} 575 576### Using Python {#using-python} 577 578#### Overview {#overview} 579 580Several versions of the Python interpreter are available on Nix, as well as a 581high amount of packages. The attribute `python3` refers to the default 582interpreter, which is currently CPython 3.13. The attribute `python` refers to 583CPython 2.7 for backwards compatibility. It is also possible to refer to 584specific versions, e.g., `python313` refers to CPython 3.13, and `pypy` refers to 585the default PyPy interpreter. 586 587Python is used a lot, and in different ways. This affects also how it is 588packaged. In the case of Python on Nix, an important distinction is made between 589whether the package is considered primarily an application, or whether it should 590be used as a library, i.e., of primary interest are the modules in 591`site-packages` that should be importable. 592 593In the Nixpkgs tree Python applications can be found throughout, depending on 594what they do, and are called from the main package set. Python libraries, 595however, are in separate sets, with one set per interpreter version. 596 597The interpreters have several common attributes. One of these attributes is 598`pkgs`, which is a package set of Python libraries for this specific 599interpreter. E.g., the `toolz` package corresponding to the default interpreter 600is `python3.pkgs.toolz`, and the CPython 3.13 version is `python313.pkgs.toolz`. 601The main package set contains aliases to these package sets, e.g. 602`pythonPackages` refers to `python.pkgs` and `python313Packages` to 603`python313.pkgs`. 604 605#### Installing Python and packages {#installing-python-and-packages} 606 607The Nix and NixOS manuals explain how packages are generally installed. In the 608case of Python and Nix, it is important to make a distinction between whether the 609package is considered an application or a library. 610 611Applications on Nix are typically installed into your user profile imperatively 612using `nix-env -i`, and on NixOS declaratively by adding the package name to 613`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies 614such as libraries are automatically installed and should not be installed 615explicitly. 616 617The same goes for Python applications. Python applications can be installed in 618your profile, and will be wrapped to find their exact library dependencies, 619without impacting other applications or polluting your user environment. 620 621But Python libraries you would like to use for development cannot be installed, 622at least not individually, because they won't be able to find each other 623resulting in import errors. Instead, it is possible to create an environment 624with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#python.withpackages-function) where the interpreter and other 625executables are wrapped to be able to find each other and all of the modules. 626 627In the following examples we will start by creating a simple, ad-hoc environment 628with a nix-shell that has `numpy` and `toolz` in Python 3.13; then we will create 629a re-usable environment in a single-file Python script; then we will create a 630full Python environment for development with this same environment. 631 632Philosophically, this should be familiar to users who are used to a `venv` style 633of development: individual projects create their own Python environments without 634impacting the global environment or each other. 635 636#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell} 637 638The simplest way to start playing with the way nix wraps and sets up Python 639environments is with `nix-shell` at the cmdline. These environments create a 640temporary shell session with a Python and a *precise* list of packages (plus 641their runtime dependencies), with no other Python packages in the Python 642interpreter's scope. 643 644To create a Python 3.13 session with `numpy` and `toolz` available, run: 645 646```sh 647$ nix-shell -p 'python313.withPackages(ps: with ps; [ numpy toolz ])' 648``` 649 650By default `nix-shell` will start a `bash` session with this interpreter in our 651`PATH`, so if we then run: 652 653```Python console 654[nix-shell:~/src/nixpkgs]$ python3 655Python 3.13.3 (main, Apr 8 2025, 13:54:08) [GCC 14.2.1 20250322] on linux 656Type "help", "copyright", "credits" or "license" for more information. 657>>> import numpy; import toolz 658``` 659 660Note that no other modules are in scope, even if they were imperatively 661installed into our user environment as a dependency of a Python application: 662 663```Python console 664>>> import requests 665Traceback (most recent call last): 666 File "<stdin>", line 1, in <module> 667ModuleNotFoundError: No module named 'requests' 668``` 669 670We can add as many additional modules onto the `nix-shell` as we need, and we 671will still get 1 wrapped Python interpreter. We can start the interpreter 672directly like so: 673 674```sh 675$ nix-shell -p "python313.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3 676Python 3.13.3 (main, Apr 8 2025, 13:54:08) [GCC 14.2.1 20250322] on linux 677Type "help", "copyright", "credits" or "license" for more information. 678>>> import requests 679>>> 680``` 681 682Notice that this time it built a new Python environment, which now includes 683`requests`. Building an environment just creates wrapper scripts that expose the 684selected dependencies to the interpreter while re-using the actual modules. This 685means if any other env has installed `requests` or `numpy` in a different 686context, we don't need to recompile them -- we just recompile the wrapper script 687that sets up an interpreter pointing to them. This matters much more for "big" 688modules like `pytorch` or `tensorflow`. 689 690Module names usually match their names on [pypi.org](https://pypi.org/), but 691normalized according to PEP 503/508. (e.g. Foo__Bar.baz -> foo-bar-baz) 692You can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html) 693to find them as well (along with non-python packages). 694 695At this point we can create throwaway experimental Python environments with 696arbitrary dependencies. This is a good way to get a feel for how the Python 697interpreter and dependencies work in Nix and NixOS, but to do some actual 698development, we'll want to make it a bit more persistent. 699 700##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang} 701 702Sometimes, we have a script whose header looks like this: 703 704```python 705#!/usr/bin/env python3 706import numpy as np 707a = np.array([1,2]) 708b = np.array([3,4]) 709print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") 710``` 711 712Executing this script requires a `python3` that has `numpy`. Using what we learned 713in the previous section, we could startup a shell and just run it like so: 714 715```ShellSession 716$ nix-shell -p 'python313.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py' 717The dot product of [1 2] and [3 4] is: 11 718``` 719 720But if we maintain the script ourselves, and if there are more dependencies, it 721may be nice to encode those dependencies in source to make the script re-usable 722without that bit of knowledge. That can be done by using `nix-shell` as a 723[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so: 724 725```python 726#!/usr/bin/env nix-shell 727#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])" 728import numpy as np 729a = np.array([1,2]) 730b = np.array([3,4]) 731print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") 732``` 733 734Then we execute it, without requiring any environment setup at all! 735 736```sh 737$ ./foo.py 738The dot product of [1 2] and [3 4] is: 11 739``` 740 741If the dependencies are not available on the host where `foo.py` is executed, it 742will build or download them from a Nix binary cache prior to starting up, prior 743that it is executed on a machine with a multi-user Nix installation. 744 745This provides a way to ship a self bootstrapping Python script, akin to a 746statically linked binary, where it can be run on any machine (provided nix is 747installed) without having to assume that `numpy` is installed globally on the 748system. 749 750By default it is pulling the import checkout of Nixpkgs itself from our nix 751channel, which is nice as it cache-aligns with our other package builds, but we 752can make it fully reproducible by pinning the `nixpkgs` import: 753 754```python 755#!/usr/bin/env nix-shell 756#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.numpy ])" 757#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/e51209796c4262bfb8908e3d6d72302fe4e96f5f.tar.gz 758import numpy as np 759a = np.array([1,2]) 760b = np.array([3,4]) 761print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") 762``` 763 764This will execute with the exact same versions of Python 3.10, numpy, and system 765dependencies a year from now as it does today, because it will always use 766exactly git commit `e51209796c4262bfb8908e3d6d72302fe4e96f5f` of Nixpkgs for all 767of the package versions. 768 769This is also a great way to ensure the script executes identically on different 770servers. 771 772##### Load environment from `.nix` expression {#load-environment-from-.nix-expression} 773 774We've now seen how to create an ad-hoc temporary shell session, and how to 775create a single script with Python dependencies, but in the course of normal 776development we're usually working in an entire package repository. 777 778As explained [in the `nix-shell` section](https://nixos.org/manual/nix/stable/command-ref/nix-shell) of the Nix manual, `nix-shell` can also load an expression from a `.nix` file. 779Say we want to have Python 3.13, `numpy` and `toolz`, like before, 780in an environment. We can add a `shell.nix` file describing our dependencies: 781 782```nix 783with import <nixpkgs> { }; 784(python313.withPackages ( 785 ps: with ps; [ 786 numpy 787 toolz 788 ] 789)).env 790``` 791 792And then at the command line, just typing `nix-shell` produces the same 793environment as before. In a normal project, we'll likely have many more 794dependencies; this can provide a way for developers to share the environments 795with each other and with CI builders. 796 797What's happening here? 798 7991. We begin with importing the Nix Packages collections. `import <nixpkgs>` 800 imports the `<nixpkgs>` function, `{}` calls it and the `with` statement 801 brings all attributes of `nixpkgs` in the local scope. These attributes form 802 the main package set. 8032. Then we create a Python 3.13 environment with the [`withPackages`](#python.withpackages-function) function, as before. 8043. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument 805 that takes the set of all Python packages and returns a list of packages to 806 include in the environment. Here, we select the packages `numpy` and `toolz` 807 from the package set. 808 809To combine this with `mkShell` you can: 810 811```nix 812with import <nixpkgs> { }; 813let 814 pythonEnv = python313.withPackages (ps: [ 815 ps.numpy 816 ps.toolz 817 ]); 818in 819mkShell { 820 packages = [ 821 pythonEnv 822 823 black 824 mypy 825 826 libffi 827 openssl 828 ]; 829} 830``` 831 832This will create a unified environment that has not just our Python interpreter 833and its Python dependencies, but also tools like `black` or `mypy` and libraries 834like `libffi` and `openssl` in scope. This is generic and can span any number of 835tools or languages across the Nixpkgs ecosystem. 836 837##### Installing environments globally on the system {#installing-environments-globally-on-the-system} 838 839Up to now, we've been creating environments scoped to an ad-hoc shell session, 840or a single script, or a single project. This is generally advisable, as it 841avoids pollution across contexts. 842 843However, sometimes we know we will often want a Python with some basic packages, 844and want this available without having to enter into a shell or build context. 845This can be useful to have things like vim/emacs editors and plugins or shell 846tools "just work" without having to set them up, or when running other software 847that expects packages to be installed globally. 848 849To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/` 850that looks like this: 851 852```nix 853# ~/.config/nixpkgs/overlays/myEnv.nix 854self: super: { 855 myEnv = super.buildEnv { 856 name = "myEnv"; 857 paths = [ 858 # A Python 3 interpreter with some packages 859 (self.python3.withPackages ( 860 ps: with ps; [ 861 pyflakes 862 pytest 863 black 864 ] 865 )) 866 867 # Some other packages we'd like as part of this env 868 self.mypy 869 self.black 870 self.ripgrep 871 self.tmux 872 ]; 873 }; 874} 875``` 876 877You can then build and install this to your profile with: 878 879```sh 880nix-env -iA myEnv 881``` 882 883One limitation of this is that you can only have 1 Python env installed 884globally, since they conflict on the `python` to load out of your `PATH`. 885 886If you get a conflict or prefer to keep the setup clean, you can have `nix-env` 887atomically *uninstall* all other imperatively installed packages and replace 888your profile with just `myEnv` by using the `--replace` flag. 889 890##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix} 891 892For the sake of completeness, here's how to install the environment system-wide 893on NixOS. 894 895```nix 896{ 897 # ... 898 899 environment.systemPackages = with pkgs; [ 900 (python314.withPackages ( 901 ps: with ps; [ 902 numpy 903 toolz 904 ] 905 )) 906 ]; 907} 908``` 909 910### Developing with Python {#developing-with-python} 911 912Above, we were mostly just focused on use cases and what to do to get started 913creating working Python environments in nix. 914 915Now that you know the basics to be up and running, it is time to take a step 916back and take a deeper look at how Python packages are packaged on Nix. 917 918#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs} 919 920With Nix all packages are built by functions. The main function in Nix for 921building Python libraries is [`buildPythonPackage`](#buildpythonpackage-function). Let's see how we can build the 922`toolz` package. 923 924```nix 925{ 926 lib, 927 buildPythonPackage, 928 fetchPypi, 929 setuptools, 930}: 931 932buildPythonPackage (finalAttrs: { 933 pname = "toolz"; 934 version = "0.10.0"; 935 pyproject = true; 936 937 src = fetchPypi { 938 inherit (finalAttrs) pname version; 939 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA="; 940 }; 941 942 build-system = [ setuptools ]; 943 944 # has no tests 945 doCheck = false; 946 947 pythonImportsCheck = [ 948 "toolz.itertoolz" 949 "toolz.functoolz" 950 "toolz.dicttoolz" 951 ]; 952 953 meta = { 954 changelog = "https://github.com/pytoolz/toolz/releases/tag/${finalAttrs.version}"; 955 homepage = "https://github.com/pytoolz/toolz"; 956 description = "List processing tools and functional utilities"; 957 license = lib.licenses.bsd3; 958 }; 959}) 960``` 961 962What happens here? The function [`buildPythonPackage`](#buildpythonpackage-function) is called and as argument 963it accepts a set. In this case the set is a recursive set, `rec`. One of the 964arguments is the name of the package, which consists of a basename (generally 965following the name on PyPI) and a version. Another argument, `src` specifies the 966source, which in this case is fetched from PyPI using the helper function 967`fetchPypi`. The argument `doCheck` is used to set whether tests should be run 968when building the package. Since there are no tests, we rely on [`pythonImportsCheck`](#using-pythonimportscheck) 969to test whether the package can be imported. Furthermore, we specify some meta 970information. The output of the function is a derivation. 971 972An expression for `toolz` can be found in the Nixpkgs repository. As explained 973in the introduction of this Python section, a derivation of `toolz` is available 974for each interpreter version, e.g. `python313.pkgs.toolz` refers to the `toolz` 975derivation corresponding to the CPython 3.13 interpreter. 976 977The above example works when you're directly working on 978`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though, 979you will want to test a Nix expression outside of the Nixpkgs tree. 980 981The following expression creates a derivation for the `toolz` package, 982and adds it along with a `numpy` package to a Python environment. 983 984```nix 985with import <nixpkgs> { }; 986 987( 988 let 989 my_toolz = python313.pkgs.buildPythonPackage (finalAttrs: { 990 pname = "toolz"; 991 version = "0.10.0"; 992 pyproject = true; 993 994 src = fetchPypi { 995 inherit (finalAttrs) pname version; 996 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA="; 997 }; 998 999 build-system = [ python313.pkgs.setuptools ]; 1000 1001 # has no tests 1002 doCheck = false; 1003 1004 meta = { 1005 homepage = "https://github.com/pytoolz/toolz/"; 1006 description = "List processing tools and functional utilities"; 1007 # [...] 1008 }; 1009 }); 1010 1011 in 1012 python313.withPackages ( 1013 ps: with ps; [ 1014 numpy 1015 my_toolz 1016 ] 1017 ) 1018).env 1019``` 1020 1021Executing `nix-shell` will result in an environment in which you can use 1022Python 3.13 and the `toolz` package. As you can see we had to explicitly mention 1023for which Python version we want to build a package. 1024 1025So, what did we do here? Well, we took the Nix expression that we used earlier 1026to build a Python environment, and said that we wanted to include our own 1027version of `toolz`, named `my_toolz`. To introduce our own package in the scope 1028of [`withPackages`](#python.withpackages-function) we used a `let` expression. You can see that we used 1029`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take 1030`toolz` from the Nixpkgs package set this time, but instead took our own version 1031that we introduced with the `let` expression. 1032 1033#### Handling dependencies {#handling-dependencies} 1034 1035Our example, `toolz`, does not have any dependencies on other Python packages or system libraries. 1036[`buildPythonPackage`](#buildpythonpackage-function) uses the following arguments in the following circumstances: 1037 1038- `dependencies` - For Python runtime dependencies. 1039- `build-system` - For Python build-time requirements. 1040- [`buildInputs`](#var-stdenv-buildInputs) - For non-Python build-time requirements. 1041- [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) - For test dependencies 1042 1043Dependencies can belong to multiple arguments, for example if something is both a build time requirement & a runtime dependency. 1044 1045The following example shows which arguments are given to [`buildPythonPackage`](#buildpythonpackage-function) in 1046order to build [`datashape`](https://github.com/blaze/datashape). 1047 1048```nix 1049{ 1050 lib, 1051 buildPythonPackage, 1052 fetchPypi, 1053 1054 # build dependencies 1055 setuptools, 1056 1057 # dependencies 1058 numpy, 1059 multipledispatch, 1060 python-dateutil, 1061 1062 # tests 1063 pytestCheckHook, 1064}: 1065 1066buildPythonPackage (finalAttrs: { 1067 pname = "datashape"; 1068 version = "0.4.7"; 1069 pyproject = true; 1070 1071 src = fetchPypi { 1072 inherit (finalAttrs) pname version; 1073 hash = "sha256-FLLvdm1MllKrgTGC6Gb0k0deZeVYvtCCLji/B7uhong="; 1074 }; 1075 1076 build-system = [ setuptools ]; 1077 1078 dependencies = [ 1079 multipledispatch 1080 numpy 1081 python-dateutil 1082 ]; 1083 1084 nativeCheckInputs = [ pytestCheckHook ]; 1085 1086 meta = { 1087 changelog = "https://github.com/blaze/datashape/releases/tag/${finalAttrs.version}"; 1088 homepage = "https://github.com/ContinuumIO/datashape"; 1089 description = "Data description language"; 1090 license = lib.licenses.bsd2; 1091 }; 1092}) 1093``` 1094 1095We can see several runtime dependencies, `numpy`, `multipledispatch`, and 1096`python-dateutil`. Furthermore, we have [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) with `pytestCheckHook`. 1097`pytestCheckHook` is a test runner hook and is only used during the [`checkPhase`](#ssec-check-phase) and is 1098therefore not added to `dependencies`. 1099 1100In the previous case we had only dependencies on other Python packages to consider. 1101Occasionally you have also system libraries to consider. E.g., `lxml` provides 1102Python bindings to `libxml2` and `libxslt`. These libraries are only required 1103when building the bindings and are therefore added as [`buildInputs`](#var-stdenv-buildInputs). 1104 1105```nix 1106{ 1107 lib, 1108 buildPythonPackage, 1109 fetchPypi, 1110 setuptools, 1111 libxml2, 1112 libxslt, 1113}: 1114 1115buildPythonPackage (finalAttrs: { 1116 pname = "lxml"; 1117 version = "3.4.4"; 1118 pyproject = true; 1119 1120 src = fetchPypi { 1121 inherit (finalAttrs) pname version; 1122 hash = "sha256-s9NiusRxFydHzaNRMjjxFcvWxfi45jGb9ql6eJJyQJk="; 1123 }; 1124 1125 build-system = [ setuptools ]; 1126 1127 buildInputs = [ 1128 libxml2 1129 libxslt 1130 ]; 1131 1132 # tests are meant to be ran "in-place" in the same directory as src 1133 doCheck = false; 1134 1135 pythonImportsCheck = [ 1136 "lxml" 1137 "lxml.etree" 1138 ]; 1139 1140 meta = { 1141 changelog = "https://github.com/lxml/lxml/releases/tag/lxml-${finalAttrs.version}"; 1142 description = "Pythonic binding for the libxml2 and libxslt libraries"; 1143 homepage = "https://lxml.de"; 1144 license = lib.licenses.bsd3; 1145 maintainers = with lib.maintainers; [ sjourdois ]; 1146 }; 1147}) 1148``` 1149 1150In this example `lxml` and Nix are able to work out exactly where the relevant 1151files of the dependencies are. This is not always the case. 1152 1153The example below shows bindings to The Fastest Fourier Transform in the West, 1154commonly known as FFTW. On Nix we have separate packages of FFTW for the 1155different types of floats (`"single"`, `"double"`, `"long-double"`). The 1156bindings need all three types, and therefore we add all three as [`buildInputs`](#var-stdenv-buildInputs). 1157The bindings don't expect to find each of them in a different folder, and 1158therefore we have to set `LDFLAGS` and `CFLAGS`. 1159 1160```nix 1161{ 1162 lib, 1163 buildPythonPackage, 1164 fetchPypi, 1165 1166 # build dependencies 1167 setuptools, 1168 1169 # dependencies 1170 fftw, 1171 fftwFloat, 1172 fftwLongDouble, 1173 numpy, 1174 scipy, 1175}: 1176 1177buildPythonPackage (finalAttrs: { 1178 pname = "pyfftw"; 1179 version = "0.9.2"; 1180 pyproject = true; 1181 1182 src = fetchPypi { 1183 inherit (finalAttrs) pname version; 1184 hash = "sha256-9ru2r6kwhUCaskiFoaPNuJCfCVoUL01J40byvRt4kHQ="; 1185 }; 1186 1187 build-system = [ setuptools ]; 1188 1189 buildInputs = [ 1190 fftw 1191 fftwFloat 1192 fftwLongDouble 1193 ]; 1194 1195 dependencies = [ 1196 numpy 1197 scipy 1198 ]; 1199 1200 preConfigure = '' 1201 export LDFLAGS="-L${fftw.dev}/lib -L${fftwFloat.out}/lib -L${fftwLongDouble.out}/lib" 1202 export CFLAGS="-I${fftw.dev}/include -I${fftwFloat.dev}/include -I${fftwLongDouble.dev}/include" 1203 ''; 1204 1205 # Tests cannot import pyfftw. pyfftw works fine though. 1206 doCheck = false; 1207 1208 pythonImportsCheck = [ "pyfftw" ]; 1209 1210 meta = { 1211 changelog = "https://github.com/pyFFTW/pyFFTW/releases/tag/v${finalAttrs.version}"; 1212 description = "Pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms"; 1213 homepage = "http://hgomersall.github.com/pyFFTW"; 1214 license = with lib.licenses; [ 1215 bsd2 1216 bsd3 1217 ]; 1218 }; 1219}) 1220``` 1221 1222Note also the line [`doCheck = false;`](#var-stdenv-doCheck), we explicitly disabled running the test-suite. 1223 1224#### Testing Python Packages {#testing-python-packages} 1225 1226It is highly encouraged to have testing as part of the package build. This 1227helps to avoid situations where the package was able to build and install, 1228but is not usable at runtime. 1229Your package should provide its own [`checkPhase`](#ssec-check-phase). 1230 1231::: {.note} 1232The [`checkPhase`](#ssec-check-phase) for python maps to the `installCheckPhase` on a 1233normal derivation. This is due to many python packages not behaving well 1234to the pre-installed version of the package. Version info, and natively 1235compiled extensions generally only exist in the install directory, and 1236thus can cause issues when a test suite asserts on that behavior. 1237::: 1238 1239::: {.note} 1240Tests should only be disabled if they don't agree with nix 1241(e.g. external dependencies, network access, flakey tests), however, 1242as many tests should be enabled as possible. Failing tests can still be 1243a good indication that the package is not in a valid state. 1244::: 1245 1246::: {.note} 1247We only want to test the functionality of a package. In particular, we are not 1248interested in coverage, formatting, and type checking. If pytest fails with 1249`unrecognized arguments: --cov`, add `pytest-cov-stub` to `nativeCheckInputs` 1250rather than `pytest-cov`. 1251::: 1252 1253#### Using pytest {#using-pytest} 1254 1255Pytest is the most common test runner for python repositories. A trivial 1256test run would be: 1257 1258```nix 1259{ 1260 nativeCheckInputs = [ pytest ]; 1261 checkPhase = '' 1262 runHook preCheck 1263 1264 pytest 1265 1266 runHook postCheck 1267 ''; 1268} 1269``` 1270 1271However, many repositories' test suites do not translate well to nix's build 1272sandbox, and will generally need many tests to be disabled. 1273 1274This is achievable by 1275- Including paths or test items (`path/to/file.py::MyClass` or `path/to/file.py::MyClass::test_method`) with positional arguments. 1276- Excluding paths with `--ignore` or globbed paths with `--ignore-glob`. 1277- Excluding test items using the `--deselect` flag. 1278- Including or excluding classes or test methods by their name using the `-k` flag. 1279- Including or excluding test by their marks using the `-m` flag. 1280 1281We highly recommend `pytestCheckHook` for an easier and more structural setup. 1282 1283#### Using pytestCheckHook {#using-pytestcheckhook} 1284 1285`pytestCheckHook` is a convenient hook which will set up (or configure) 1286a [`checkPhase`](#ssec-check-phase) to run `pytest`. This is also beneficial 1287when a package may need many items disabled to run the test suite. 1288Most packages use `pytest` or `unittest`, which is compatible with `pytest`, 1289so you will most likely use `pytestCheckHook`. 1290 1291To use `pytestCheckHook`, add it to `nativeCheckInputs`. 1292Adding `pytest` is not required, since it is included with `pytestCheckHook`. 1293 1294```nix 1295{ nativeCheckInputs = [ pytestCheckHook ]; } 1296``` 1297 1298`pytestCheckHook` recognizes the following attributes: 1299 1300`enabledTestPaths` and `disabledTestPaths` 1301 1302: To specify path globs (files or directories) or test items. 1303 1304`enabledTests` and `disabledTests` 1305 1306: To specify keywords for class names or test method names. 1307 1308`enabledTestMarks` and `disabledTestMarks` 1309 1310: To specify test marks. 1311 1312`pytestFlags` 1313 1314: To append additional command-line arguments to `pytest`. 1315 1316By default, `pytest` automatically discovers which tests to run. 1317If tests are explicitly enabled, only those tests will run. 1318A test, that is both enabled and disabled, will not run. 1319 1320The following example demonstrates usage of various `pytestCheckHook` attributes: 1321 1322```nix 1323{ 1324 nativeCheckInputs = [ pytestCheckHook ]; 1325 1326 # Allow running the following test paths and test objects. 1327 enabledTestPaths = [ 1328 # Find tests under the tests directory. 1329 # The trailing slash is not necessary. 1330 "tests/" 1331 1332 # Additionally run test_foo 1333 "other-tests/test_foo.py::Foo::test_foo" 1334 ]; 1335 1336 # Override the above-enabled test paths and test objects. 1337 disabledTestPaths = [ 1338 # Tests under tests/integration requires additional data. 1339 "tests/integration" 1340 ]; 1341 1342 # Allow tests by keywords matching their class names or method names. 1343 enabledTests = [ 1344 # pytest by default only runs test methods begin with "test_" or end with "_test". 1345 # This includes all functions whose name contains "test". 1346 "test" 1347 ]; 1348 1349 # Override the above-enabled tests by keywords matching their class names or method names. 1350 disabledTests = [ 1351 # Tests touching networks. 1352 "upload" 1353 "download" 1354 ]; 1355 1356 # Additional pytest flags 1357 pytestFlags = [ 1358 # Disable benchmarks and run benchmarking tests only once. 1359 "--benchmark-disable" 1360 ]; 1361} 1362``` 1363 1364These attributes are all passed into the derivation directly 1365and added to the `pytest` command without additional Bash expansion. 1366It requires `__structuredAttrs = true` to pass list elements containing spaces. 1367 1368The `<enabled/disabled>TestsPaths` attributes expand Unix-style globs. 1369If a test path contains characters like `*`, `?`, `[`, or `]`, you can 1370quote them with square brackets (`[*]`, `[?]`, `[[]`, and `[]]`) to match literally. 1371 1372The `<enabled/disabled>Tests` and `<enabled/disabled>TestMarks` attribute pairs 1373form a logical expression `((included_element1) or (included_element2)) and not (excluded_element1) and not (excluded_element2)` 1374which will be passed to pytest's `-k` and `-m` flags respectively. 1375With `__structuredAttrs = true` enabled, they additionally support sub-expressions. 1376 1377For example, you could disable test items like `TestFoo::test_bar_functionality` 1378by disabling tests that match both `"Foo"` **and** `"bar"`: 1379 1380```nix 1381{ 1382 __structuredAttrs = true; 1383 1384 disabledTests = [ "Foo and bar" ]; 1385} 1386``` 1387 1388The main benefits of using `pytestCheckHook` to construct `pytest` commands 1389is structuralization and eval-time accessibility. 1390This is especially helpful to select tests or specify flags conditionally: 1391 1392```nix 1393{ 1394 disabledTests = [ 1395 # touches network 1396 "download" 1397 "update" 1398 ] 1399 ++ lib.optionals (pythonAtLeast "3.8") [ 1400 # broken due to python3.8 async changes 1401 "async" 1402 ] 1403 ++ lib.optionals stdenv.buildPlatform.isDarwin [ 1404 # can fail when building with other packages 1405 "socket" 1406 ]; 1407} 1408``` 1409 1410#### Using pythonImportsCheck {#using-pythonimportscheck} 1411 1412Although unit tests are highly preferred to validate correctness of a package, not 1413all packages have test suites that can be run easily, and some have none at all. 1414To help ensure the package still works, [`pythonImportsCheck`](#using-pythonimportscheck) can attempt to import 1415the listed modules. 1416 1417```nix 1418{ 1419 pythonImportsCheck = [ 1420 "requests" 1421 "urllib" 1422 ]; 1423} 1424``` 1425 1426roughly translates to: 1427 1428```nix 1429{ 1430 postCheck = '' 1431 PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH 1432 python -c "import requests; import urllib" 1433 ''; 1434} 1435``` 1436 1437However, this is done in its own phase, and not dependent on whether [`doCheck = true;`](#var-stdenv-doCheck). 1438 1439This can also be useful in verifying that the package doesn't assume commonly 1440present packages (e.g. `setuptools`). 1441 1442#### Using pythonRelaxDepsHook {#using-pythonrelaxdepshook} 1443 1444It is common for upstream to specify a range of versions for its package 1445dependencies. This makes sense, since it ensures that the package will be built 1446with a subset of packages that is well tested. However, this commonly causes 1447issues when packaging in Nixpkgs, because the dependencies that this package 1448may need are too new or old for the package to build correctly. We also cannot 1449package multiple versions of the same package since this may cause conflicts 1450in `PYTHONPATH`. 1451 1452One way to side step this issue is to relax the dependencies. This can be done 1453by either removing the package version range or by removing the package 1454declaration entirely. This can be done using the `pythonRelaxDepsHook` hook. For 1455example, given the following `requirements.txt` file: 1456 1457``` 1458pkg1<1.0 1459pkg2 1460pkg3>=1.0,<=2.0 1461``` 1462 1463we can do: 1464 1465```nix 1466{ 1467 pythonRelaxDeps = [ 1468 "pkg1" 1469 "pkg3" 1470 ]; 1471 pythonRemoveDeps = [ "pkg2" ]; 1472} 1473``` 1474 1475which would result in the following `requirements.txt` file: 1476 1477``` 1478pkg1 1479pkg3 1480``` 1481 1482Another option is to pass `true`, that will relax/remove all dependencies, for 1483example: 1484 1485```nix 1486{ pythonRelaxDeps = true; } 1487``` 1488 1489which would result in the following `requirements.txt` file: 1490 1491``` 1492pkg1 1493pkg2 1494pkg3 1495``` 1496 1497In general you should always use `pythonRelaxDeps`, because `pythonRemoveDeps` 1498will convert build errors into runtime errors. However `pythonRemoveDeps` may 1499still be useful in exceptional cases, and also to remove dependencies wrongly 1500declared by upstream (for example, declaring `black` as a runtime dependency 1501instead of a dev dependency). 1502 1503Keep in mind that while the examples above are done with `requirements.txt`, 1504`pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should 1505work with any of the [existing hooks](#setup-hooks). 1506 1507The `pythonRelaxDepsHook` has no effect on build time dependencies, such as 1508those specified in `build-system`. If a package requires incompatible build 1509time dependencies, they should be removed in `postPatch` through 1510`substituteInPlace` or similar. 1511 1512For ease of use, both `buildPythonPackage` and `buildPythonApplication` will 1513automatically add `pythonRelaxDepsHook` if either `pythonRelaxDeps` or 1514`pythonRemoveDeps` is specified. 1515 1516#### Using unittestCheckHook {#using-unittestcheckhook} 1517 1518`unittestCheckHook` is a hook which will set up (or configure) a [`checkPhase`](#ssec-check-phase) to run `python -m unittest discover`: 1519 1520```nix 1521{ 1522 nativeCheckInputs = [ unittestCheckHook ]; 1523 1524 unittestFlags = [ 1525 "-s" 1526 "tests" 1527 "-v" 1528 ]; 1529} 1530``` 1531 1532`pytest` is compatible with `unittest`, so in most cases you can use `pytestCheckHook` instead. 1533 1534#### Using sphinxHook {#using-sphinxhook} 1535 1536The `sphinxHook` is a helpful tool to build documentation and manpages 1537using the popular Sphinx documentation generator. 1538It is setup to automatically find common documentation source paths and 1539render them using the default `html` style. 1540 1541```nix 1542{ 1543 outputs = [ 1544 "out" 1545 "doc" 1546 ]; 1547 1548 nativeBuildInputs = [ sphinxHook ]; 1549} 1550``` 1551 1552The hook will automatically build and install the artifact into the 1553`doc` output, if it exists. It also provides an automatic diversion 1554for the artifacts of the `man` builder into the `man` target. 1555 1556```nix 1557{ 1558 outputs = [ 1559 "out" 1560 "doc" 1561 "man" 1562 ]; 1563 1564 # Use multiple builders 1565 sphinxBuilders = [ 1566 "singlehtml" 1567 "man" 1568 ]; 1569} 1570``` 1571 1572Overwrite `sphinxRoot` when the hook is unable to find your 1573documentation source root. 1574 1575```nix 1576{ 1577 # Configure sphinxRoot for uncommon paths 1578 sphinxRoot = "weird/docs/path"; 1579} 1580``` 1581 1582The hook is also available to packages outside the python ecosystem by 1583referencing it using `sphinxHook` from top-level. 1584 1585### Organising your packages {#organising-your-packages} 1586 1587So far we discussed how you can use Python on Nix, and how you can develop with 1588it. We've looked at how you write expressions to package Python packages, and we 1589looked at how you can create environments in which specified packages are 1590available. 1591 1592At some point you'll likely have multiple packages which you would 1593like to be able to use in different projects. In order to minimise unnecessary 1594duplication we now look at how you can maintain a repository with your 1595own packages. The important functions here are `import` and `callPackage`. 1596 1597### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage} 1598 1599Earlier we created a Python environment using [`withPackages`](#python.withpackages-function), and included the 1600`toolz` package via a `let` expression. 1601Let's split the package definition from the environment definition. 1602 1603We first create a function that builds `toolz` in `~/path/to/toolz/release.nix` 1604 1605```nix 1606{ 1607 lib, 1608 buildPythonPackage, 1609 fetchPypi, 1610 setuptools, 1611}: 1612 1613buildPythonPackage (finalAttrs: { 1614 pname = "toolz"; 1615 version = "0.10.0"; 1616 pyproject = true; 1617 1618 src = fetchPypi { 1619 inherit (finalAttrs) pname version; 1620 hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA="; 1621 }; 1622 1623 build-system = [ setuptools ]; 1624 1625 meta = { 1626 changelog = "https://github.com/pytoolz/toolz/releases/tag/${version}"; 1627 homepage = "https://github.com/pytoolz/toolz/"; 1628 description = "List processing tools and functional utilities"; 1629 license = lib.licenses.bsd3; 1630 }; 1631}) 1632``` 1633 1634It takes an argument [`buildPythonPackage`](#buildpythonpackage-function). We now call this function using 1635`callPackage` in the definition of our environment 1636 1637```nix 1638with import <nixpkgs> { }; 1639 1640( 1641 let 1642 toolz = callPackage /path/to/toolz/release.nix { 1643 buildPythonPackage = python3Packages.buildPythonPackage; 1644 }; 1645 in 1646 python3.withPackages (ps: [ 1647 ps.numpy 1648 toolz 1649 ]) 1650).env 1651``` 1652 1653Important to remember is that the Python version for which the package is made 1654depends on the `python` derivation that is passed to [`buildPythonPackage`](#buildpythonpackage-function). Nix 1655tries to automatically pass arguments when possible, which is why generally you 1656don't explicitly define which `python` derivation should be used. In the above 1657example we use [`buildPythonPackage`](#buildpythonpackage-function) that is part of the set `python3Packages`, 1658and in this case the `python3` interpreter is automatically used. 1659 1660## FAQ {#faq} 1661 1662### How to solve circular dependencies? {#how-to-solve-circular-dependencies} 1663 1664Consider the packages `A` and `B` that depend on each other. When packaging `B`, 1665a solution is to override package `A` not to depend on `B` as an input. The same 1666should also be done when packaging `A`. 1667 1668### How to override a Python package? {#how-to-override-a-python-package} 1669 1670We can override the interpreter and pass `packageOverrides`. In the following 1671example we rename the `pandas` package and build it. 1672 1673```nix 1674with import <nixpkgs> { }; 1675 1676( 1677 let 1678 python = 1679 let 1680 packageOverrides = self: super: { 1681 pandas = super.pandas.overridePythonAttrs (old: { 1682 name = "foo"; 1683 }); 1684 }; 1685 in 1686 pkgs.python313.override { inherit packageOverrides; }; 1687 1688 in 1689 python.withPackages (ps: [ ps.pandas ]) 1690).env 1691``` 1692 1693Using `nix-build` on this expression will build an environment that contains the 1694package `pandas` but with the new name `foo`. 1695 1696All packages in the package set will use the renamed package. A typical use case 1697is to switch to another version of a certain package. For example, in the 1698Nixpkgs repository we have multiple versions of `django` and `scipy`. In the 1699following example we use a different version of `scipy` and create an 1700environment that uses it. All packages in the Python package set will now use 1701the updated `scipy` version. 1702 1703```nix 1704with import <nixpkgs> { }; 1705 1706( 1707 let 1708 packageOverrides = self: super: { scipy = super.scipy_0_17; }; 1709 in 1710 (pkgs.python313.override { inherit packageOverrides; }).withPackages (ps: [ ps.blaze ]) 1711).env 1712``` 1713 1714The requested package `blaze` depends on `pandas` which itself depends on `scipy`. 1715 1716If you want the whole of Nixpkgs to use your modifications, then you can use 1717`overlays` as explained in this manual. In the following example we build a 1718`inkscape` using a different version of `numpy`. 1719 1720```nix 1721let 1722 pkgs = import <nixpkgs> { }; 1723 newpkgs = import pkgs.path { 1724 overlays = [ 1725 (self: super: { 1726 python313 = 1727 let 1728 packageOverrides = python-self: python-super: { 1729 numpy = python-super.numpy_1_18; 1730 }; 1731 in 1732 super.python313.override { inherit packageOverrides; }; 1733 }) 1734 ]; 1735 }; 1736in 1737newpkgs.inkscape 1738``` 1739 1740### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl} 1741 1742Executing `python setup.py bdist_wheel` in a `nix-shell`fails with 1743 1744``` 1745ValueError: ZIP does not support timestamps before 1980 1746``` 1747 1748This is because files from the Nix store (which have a timestamp of the UNIX 1749epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the 1750DOS convention of counting timestamps from 1980. 1751 1752The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable, 1753which `nix-shell` sets to 1. Unsetting this variable or giving it a value 1754corresponding to 1980 or later enables building wheels. 1755 1756Use 1980 as timestamp: 1757 1758```shell 1759nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel" 1760``` 1761 1762or the current time: 1763 1764```shell 1765nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel" 1766``` 1767 1768or unset `SOURCE_DATE_EPOCH`: 1769 1770```shell 1771nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel" 1772``` 1773 1774### `install_data` / `data_files` problems {#install_data-data_files-problems} 1775 1776If you get the following error: 1777 1778``` 1779could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc': 1780Permission denied 1781``` 1782 1783This is a [known bug](https://github.com/pypa/setuptools/issues/130) in 1784`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example 1785of such package using the feature is `pkgs/tools/X11/xpra/default.nix`. 1786 1787As workaround install it as an extra `preInstall` step: 1788 1789```shell 1790${python.pythonOnBuildForHost.interpreter} setup.py install_data --install-dir=$out --root=$out 1791sed -i '/ = data\_files/d' setup.py 1792``` 1793 1794### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages} 1795 1796On most operating systems a global `site-packages` is maintained. This however 1797becomes problematic if you want to run multiple Python versions or have multiple 1798versions of certain libraries for your projects. Generally, you would solve such 1799issues by creating virtual environments using `virtualenv`. 1800 1801On Nix each package has an isolated dependency tree which, in the case of 1802Python, guarantees the right versions of the interpreter and libraries or 1803packages are available. There is therefore no need to maintain a global `site-packages`. 1804 1805If you want to create a Python environment for development, then the recommended 1806method is to use `nix-shell`, either with or without the [`python.buildEnv`](#python.buildenv-function) 1807function. 1808 1809### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems} 1810 1811While this approach is not very idiomatic from Nix perspective, it can still be 1812useful when dealing with pre-existing projects or in situations where it's not 1813feasible or desired to write derivations for all required dependencies. 1814 1815This is an example of a `default.nix` for a `nix-shell`, which allows to consume 1816a virtual environment created by `venv`, and install Python modules through 1817`pip` the traditional way. 1818 1819Create this `default.nix` file, together with a `requirements.txt` and 1820execute `nix-shell`. 1821 1822```nix 1823with import <nixpkgs> { }; 1824 1825let 1826 pythonPackages = python3Packages; 1827in 1828pkgs.mkShell rec { 1829 name = "impurePythonEnv"; 1830 venvDir = "./.venv"; 1831 buildInputs = [ 1832 # A Python interpreter including the 'venv' module is required to bootstrap 1833 # the environment. 1834 pythonPackages.python 1835 1836 # This executes some shell code to initialize a venv in $venvDir before 1837 # dropping into the shell 1838 pythonPackages.venvShellHook 1839 1840 # Those are dependencies that we would like to use from nixpkgs, which will 1841 # add them to PYTHONPATH and thus make them accessible from within the venv. 1842 pythonPackages.numpy 1843 pythonPackages.requests 1844 1845 # In this particular example, in order to compile any binary extensions they may 1846 # require, the Python modules listed in the hypothetical requirements.txt need 1847 # the following packages to be installed locally: 1848 taglib 1849 openssl 1850 git 1851 libxml2 1852 libxslt 1853 libzip 1854 zlib 1855 ]; 1856 1857 # Run this command, only after creating the virtual environment 1858 postVenvCreation = '' 1859 unset SOURCE_DATE_EPOCH 1860 pip install -r requirements.txt 1861 ''; 1862 1863 # Now we can execute any commands within the virtual environment. 1864 # This is optional and can be left out to run pip manually. 1865 postShellHook = '' 1866 # allow pip to install wheels 1867 unset SOURCE_DATE_EPOCH 1868 ''; 1869 1870} 1871``` 1872 1873In case the supplied venvShellHook is insufficient, or when Python 2 support is 1874needed, you can define your own shell hook and adapt to your needs like in the 1875following example: 1876 1877```nix 1878with import <nixpkgs> { }; 1879 1880let 1881 venvDir = "./.venv"; 1882 pythonPackages = python3Packages; 1883in 1884pkgs.mkShell rec { 1885 name = "impurePythonEnv"; 1886 buildInputs = [ 1887 pythonPackages.python 1888 # Needed when using python 2.7 1889 # pythonPackages.virtualenv 1890 # ... 1891 ]; 1892 1893 # This is very close to how venvShellHook is implemented, but 1894 # adapted to use 'virtualenv' 1895 shellHook = '' 1896 SOURCE_DATE_EPOCH=$(date +%s) 1897 1898 if [ -d "${venvDir}" ]; then 1899 echo "Skipping venv creation, '${venvDir}' already exists" 1900 else 1901 echo "Creating new venv environment in path: '${venvDir}'" 1902 # Note that the module venv was only introduced in python 3, so for 2.7 1903 # this needs to be replaced with a call to virtualenv 1904 ${pythonPackages.python.interpreter} -m venv "${venvDir}" 1905 fi 1906 1907 # Under some circumstances it might be necessary to add your virtual 1908 # environment to PYTHONPATH, which you can do here too; 1909 # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH 1910 1911 source "${venvDir}/bin/activate" 1912 1913 # As in the previous example, this is optional. 1914 pip install -r requirements.txt 1915 ''; 1916} 1917``` 1918 1919Note that the `pip install` is an imperative action. So every time `nix-shell` 1920is executed it will attempt to download the Python modules listed in 1921requirements.txt. However these will be cached locally within the `virtualenv` 1922folder and not downloaded again. 1923 1924### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix} 1925 1926If you need to change a package's attribute(s) from `configuration.nix` you could do: 1927 1928```nix 1929{ 1930 nixpkgs.config.packageOverrides = super: { 1931 python3 = super.python3.override { 1932 packageOverrides = python-self: python-super: { 1933 twisted = python-super.twisted.overridePythonAttrs (oldAttrs: { 1934 src = super.fetchPypi { 1935 pname = "Twisted"; 1936 version = "19.10.0"; 1937 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0="; 1938 extension = "tar.bz2"; 1939 }; 1940 }); 1941 }; 1942 }; 1943 }; 1944} 1945``` 1946 1947`python3Packages.twisted` is now globally overridden. 1948All packages and also all NixOS services that reference `twisted` 1949(such as `services.buildbot-worker`) now use the new definition. 1950Note that `python-super` refers to the old package set and `python-self` 1951to the new, overridden version. 1952 1953To modify only a Python package set instead of a whole Python derivation, use 1954this snippet: 1955 1956```nix 1957{ 1958 myPythonPackages = python3Packages.override { overrides = self: super: { twisted = <...>; }; }; 1959} 1960``` 1961 1962### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays} 1963 1964Use the following overlay template: 1965 1966```nix 1967self: super: { 1968 python = super.python.override { 1969 packageOverrides = python-self: python-super: { 1970 twisted = python-super.twisted.overrideAttrs (oldAttrs: { 1971 src = super.fetchPypi { 1972 pname = "Twisted"; 1973 version = "19.10.0"; 1974 hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0="; 1975 extension = "tar.bz2"; 1976 }; 1977 }); 1978 }; 1979 }; 1980} 1981``` 1982 1983### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions} 1984 1985The following overlay overrides the call to [`buildPythonPackage`](#buildpythonpackage-function) for the 1986`foo` package for all interpreters by appending a Python extension to the 1987`pythonPackagesExtensions` list of extensions. 1988 1989```nix 1990final: prev: { 1991 pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [ 1992 (python-final: python-prev: { 1993 foo = python-prev.foo.overridePythonAttrs (oldAttrs: { 1994 # ... 1995 }); 1996 }) 1997 ]; 1998} 1999``` 2000 2001### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy} 2002 2003MKL can be configured using an overlay. See the section "[Using overlays to 2004configure alternatives](#sec-overlays-alternatives-blas-lapack)". 2005 2006### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to} 2007 2008In a `setup.py` or `setup.cfg` it is common to declare dependencies: 2009 2010* `setup_requires` corresponds to `build-system` 2011* `install_requires` corresponds to `dependencies` 2012* `tests_require` corresponds to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) 2013 2014### How to enable interpreter optimizations? {#optimizations} 2015 2016The Python interpreters are by default not built with optimizations enabled, because 2017the builds are in that case not reproducible. To enable optimizations, override the 2018interpreter of interest, e.g using 2019 2020```nix 2021let 2022 pkgs = import ./. { }; 2023 mypython = pkgs.python3.override { 2024 enableOptimizations = true; 2025 reproducibleBuild = false; 2026 self = mypython; 2027 }; 2028in 2029mypython 2030``` 2031 2032### How to add optional dependencies? {#python-optional-dependencies} 2033 2034Some packages define optional dependencies for additional features. With 2035`setuptools` this is called `extras_require` and `flit` calls it 2036`extras-require`, while PEP 621 calls these `optional-dependencies`. 2037 2038```nix 2039{ 2040 optional-dependencies = { 2041 complete = [ distributed ]; 2042 }; 2043} 2044``` 2045 2046and letting the package requiring the extra add the list to its dependencies 2047 2048```nix 2049{ 2050 dependencies = [ 2051 # ... 2052 ] 2053 ++ dask.optional-dependencies.complete; 2054} 2055``` 2056 2057This method is using `passthru`, meaning that changing `optional-dependencies` of a package won't cause it to rebuild. 2058 2059Note this method is preferred over adding parameters to builders, as that can 2060result in packages depending on different variants and thereby causing 2061collisions. 2062 2063::: {.note} 2064The `optional-dependencies` attribute should only be used for dependency groups 2065as defined in package metadata. If a package gracefully handles missing 2066dependencies in runtime but doesn't advertise it through package metadata, then 2067these dependencies should not be listed at all. (One may still have to list 2068them in `nativeCheckInputs` to pass test suite.) 2069::: 2070 2071### How to contribute a Python package to nixpkgs? {#tools} 2072 2073Packages inside nixpkgs must use the [`buildPythonPackage`](#buildpythonpackage-function) or [`buildPythonApplication`](#buildpythonapplication-function) function directly, 2074because we can only provide security support for non-vendored dependencies. 2075 2076We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs, 2077as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`. 2078When using the tool, pull from the original source repository instead of PyPI, if possible. 2079 2080See also [contributing section](#contributing). 2081 2082### Are Python interpreters built deterministically? {#deterministic-builds} 2083 2084The Python interpreters are now built deterministically. Minor modifications had 2085to be made to the interpreters in order to generate deterministic bytecode. This 2086has security implications and is relevant for those using Python in a 2087`nix-shell`. 2088 2089When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will 2090have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1` 2091and [PYTHONHASHSEED=0](https://docs.python.org/3.13/using/cmdline.html#envvar-PYTHONHASHSEED). 2092Both are also exported in `nix-shell`. 2093 2094### How to provide automatic tests to Python packages? {#automatic-tests} 2095 2096It is recommended to test packages as part of the build process. 2097Source distributions (`sdist`) often include test files, but not always. 2098 2099The best practice today is to pass a test hook (e.g. pytestCheckHook, unittestCheckHook) into nativeCheckInputs. 2100This will reconfigure the checkPhase to make use of that particular test framework. 2101Occasionally packages don't make use of a common test framework, which may then require a custom checkPhase. 2102 2103#### Common issues {#common-issues} 2104 2105* Tests that attempt to access `$HOME` can be fixed by using `writableTmpDirAsHomeHook` in 2106 `nativeCheckInputs`, which sets up a writable temporary directory as the home directory. Alternatively, 2107 you can achieve the same effect manually (e.g. in `preCheck`) with: `export HOME=$(mktemp -d)`. 2108* Compiling with Cython causes tests to fail with a `ModuleNotLoadedError`. 2109 This can be fixed with two changes in the derivation: 1) replacing `pytest` with 2110 `pytestCheckHook` and 2) adding a `preCheck` containing `cd $out` to run 2111 tests within the built output. 2112 2113## Contributing {#contributing} 2114 2115### Contributing guidelines {#contributing-guidelines} 2116 2117The following rules are desired to be respected: 2118 2119* Python libraries are called from `python-packages.nix` and packaged with 2120 [`buildPythonPackage`](#buildpythonpackage-function). The expression of a library should be in 2121 `pkgs/development/python-modules/<name>/default.nix`. 2122* Python applications live outside of `python-packages.nix` and are packaged 2123 with [`buildPythonApplication`](#buildpythonapplication-function). 2124* Make sure libraries build for all Python interpreters. 2125 If it fails to build on some Python versions, consider disabling them by setting `disable = pythonAtLeast "3.x"` along with a comment. 2126* The two parameters, `pyproject` and `build-system` are set to avoid the legacy setuptools/distutils build. 2127* Only unversioned attributes (e.g. `pydantic`, but not `pypdantic_1`) can be included in `dependencies`, 2128 since due to `PYTHONPATH` limitations we can only ever support a single version for libraries 2129 without running into duplicate module name conflicts. 2130* The version restrictions of `dependencies` can be relaxed by [`pythonRelaxDepsHook`](#using-pythonrelaxdepshook). 2131* Make sure the tests are enabled using for example [`pytestCheckHook`](#using-pytestcheckhook) and, in the case of 2132 libraries, are passing for all interpreters. If certain tests fail they can be 2133 disabled individually. Try to avoid disabling the tests altogether. In any 2134 case, when you disable tests, leave a comment explaining not only _what_ the failure 2135 is but _why_ the test failure can be ignored for safe distribution with nixpkgs. 2136* `pythonImportsCheck` is set. This is still a good smoke test even if `pytestCheckHook` is set. 2137* `meta.platforms` takes the default value in many cases. 2138 It does not need to be set explicitly unless the package requires a specific platform. 2139* The file is formatted correctly (e.g., `nix-shell --run treefmt`). 2140* Commit names of Python libraries must reflect that they are Python 2141 libraries (e.g. `python3Packages.numpy: 1.11 -> 1.12` rather than `numpy: 1.11 -> 1.12`). 2142 See also [`pkgs/README.md`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#commit-conventions). 2143* Attribute names in `python-packages.nix` as well as `pname`s should match the 2144 library's name on PyPI, but be normalized according to [PEP 2145 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means 2146 that characters should be converted to lowercase and `.` and `_` should be 2147 replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz). 2148 If necessary, `pname` has to be given a different value within `fetchPypi`. 2149* It's generally preferable to fetch `src` directly from the repo and not from 2150 PyPI. Use `fetchPypi` when there's a clear technical reason to do so. 2151* Packages from sources such as GitHub and GitLab that do not exist on PyPI 2152 should not use a name that is already used on PyPI. When possible, they should 2153 use the package repository name prefixed with the owner (e.g. organization) name 2154 and using a `-` as delimiter. 2155* Attribute names in `python-packages.nix` should be sorted alphanumerically to 2156 avoid merge conflicts and ease locating attributes. 2157* Non-python runtime dependencies should be added via explicit wrapping or 2158 patching (using e.g. `substituteInPlace`), rather than through propagation via 2159 `dependencies`/`propagatedBuildInputs`, to reduce clutter in `$PATH`. 2160 2161This list is useful for reviewers as well as for self-checking when submitting packages. 2162 2163## Package set maintenance {#python-package-set-maintenance} 2164 2165The whole Python package set has a lot of packages that do not see regular 2166updates, because they either are a very fragile component in the Python 2167ecosystem, like for example the `hypothesis` package, or packages that have 2168no maintainer, so maintenance falls back to the package set maintainers. 2169 2170### Updating packages in bulk {#python-package-bulk-updates} 2171 2172A tool to bulk-update numerous Python libraries is available in the 2173repository at `maintainers/scripts/update-python-libraries`. 2174 2175It can quickly update minor or major versions for all packages selected 2176and create update commits, and supports the `fetchPypi`, `fetchurl` and 2177`fetchFromGitHub` fetchers. When updating lots of packages that are 2178hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended. 2179 2180Updating packages in bulk leads to lots of breakages, which is why a 2181stabilization period on the `python-updates` branch is required. 2182 2183If a package is fragile and often breaks during these bulks updates, it 2184may be reasonable to set `passthru.skipBulkUpdate = true` in the 2185derivation. This decision should not be made on a whim and should 2186always be supported by a qualifying comment. 2187 2188Once the branch is sufficiently stable it should normally be merged 2189into the `staging` branch. 2190 2191An exemplary call to update all python libraries between minor versions 2192would be: 2193 2194```ShellSession 2195$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix 2196``` 2197 2198## CPython Update Schedule {#python-cpython-update-schedule} 2199 2200With [PEP 602](https://www.python.org/dev/peps/pep-0602/), CPython now 2201follows a yearly release cadence. In nixpkgs, all supported interpreters 2202are made available, but only the most recent two 2203interpreters package sets are built; this is a compromise between being 2204the latest interpreter, and what the majority of the Python packages support. 2205 2206New CPython interpreters are released in October. Generally, it takes some 2207time for the majority of active Python projects to support the latest stable 2208interpreter. To help ease the migration for Nixpkgs users 2209between Python interpreters the schedule below will be used: 2210 2211| When | Event | 2212| --- | --- | 2213| After YY.11 Release | Bump CPython package set window. The latest and previous latest stable should now be built. | 2214| After YY.05 Release | Bump default CPython interpreter to latest stable. | 2215 2216In practice, this means that the Python community will have had a stable interpreter 2217for ~2 months before attempting to update the package set. And this will 2218allow for ~7 months for Python applications to support the latest interpreter.