pyrox.dev
1<section xmlns="http://docbook.org/ns/docbook"
2 xmlns:xlink="http://www.w3.org/1999/xlink"
3 xml:id="sec-python">
4
5<title>Python</title>
6
7<para>
8 Currently supported interpreters are <varname>python26</varname>, <varname>python27</varname>,
9 <varname>python33</varname>, <varname>python34</varname>, <varname>python35</varname>
10 and <varname>pypy</varname>.
11</para>
12
13<para>
14 <varname>python</varname> is an alias to <varname>python27</varname> and <varname>python3</varname> is an alias to <varname>python34</varname>.
15</para>
16
17<para>
18 <varname>python26</varname> and <varname>python27</varname> do not include modules that require
19 external dependencies (to reduce dependency bloat). Following modules need to be added as
20 <varname>buildInput</varname> explicitly:
21</para>
22
23<itemizedlist>
24 <listitem><para><varname>python.modules.bsddb</varname></para></listitem>
25 <listitem><para><varname>python.modules.curses</varname></para></listitem>
26 <listitem><para><varname>python.modules.curses_panel</varname></para></listitem>
27 <listitem><para><varname>python.modules.crypt</varname></para></listitem>
28 <listitem><para><varname>python.modules.gdbm</varname></para></listitem>
29 <listitem><para><varname>python.modules.sqlite3</varname></para></listitem>
30 <listitem><para><varname>python.modules.tkinter</varname></para></listitem>
31 <listitem><para><varname>python.modules.readline</varname></para></listitem>
32</itemizedlist>
33
34<para>For convenience <varname>python27Full</varname> and <varname>python26Full</varname>
35are provided with all modules included.</para>
36
37<para>
38 Python packages that
39 use <link xlink:href="http://pypi.python.org/pypi/setuptools/"><literal>setuptools</literal></link> or <literal>distutils</literal>,
40 can be built using the <varname>buildPythonPackage</varname> function as documented below.
41</para>
42
43<para>
44 All packages depending on any Python interpreter get appended <varname>$out/${python.sitePackages}</varname>
45 to <literal>$PYTHONPATH</literal> if such directory exists.
46</para>
47
48<variablelist>
49 <title>
50 Useful attributes on interpreters packages:
51 </title>
52
53 <varlistentry>
54 <term><varname>libPrefix</varname></term>
55 <listitem><para>
56 Name of the folder in <literal>${python}/lib/</literal> for corresponding interpreter.
57 </para></listitem>
58 </varlistentry>
59
60 <varlistentry>
61 <term><varname>interpreter</varname></term>
62 <listitem><para>
63 Alias for <literal>${python}/bin/${executable}.</literal>
64 </para></listitem>
65 </varlistentry>
66
67 <varlistentry>
68 <term><varname>buildEnv</varname></term>
69 <listitem><para>
70 Function to build python interpreter environments with extra packages bundled together.
71 See <xref linkend="ssec-python-build-env" /> for usage and documentation.
72 </para></listitem>
73 </varlistentry>
74
75 <varlistentry>
76 <term><varname>sitePackages</varname></term>
77 <listitem><para>
78 Alias for <literal>lib/${libPrefix}/site-packages</literal>.
79 </para></listitem>
80 </varlistentry>
81
82 <varlistentry>
83 <term><varname>executable</varname></term>
84 <listitem><para>
85 Name of the interpreter executable, ie <literal>python3.4</literal>.
86 </para></listitem>
87 </varlistentry>
88
89</variablelist>
90<section xml:id="ssec-build-python-package"><title><varname>buildPythonPackage</varname> function</title>
91
92 <para>
93 The function is implemented in <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/generic/default.nix">
94 <filename>pkgs/development/python-modules/generic/default.nix</filename></link>.
95 Example usage:
96
97<programlisting language="nix">
98twisted = buildPythonPackage {
99 name = "twisted-8.1.0";
100
101 src = pkgs.fetchurl {
102 url = http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/Twisted-8.1.0.tar.bz2;
103 sha256 = "0q25zbr4xzknaghha72mq57kh53qw1bf8csgp63pm9sfi72qhirl";
104 };
105
106 propagatedBuildInputs = [ self.ZopeInterface ];
107
108 meta = {
109 homepage = http://twistedmatrix.com/;
110 description = "Twisted, an event-driven networking engine written in Python";
111 license = stdenv.lib.licenses.mit;
112 };
113};
114</programlisting>
115
116 Most of Python packages that use <varname>buildPythonPackage</varname> are defined
117 in <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/python-packages.nix"><filename>pkgs/top-level/python-packages.nix</filename></link>
118 and generated for each python interpreter separately into attribute sets <varname>python26Packages</varname>,
119 <varname>python27Packages</varname>, <varname>python35Packages</varname>, <varname>python33Packages</varname>,
120 <varname>python34Packages</varname> and <varname>pypyPackages</varname>.
121 </para>
122
123 <para>
124 <function>buildPythonPackage</function> mainly does four things:
125
126 <orderedlist>
127 <listitem><para>
128 In the <varname>buildPhase</varname>, it calls
129 <literal>${python.interpreter} setup.py bdist_wheel</literal> to build a wheel binary zipfile.
130 </para></listitem>
131
132 <listitem><para>
133 In the <varname>installPhase</varname>, it installs the wheel file using
134 <literal>pip install *.whl</literal>.
135 </para></listitem>
136
137 <listitem><para>
138 In the <varname>postFixup</varname> phase, <literal>wrapPythonPrograms</literal>
139 bash function is called to wrap all programs in <filename>$out/bin/*</filename>
140 directory to include <literal>$PYTHONPATH</literal> and <literal>$PATH</literal>
141 environment variables.
142 </para></listitem>
143
144 <listitem><para>
145 In the <varname>installCheck</varname> phase, <literal>${python.interpreter} setup.py test</literal>
146 is ran.
147 </para></listitem>
148 </orderedlist>
149 </para>
150
151 <para>By default <varname>doCheck = true</varname> is set</para>
152
153 <para>
154 As in Perl, dependencies on other Python packages can be specified in the
155 <varname>buildInputs</varname> and
156 <varname>propagatedBuildInputs</varname> attributes. If something is
157 exclusively a build-time dependency, use
158 <varname>buildInputs</varname>; if it’s (also) a runtime dependency,
159 use <varname>propagatedBuildInputs</varname>.
160 </para>
161
162 <para>
163 By default <varname>meta.platforms</varname> is set to the same value
164 as the interpreter unless overriden otherwise.
165 </para>
166
167 <variablelist>
168 <title>
169 <varname>buildPythonPackage</varname> parameters
170 (all parameters from <varname>mkDerivation</varname> function are still supported)
171 </title>
172
173 <varlistentry>
174 <term><varname>namePrefix</varname></term>
175 <listitem><para>
176 Prepended text to <varname>${name}</varname> parameter.
177 Defaults to <literal>"python3.3-"</literal> for Python 3.3, etc. Set it to
178 <literal>""</literal>
179 if you're packaging an application or a command line tool.
180 </para></listitem>
181 </varlistentry>
182
183 <varlistentry>
184 <term><varname>disabled</varname></term>
185 <listitem><para>
186 If <varname>true</varname>, package is not build for
187 particular python interpreter version. Grep around
188 <filename>pkgs/top-level/python-packages.nix</filename>
189 for examples.
190 </para></listitem>
191 </varlistentry>
192
193 <varlistentry>
194 <term><varname>setupPyBuildFlags</varname></term>
195 <listitem><para>
196 List of flags passed to <command>setup.py build_ext</command> command.
197 </para></listitem>
198 </varlistentry>
199
200 <varlistentry>
201 <term><varname>pythonPath</varname></term>
202 <listitem><para>
203 List of packages to be added into <literal>$PYTHONPATH</literal>.
204 Packages in <varname>pythonPath</varname> are not propagated
205 (contrary to <varname>propagatedBuildInputs</varname>).
206 </para></listitem>
207 </varlistentry>
208
209 <varlistentry>
210 <term><varname>preShellHook</varname></term>
211 <listitem><para>
212 Hook to execute commands before <varname>shellHook</varname>.
213 </para></listitem>
214 </varlistentry>
215
216 <varlistentry>
217 <term><varname>postShellHook</varname></term>
218 <listitem><para>
219 Hook to execute commands after <varname>shellHook</varname>.
220 </para></listitem>
221 </varlistentry>
222
223 <varlistentry>
224 <term><varname>makeWrapperArgs</varname></term>
225 <listitem><para>
226 A list of strings. Arguments to be passed to
227 <varname>makeWrapper</varname>, which wraps generated binaries. By
228 default, the arguments to <varname>makeWrapper</varname> set
229 <varname>PATH</varname> and <varname>PYTHONPATH</varname> environment
230 variables before calling the binary. Additional arguments here can
231 allow a developer to set environment variables which will be
232 available when the binary is run. For example,
233 <varname>makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]</varname>.
234 </para></listitem>
235 </varlistentry>
236
237 </variablelist>
238
239</section>
240
241<section xml:id="ssec-python-build-env"><title><function>python.buildEnv</function> function</title>
242 <para>
243 Create Python environments using low-level <function>pkgs.buildEnv</function> function. Example <filename>default.nix</filename>:
244
245<programlisting language="nix">
246<![CDATA[with import <nixpkgs> {};
247
248python.buildEnv.override {
249 extraLibs = [ pkgs.pythonPackages.pyramid ];
250 ignoreCollisions = true;
251}]]>
252</programlisting>
253
254 Running <command>nix-build</command> will create
255 <filename>/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env</filename>
256 with wrapped binaries in <filename>bin/</filename>.
257 </para>
258
259 <para>
260 You can also use <varname>env</varname> attribute to create local
261 environments with needed packages installed (somewhat comparable to
262 <literal>virtualenv</literal>). For example, with the following
263 <filename>shell.nix</filename>:
264
265<programlisting language="nix">
266<![CDATA[with import <nixpkgs> {};
267
268(python3.buildEnv.override {
269 extraLibs = with python3Packages;
270 [ numpy
271 requests
272 ];
273}).env]]>
274</programlisting>
275
276 Running <command>nix-shell</command> will drop you into a shell where
277 <command>python</command> will have specified packages in its path.
278 </para>
279
280 <variablelist>
281 <title>
282 <function>python.buildEnv</function> arguments
283 </title>
284
285 <varlistentry>
286 <term><varname>extraLibs</varname></term>
287 <listitem><para>
288 List of packages installed inside the environment.
289 </para></listitem>
290 </varlistentry>
291
292 <varlistentry>
293 <term><varname>postBuild</varname></term>
294 <listitem><para>
295 Shell command executed after the build of environment.
296 </para></listitem>
297 </varlistentry>
298
299 <varlistentry>
300 <term><varname>ignoreCollisions</varname></term>
301 <listitem><para>
302 Ignore file collisions inside the environment (default is <varname>false</varname>).
303 </para></listitem>
304 </varlistentry>
305 </variablelist>
306</section>
307
308<section xml:id="ssec-python-tools"><title>Tools</title>
309
310<para>Packages inside nixpkgs are written by hand. However many tools
311exist in community to help save time. No tool is preferred at the moment.
312</para>
313
314<itemizedlist>
315
316 <listitem><para>
317 <link xlink:href="https://github.com/proger/python2nix">python2nix</link>
318 by Vladimir Kirillov
319 </para></listitem>
320
321 <listitem><para>
322 <link xlink:href="https://github.com/garbas/pypi2nix">pypi2nix</link>
323 by Rok Garbas
324 </para></listitem>
325
326 <listitem><para>
327 <link xlink:href="https://github.com/offlinehacker/pypi2nix">pypi2nix</link>
328 by Jaka Hudoklin
329 </para></listitem>
330
331</itemizedlist>
332
333</section>
334
335<section xml:id="ssec-python-development"><title>Development</title>
336
337 <para>
338 To develop Python packages <function>buildPythonPackage</function> has
339 additional logic inside <varname>shellPhase</varname> to run
340 <command>pip install -e . --prefix $TMPDIR/</command> for the package.
341 </para>
342
343 <warning><para><varname>shellPhase</varname> is executed only if <filename>setup.py</filename>
344 exists.</para></warning>
345
346 <para>
347 Given a <filename>default.nix</filename>:
348
349<programlisting language="nix">
350<![CDATA[with import <nixpkgs> {};
351
352buildPythonPackage {
353 name = "myproject";
354
355 buildInputs = with pkgs.pythonPackages; [ pyramid ];
356
357 src = ./.;
358}]]>
359</programlisting>
360
361 Running <command>nix-shell</command> with no arguments should give you
362 the environment in which the package would be build with
363 <command>nix-build</command>.
364 </para>
365
366 <para>
367 Shortcut to setup environments with C headers/libraries and python packages:
368
369 <programlisting language="bash">$ nix-shell -p pythonPackages.pyramid zlib libjpeg git</programlisting>
370 </para>
371
372 <note><para>
373 There is a boolean value <varname>lib.inNixShell</varname> set to
374 <varname>true</varname> if nix-shell is invoked.
375 </para></note>
376
377</section>
378
379<section xml:id="ssec-python-faq"><title>FAQ</title>
380
381<variablelist>
382
383 <varlistentry>
384 <term>How to solve circular dependencies?</term>
385 <listitem><para>
386 If you have packages <varname>A</varname> and <varname>B</varname> that
387 depend on each other, when packaging <varname>B</varname> override package
388 <varname>A</varname> not to depend on <varname>B</varname> as input
389 (and also the other way around).
390 </para></listitem>
391 </varlistentry>
392
393 <varlistentry>
394 <term><varname>install_data / data_files</varname> problems resulting into <literal>error: could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc': Permission denied</literal></term>
395 <listitem><para>
396 <link xlink:href="https://bitbucket.org/pypa/setuptools/issue/130/install_data-doesnt-respect-prefix">
397 Known bug in setuptools <varname>install_data</varname> does not respect --prefix</link>. Example of
398 such package using the feature is <filename>pkgs/tools/X11/xpra/default.nix</filename>. As workaround
399 install it as an extra <varname>preInstall</varname> step:
400
401 <programlisting>${python.interpreter} setup.py install_data --install-dir=$out --root=$out
402sed -i '/ = data_files/d' setup.py</programlisting>
403 </para></listitem>
404 </varlistentry>
405
406 <varlistentry>
407 <term>Rationale of non-existent global site-packages</term>
408 <listitem><para>
409 There is no need to have global site-packages in Nix. Each package has isolated
410 dependency tree and installing any python package will only populate <varname>$PATH</varname>
411 inside user environment. See <xref linkend="ssec-python-build-env" /> to create self-contained
412 interpreter with a set of packages.
413 </para></listitem>
414 </varlistentry>
415
416</variablelist>
417
418</section>
419
420
421<section xml:id="ssec-python-contrib"><title>Contributing guidelines</title>
422<para>
423 Following rules are desired to be respected:
424</para>
425
426<itemizedlist>
427
428 <listitem><para>
429 Make sure package builds for all python interpreters. Use <varname>disabled</varname> argument to
430 <function>buildPythonPackage</function> to set unsupported interpreters.
431 </para></listitem>
432
433 <listitem><para>
434 If tests need to be disabled for a package, make sure you leave a comment about reasoning.
435 </para></listitem>
436
437 <listitem><para>
438 Packages in <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/python-packages.nix"><filename>pkgs/top-level/python-packages.nix</filename></link>
439 are sorted quasi-alphabetically to avoid merge conflicts.
440 </para></listitem>
441
442</itemizedlist>
443
444</section>
445
446</section>
447