tjh.dev
nixpkgs mirror (for testing)
github.com/NixOS/nixpkgs
nix
1
2# Maintainer Workflow
3
4The goal of the [@NixOS/haskell](https://github.com/orgs/NixOS/teams/haskell)
5team is to keep the Haskell packages in Nixpkgs up-to-date, while making sure
6there are no Haskell-related evaluation errors or build errors that get into
7the Nixpkgs `master` branch.
8
9We do this by periodically merging an updated set of Haskell packages on the
10`haskell-updates` branch into the `staging` branch. Each member of the team
11takes a two week period where they are in charge of merging the
12`haskell-updates` branch into `staging`. This is the documentation for this
13workflow.
14
15The workflow generally proceeds in three main steps:
16
171. Create the initial `haskell-updates` PR, and update Stackage and Hackage snapshots
181. Wait for contributors to fix newly broken Haskell packages
191. Merge `haskell-updates` into `staging`
20
21Each of these steps is described in a separate section.
22
23There is a script that automates the workflow for merging the currently open
24`haskell-updates` PR into `staging` and opening the next PR. It is described
25at the end of this document.
26
27## Initial `haskell-updates` PR
28
29In this section we create the PR for merging `haskell-updates` into `staging`.
30
311. Make sure the `haskell-updates` branch is up-to-date with a _merge base_ of
32 `staging` and `master`. `haskell-updates` is not based _on_ `staging`,
33 so that it can share binary cache with `master`.
34
351. Update the Stackage Nightly resolver used by Nixpkgs and create a commit:
36
37 ```console
38 $ ./maintainers/scripts/haskell/update-stackage.sh --do-commit
39 ```
40
411. Update the Hackage package set used by Nixpkgs and create a commit:
42
43 ```console
44 $ ./maintainers/scripts/haskell/update-hackage.sh --do-commit
45 ```
46
471. Regenerate the Haskell package set used in Nixpkgs and create a commit:
48
49 ```console
50 $ ./maintainers/scripts/haskell/regenerate-hackage-packages.sh --do-commit
51 ```
52
531. Push these commits to the `haskell-updates` branch of the NixOS/nixpkgs repository.
54
551. Open a PR on Nixpkgs for merging `haskell-updates` into `staging`. The recommended
56 PR title and body text are described in the `merge-and-open-pr.sh` section.
57
58## Notify Maintainers and Fix Broken Packages
59
60After you've done the previous steps, Hydra will start building the new and
61updated Haskell packages. You can see the progress Hydra is making at
62https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. This Hydra jobset is
63defined in the file [release-haskell.nix](../../top-level/release-haskell.nix).
64
65### Notify Maintainers
66
67When Hydra finishes building all the updated packages for the `haskell-updates`
68jobset, you should generate a build report to notify maintainers of their
69newly broken packages. You can do that with the following commands:
70
71```console
72$ ./maintainers/scripts/haskell/hydra-report.hs get-report
73$ ./maintainers/scripts/haskell/hydra-report.hs ping-maintainers
74```
75
76The `hydra-report.hs ping-maintainers` command generates a Markdown document
77that you can paste in a GitHub comment on the PR opened above. This
78comment describes which Haskell packages are now failing to build. It also
79pings the maintainers so that they know to fix up their packages.
80
81It may be helpful to pipe `hydra-report.hs ping-maintainers` into `xclip`
82(XOrg) or `wl-copy` (Wayland) in order to post on GitHub.
83
84This build report can be fetched and re-generated for new Hydra evaluations.
85It may help contributors to try to keep the GitHub comment updated with the
86most recent build report.
87
88Maintainers should be given at least 7 days to fix up their packages when they
89break. If maintainers don't fix up their packages within 7 days, then they
90may be marked broken before merging `haskell-updates` into `staging`.
91
92### Fix Broken Packages
93
94After getting the build report, you can see which packages and Hydra jobs are
95failing to build. The most important jobs are the
96[`maintained`](https://hydra.nixos.org/job/nixpkgs/haskell-updates/maintained) and
97[`mergeable`](https://hydra.nixos.org/job/nixpkgs/haskell-updates/mergeable)
98jobs. These are both defined in
99[`release-haskell.nix`](../../top-level/release-haskell.nix).
100
101`mergeable` is a set of the most important Haskell packages, including things
102like Pandoc and XMonad. These packages are widely used. We would like to
103always keep these building.
104
105`maintained` is a set of Haskell packages that have maintainers in Nixpkgs.
106We should be proactive in working with maintainers to keep their packages
107building.
108
109Steps to fix Haskell packages that are failing to build are out of scope for
110this document, but it usually requires fixing up dependencies that are now
111out-of-bounds.
112
113### Mark Broken Packages
114
115Packages that do not get fixed can be marked broken with the following
116commands. First check which packages are broken:
117
118```console
119$ ./maintainers/scripts/haskell/hydra-report.hs get-report
120$ ./maintainers/scripts/haskell/hydra-report.hs mark-broken-list
121```
122
123This shows a list of packages that reported a build failure on `x86_64-linux` on Hydra.
124
125Next, run the following command:
126
127```console
128$ ./maintainers/scripts/haskell/mark-broken.sh --do-commit
129```
130
131This first opens up an editor with the broken package list. Some of these
132packages may have a maintainer in Nixpkgs. If these maintainers have not been
133given 7 days to fix up their package, then make sure to remove those packages
134from the list before continuing. After saving and exiting the editor, the
135following will happen:
136
137- Packages from the list will be added to
138 [`configuration-hackage2nix/broken.yaml`](configuration-hackage2nix/broken.yaml).
139 This is a list of Haskell packages that are known to be broken.
140
141- [`hackage-packages.nix`](hackage-packages.nix) will be regenerated. This
142 will mark all Haskell packages in `configuration-hackage2nix/broken.yaml`
143 as `broken`.
144
145- The
146 [`configuration-hackage2nix/transitive-broken.yaml`](configuration-hackage2nix/transitive-broken.yaml)
147 file will be updated. This is a list of Haskell packages that
148 depend on a package in `configuration-hackage2nix/broken.yaml` or
149 `configuration-hackage2nix/transitive-broken.yaml`
150
151- `hackage-packages.nix` will be regenerated again. This will set
152 `hydraPlatforms = none` for all the packages in
153 `configuration-hackage2nix/transitive-broken.yaml`. This makes
154 sure that Hydra does not try to build any of these packages.
155
156- All updated files will be committed.
157
158## Merge `haskell-updates` into `staging`
159
160Now it is time to merge the `haskell-updates` PR you opened above.
161
162Before doing this, make sure of the following:
163
164- All Haskell packages that fail to build are correctly marked broken or
165 transitively broken.
166
167- The `maintained` and `mergeable` jobs are passing on Hydra.
168
169- The maintainers for any maintained Haskell packages that are newly broken
170 have been pinged on GitHub and given at least a week to fix their packages.
171 This is especially important for widely-used packages like `cachix`.
172
173- Keep an eye on the next `staging-next` iteration (which is branched off
174 from `staging`) to confirm that there are no show stopping issues stemming
175 from interactions between changes on `staging` and `haskell-updates`.
176 Also be aware that build or eval regressions from a `haskell-updates`
177 iteration may only become apparent on `staging-next`, especially when the
178 `haskell-updates` jobset had e.g. Darwin builds disabled.
179
180## Script for Merging `haskell-updates` and Opening a New PR
181
182There is a script that automates merging the current `haskell-updates` PR and
183opening the next one. When you want to merge the currently open
184`haskell-updates` PR, you can run the script with the following steps:
185
1861. Make sure you have previously authenticated with the `gh` command. The
187 script uses the `gh` command to merge the current PR and open a new one.
188 You should only need to do this once.
189
190 This command can be used to authenticate:
191
192 ```console
193 $ gh auth login
194 ```
195
196 This command can be used to confirm that you have already authenticated:
197
198 ```console
199 $ gh auth status
200 ```
201
2021. Make sure you have correctly marked packages broken. One of the previous
203 sections explains how to do this.
204
205 In short:
206
207 ```console
208 $ ./maintainers/scripts/haskell/hydra-report.hs get-report
209 $ ./maintainers/scripts/haskell/hydra-report.hs mark-broken-list
210 $ ./maintainers/scripts/haskell/mark-broken.sh --do-commit
211 ```
212
2131. Go to https://hydra.nixos.org/jobset/nixpkgs/haskell-updates and force an
214 evaluation of the `haskell-updates` jobset. See one of the following
215 sections for how to do this. Make sure there are no evaluation errors. If
216 there are remaining evaluation errors, fix them before continuing with this
217 merge.
218
2191. Run the script to merge `haskell-updates`:
220
221 ```console
222 $ ./maintainers/scripts/haskell/merge-and-open-pr.sh PR_NUM_OF_CURRENT_HASKELL_UPDATES_PR
223 ```
224
225 Find the PR number easily [here](https://github.com/nixos/nixpkgs/pulls?q=is%3Apr+is%3Aopen+head%3Ahaskell-updates)
226
227 This does the following things:
228
229 1. Fetches `origin`, makes sure you currently have the `haskell-updates`
230 branch checked out, and makes sure your currently checked-out
231 `haskell-updates` branch is on the same commit as
232 `origin/haskell-updates`.
233
234 1. Merges the currently open `haskell-updates` PR.
235
236 1. Updates Stackage and Hackage snapshots. Regenerates the Haskell package set.
237
238 1. Pushes the commits updating Stackage and Hackage and opens a new
239 `haskell-updates` PR on Nixpkgs. If you'd like to do this by hand,
240 look in the script for the recommended PR title and body text.
241
242## Update Hackage Version Information
243
244Remember to regularly update what Hackage displays as the current
245version in NixOS for every individual package. To do this you run
246`maintainers/scripts/haskell/upload-nixos-package-list-to-hackage.sh` on a checkout
247of `master` (or `nixpkgs-unstable`). See the script for how to provide credentials.
248Once you have configured credentials, running this takes only a few seconds.
249
250The best time to do this is after `staging-next` has been merged since this is
251the way Haskell package updates propagate to `master`.
252
253## Additional Info
254
255Here are some additional tips that didn't fit in above.
256
257- Hydra tries to evaluate the `haskell-updates` branch (in the
258 [`nixpkgs:haskell-updates`](https://hydra.nixos.org/jobset/nixpkgs/haskell-updates)
259 jobset) every 4 hours. It is possible to force a new Hydra evaluation without
260 waiting 4 hours by the following steps:
261
262 1. Log into Hydra with your GitHub or Google account.
263 1. Go to the [nixpkgs:haskell-updates](https://hydra.nixos.org/jobset/nixpkgs/haskell-updates) jobset.
264 1. Click the `Actions` button.
265 1. Select `Evaluate this jobset`.
266 1. If you refresh the page, there should be a new `Evaluation running since:` line.
267 1. Evaluations take about 10 minutes to finish.
268
269- It is sometimes helpful to update the version of
270 [`cabal2nix` / `hackage2nix`](https://github.com/NixOS/cabal2nix) that our
271 maintainer scripts use. This can be done with the
272 [`maintainers/scripts/haskell/update-cabal2nix-unstable.sh`](../../../maintainers/scripts/haskell/update-cabal2nix-unstable.sh)
273 script.
274
275 You might want to do this if a user contributes a fix to `cabal2nix` that
276 will immediately fix a Haskell package in Nixpkgs. First, merge in
277 the PR to `cabal2nix`, then run `update-cabal2nix-unstable.sh`. Finally, run
278 [`regenerate-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerate-hackage-packages.sh)
279 to regenerate the Hackage package set with the updated version of `hackage2nix`.
280
281- Make sure never to update the Hackage package hashes in
282 [`pkgs/data/misc/hackage/`](../../../pkgs/data/misc/hackage/), or the
283 pinned Stackage Nightly versions on the release branches (like
284 `release-21.05`).
285
286 This means that the
287 [`update-hackage.sh`](../../../maintainers/scripts/haskell/update-hackage.sh)
288 and
289 [`update-stackage.sh`](../../../maintainers/scripts/haskell/update-stackage.sh)
290 scripts should never be used on the release branches.
291
292 However, changing other files in `./.` and regenerating the package set is encouraged.
293 This can be done with
294 [`regenerate-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerate-hackage-packages.sh)
295 as described above.
296
297- The Haskell team members generally hang out in the Matrix room
298 [#haskell:nixos.org](https://matrix.to/#/#haskell:nixos.org).
299
300- This is a checklist for things that need to happen when a new
301 member is added to the Nixpkgs Haskell team.
302
303 1. Add the person to the
304 [@NixOS/haskell](https://github.com/orgs/NixOS/teams/haskell)
305 team. You may need to ask someone in the NixOS organization
306 to do this, like [@domenkozar](https://github.com/domenkozar).
307 This gives the new member access to the GitHub repos like
308 [cabal2nix](https://github.com/NixOS/cabal2nix).
309
310 1. Add the person as a maintainer for the following packages
311 on Hackage:
312 - https://hackage.haskell.org/package/cabal2nix
313 - https://hackage.haskell.org/package/distribution-nixpkgs
314 - https://hackage.haskell.org/package/hackage-db
315 - https://hackage.haskell.org/package/jailbreak-cabal
316 - https://hackage.haskell.org/package/language-nix
317
318 1. Add the person to the `haskell` team in
319 [`maintainers/team-list.nix`](../../../maintainers/team-list.nix).
320 This team is responsible for some important packages in
321 [release-haskell.nix](../../top-level/release-haskell.nix).
322
323 1. Update the
324 [Nextcloud Calendar](https://cloud.maralorn.de/apps/calendar/p/H6migHmKX7xHoTFa)
325 and work the new member into the `haskell-updates` rotation.
326
327 1. Optionally, have the new member add themselves to the Haskell
328 section in [`OWNERS`](../../../ci/OWNERS). This
329 will cause them to get pinged on most Haskell-related PRs.