···11+22+## Maintainer Workflow
33+44+The goal of the [@NixOS/haskell](https://github.com/orgs/NixOS/teams/haskell)
55+team is to keep the Haskell packages in Nixpkgs up-to-date, while making sure
66+there are no Haskell-related evaluation errors or build errors that get into
77+the Nixpkgs `master` branch.
88+99+We do this by periodically merging an updated set of Haskell packages on the
1010+`haskell-updates` branch into the `master` branch. Each member of the team
1111+takes a two week period where they are in charge of merging the
1212+`haskell-updates` branch into `master`. This is the documentation for this
1313+workflow.
1414+1515+The workflow generally proceeds in three main steps:
1616+1717+1. create the initial `haskell-updates` PR, and update Stackage and Hackage snapshots
1818+1. wait for contributors to fix newly broken Haskell packages
1919+1. merge `haskell-updates` into `master`
2020+2121+Each of these steps is described in a separate section.
2222+2323+### Initial `haskell-updates` PR
2424+2525+In this section we create the PR for merging `haskell-updates` into `master`.
2626+2727+1. Make sure the `haskell-updates` branch is up-to-date with `master`.
2828+2929+1. Update the Stackage Nightly resolver used by Nixpkgs and create a commit:
3030+3131+ ```console
3232+ $ ./maintainers/scripts/haskell/update-stackage.sh --do-commit
3333+ ```
3434+3535+1. Update the Hackage package set used by Nixpkgs and create a commit:
3636+3737+ ```console
3838+ $ ./maintainers/scripts/haskell/update-hackage.sh --do-commit
3939+ ```
4040+4141+1. Regenerate the Haskell package set used in Nixpkgs and create a commit:
4242+4343+ ```console
4444+ $ ./maintainers/scripts/haskell/regenerate-hackage-packages.sh --do-commit
4545+ ```
4646+4747+1. Push these commits to the Nixpkgs repository.
4848+4949+1. Open a PR on Nixpkgs merging `haskell-updates` into `master`.
5050+5151+Use the following message body:
5252+5353+```markdown
5454+### This Merge
5555+5656+This PR is the regular merge of the `haskell-updates` branch into `master`.
5757+5858+This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates.
5959+6060+I will aim to merge this PR **by 2021-TODO-TODO**. If I can merge it earlier, there might be successor PRs in that time window. As part of our rotation @TODO will continue these merges from 2021-TODO-TODO to 2021-TODO-TODO.
6161+6262+### haskellPackages Workflow Summary
6363+6464+Our workflow is currently described in
6565+[`pkgs/development/haskell-modules/HACKING.md`](https://github.com/NixOS/nixpkgs/blob/haskell-updates/pkgs/development/haskell-modules/HACKING.md).
6666+6767+The short version is this:
6868+* We regularly update the Stackage and Hackage pins on `haskell-updates` (normally at the beginning of a merge window).
6969+* The community fixes builds of Haskell packages on that branch.
7070+* We aim at at least one merge of `haskell-updates` into `master` every two weeks.
7171+* We only do the merge if the `mergeable` job is succeeding on hydra.
7272+* If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!)
7373+7474+---
7575+7676+This is the follow-up to #TODO.
7777+```
7878+7979+Make sure to replace all TODO with the actual values.
8080+8181+### Notify Maintainers and Fix Broken Packages
8282+8383+After you've done the previous steps, Hydra will start building the new and
8484+updated Haskell packages. You can see the progress Hydra is making at
8585+https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. This Hydra jobset is
8686+defined in the file [release-haskell.nix](../../top-level/release-haskell.nix).
8787+8888+#### Notify Maintainers
8989+9090+When Hydra finishes building all the updated packages for the `haskell-updates`
9191+jobset, you should generate a build report to notify maintainers of their
9292+newly broken packages. You can do that with the following commands:
9393+9494+```console
9595+$ ./maintainers/scripts/haskell/hydra-report.hs get-report
9696+$ ./maintainers/scripts/haskell/hydra-report.hs ping-maintainers
9797+```
9898+9999+The `hyda-report.hs ping-maintainers` command generates a Markdown document
100100+that you can paste in a GitHub comment on the PR opened above. This
101101+comment describes which Haskell packages are now failing to build. It also
102102+pings the maintainers so that they know to fix up their packages.
103103+104104+It may be helpful to pipe `hydra-report.hs ping-maintainers` into `xclip`
105105+(XOrg) or `wl-copy` (Wayland) in order to post on GitHub.
106106+107107+This build report can be fetched and re-generated for new Hydra evaluations.
108108+It may help contributors to try to keep the GitHub comment updated with the
109109+most recent build report.
110110+111111+Maintainers should be given at least 7 days to fix up their packages when they
112112+break. If maintainers don't fix up their packages with 7 days, then they
113113+may be marked broken before merging `haskell-updates` into `master`.
114114+115115+#### Fix Broken Packages
116116+117117+After getting the build report, you can see which packages and Hydra jobs are
118118+failing to build. The most important jobs are the `maintained` and `mergeable`
119119+jobs. These are both defined in
120120+[`release-haskell.nix`](../../top-level/release-haskell.nix).
121121+122122+`mergeable` is a set of the most important Haskell packages, including things
123123+like Pandoc and XMonad. These packages are widely used. We would like to
124124+always keep these building.
125125+126126+`maintained` is a set of Haskell packages that have maintainers in Nixpkgs.
127127+We should be proactive in working with maintainers to keep their packages
128128+building.
129129+130130+Steps to fix Haskell packages that are failing to build is out of scope for
131131+this document, but it usually requires fixing up dependencies that are now
132132+out-of-bounds.
133133+134134+#### Mark Broken Packages
135135+136136+Packages that do not get fixed can be marked broken with the following
137137+commands. First check which packages are broken:
138138+139139+```console
140140+$ ./maintainers/scripts/haskell/hydra-report.hs get-report
141141+$ ./maintainers/scripts/haskell/hydra-report.hs mark-broken-list
142142+```
143143+144144+This shows a list of packages that reported a build failure on `x86_64-linux` on Hydra.
145145+146146+Next, run the following command:
147147+148148+```console
149149+$ ./maintainers/scripts/haskell/mark-broken.sh --do-commit
150150+```
151151+152152+This first opens up an editor with the broken package list. Some of these
153153+packages may have a maintainer in Nixpkgs. If these maintainers have not been
154154+given 7 days to fix up their package, then make sure to remove those packages
155155+from the list before continuing. After saving and exiting the editor, the
156156+following will happen:
157157+158158+- Packages from the list will be added to
159159+ [`configuration-hackage2nix/broken.yaml`](configuration-hackage2nix/broken.yaml).
160160+ This is a list of Haskell packages that are known to be broken.
161161+162162+- [`hackage-packages.nix`](hackage-packages.nix) will be regenerated. This
163163+ will mark all Haskell pacakges in `configuration-hackage2nix/broken.yaml`
164164+ as `broken`.
165165+166166+- The
167167+ [`configuration-hackage2nix/transitive-broken.yaml`](configuration-hackage2nix/transitive-broken.yaml)
168168+ file will be updated. This is a list of Haskell packages that
169169+ depend on a package in `configuration-hackage2nix/broken.yaml` or
170170+ `configuration-hackage2nix/transitive-broken.yaml`
171171+172172+- `hackage-packages.nix` will be regenerated again. This will set
173173+ `hydraPlatforms = none` for all the packages in
174174+ `configuration-hackage2nix/transitive-broken.yaml`. This makes
175175+ sure that Hydra does not try to build any of these packages.
176176+177177+- All updated files will be committed.
178178+179179+#### Merge `master` into `haskell-updates`
180180+181181+You should occasionally merge the `master` branch into the `haskell-updates`
182182+branch.
183183+184184+In an ideal world, when we merge `haskell-updates` into `master`, it would
185185+cause few Hydra rebuilds on `master`. Ideally, the `nixos-unstable` channel
186186+would never be prevented from progressing because of needing to wait for
187187+rebuilding Haskell packages.
188188+189189+In order to make sure that there are a minimal number of rebuilds after merging
190190+`haskell-updates` into `master`, `master` should occasionally be merged into
191191+the `haskell-updates` branch.
192192+193193+This is especially important after `staging-next` is merged into `master`,
194194+since there is a high chance that this will cause all the Haskell packages to
195195+rebuild.
196196+197197+### Merge `haskell-updates` into `master`
198198+199199+Now it is time to merge the `haskell-updates` PR you opened above.
200200+201201+Before doing this, make sure of the following:
202202+203203+- All Haskell packages that fail to build are correctly marked broken or
204204+ transitively broken.
205205+206206+- The `maintained` and `mergeable` jobs are passing on Hydra.
207207+208208+- The maintainers for any maintained Haskell packages that are newly broken
209209+ have been pinged on GitHub and given at least a week to fix their packages.
210210+ This is especially important for widely-used packages like `cachix`.
211211+212212+- Make sure you first merge the `master` branch into `haskell-updates`. Wait
213213+ for Hydra to evaluate the new `haskell-updates` jobset. Make sure you only
214214+ merge `haskell-updates` into `master` when there are no evaluation errors.
215215+216216+When you've double-checked these points, go ahead and merge the `haskell-updates` PR.
217217+After merging, **make sure not to delete the `haskell-updates` branch**, since it
218218+causes all currently open Haskell-related pull-requests to be automatically closed on GitHub.
219219+220220+### Additional Info
221221+222222+Here are some additional tips that didn't fit in above.
223223+224224+- Hydra tries to evalute the `haskell-updates` branch (in the
225225+ [`nixpkgs:haskell-updates`](https://hydra.nixos.org/jobset/nixpkgs/haskell-updates)
226226+ jobset) every 4 hours. It is possible to force a new Hydra evaluation without
227227+ waiting 4 hours by the following steps:
228228+229229+ 1. Log into Hydra with your GitHub or Google account.
230230+ 1. Go to the [nixpkgs:haskell-updates](https://hydra.nixos.org/jobset/nixpkgs/haskell-updates) jobset.
231231+ 1. Click the `Actions` button.
232232+ 1. Select `Evaluate this jobset`.
233233+ 1. If you refresh the page, there should be a new `Evaluation running since:` line.
234234+ 1. Evaluations take about 10 minutes to finish.
235235+236236+- It is sometimes helpful to update the version of
237237+ [`cabal2nix` / `hackage2nix`](https://github.com/NixOS/cabal2nix) that our
238238+ maintainer scripts use. This can be done with the
239239+ [`maintainers/scripts/haskell/update-cabal2nix-unstable.sh`](../../../maintainers/scripts/haskell/update-cabal2nix-unstable.sh)
240240+ script.
241241+242242+ You might want to do this if a user contributes a fix to `cabal2nix` that
243243+ will immediately fix a Haskell package in Nixpkgs. First, merge in
244244+ the PR to `cabal2nix`, then run `update-cabal2nix-upstable.sh`. Finally, run
245245+ [`regenerate-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerate-hackage-packages.sh)
246246+ to regenerate the Hackage package set with the updated version of `hackage2nix`.
247247+248248+- Make sure never to update the Hackage package hashes in
249249+ [`pkgs/data/misc/hackage/`](../../../pkgs/data/misc/hackage/), or the
250250+ pinned Stackage Nightly versions on the release branches (like
251251+ `release-21.05`).
252252+253253+ This means that the
254254+ [`update-hackage.sh`](../../../maintainers/scripts/haskell/update-hackage.sh)
255255+ and
256256+ [`update-stackage.sh`](../../../maintainers/scripts/haskell/update-stackage.sh)
257257+ scripts should never be used on the release branches.
258258+259259+ However, changing other files in `./.` and regenerating the package set is encouraged.
260260+ This can be done with
261261+ [`regenerate-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerate-hackage-packages.sh)
262262+ as described above.
263263+264264+- The Haskell team members generally hang out in the Matrix room
265265+ [#haskell:nixos.org](https://matrix.to/#/#haskell:nixos.org).
266266+267267+## Contributor Workflow
268268+269269+(TODO: this section is to describe the type of workflow for non-committers to
270270+contribute to `haskell-updates`)
···168168 '' + (drv.postPatch or "");
169169 });
170170171171+ # 2021-05-25: Tests fail and I have no way to debug them.
172172+ hls-class-plugin = dontCheck super.hls-class-plugin;
173173+171174}
···196196 tcp-streams = dontCheck super.tcp-streams;
197197 holy-project = dontCheck super.holy-project;
198198 mustache = dontCheck super.mustache;
199199+ arch-web = dontCheck super.arch-web;
199200200201 # Tries to mess with extended POSIX attributes, but can't in our chroot environment.
201202 xattr = dontCheck super.xattr;
···259260 '';
260261 }));
261262262262- # Patch to consider NIX_GHC just like xmonad does
263263- dyre = appendPatch super.dyre ./patches/dyre-nix.patch;
263263+ dyre =
264264+ appendPatch
265265+ # dyre's tests appear to be trying to directly call GHC.
266266+ (dontCheck super.dyre)
267267+ # Dyre needs special support for reading the NIX_GHC env var. This is
268268+ # available upstream in https://github.com/willdonnelly/dyre/pull/43, but
269269+ # hasn't been released to Hackage as of dyre-0.9.1. Likely included in
270270+ # next version.
271271+ (pkgs.fetchpatch {
272272+ url = "https://github.com/willdonnelly/dyre/commit/c7f29d321aae343d6b314f058812dffcba9d7133.patch";
273273+ sha256 = "10m22k35bi6cci798vjpy4c2l08lq5nmmj24iwp0aflvmjdgscdb";
274274+ });
264275265276 # https://github.com/edwinb/EpiVM/issues/13
266277 # https://github.com/edwinb/EpiVM/issues/14
···803814 hw-prim-bits = overrideCabal super.hw-prim-bits {
804815 platforms = pkgs.lib.platforms.x86;
805816 };
817817+818818+ # random 1.2.0 has tests that indirectly depend on
819819+ # itself causing an infinite recursion at evaluation
820820+ # time
821821+ random = dontCheck super.random;
806822}