lol
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

nixpkgs-vet: update CI, docs, and release to 0.1.4

Everything gets moved into the `ci/` top-level directory.

We keep behind `maintainers/scripts/check-by-name.sh` and `pkgs/test/check-by-name/pinned-version.txt` as they are going to cause CI errors and confusion until we get all the way through the various channels.
They'll be removed in about a week or so.

+161 -180
+5 -8
.github/CODEOWNERS
··· 14 14 # CI 15 15 /.github/workflows @NixOS/Security @Mic92 @zowoq 16 16 /.github/workflows/check-nix-format.yml @infinisil 17 - /ci @infinisil @NixOS/Security 17 + /.github/workflows/nixpkgs-vet.yml @infinisil @philiptaron 18 + /ci @infinisil @philiptaron @NixOS/Security 18 19 19 - # Develompent support 20 + # Development support 20 21 /.editorconfig @Mic92 @zowoq 21 22 /shell.nix @infinisil @NixOS/Security 22 23 ··· 43 44 /pkgs/top-level/stage.nix @Ericson2314 44 45 /pkgs/top-level/splice.nix @Ericson2314 45 46 /pkgs/top-level/release-cross.nix @Ericson2314 47 + /pkgs/top-level/by-name-overlay.nix @infinisil @philiptaron 46 48 /pkgs/stdenv @philiptaron 47 49 /pkgs/stdenv/generic @Ericson2314 48 50 /pkgs/stdenv/generic/check-meta.nix @Ericson2314 ··· 58 60 /pkgs/pkgs-lib/formats/libconfig @h7x4 59 61 /pkgs/pkgs-lib/formats/hocon @h7x4 60 62 61 - # pkgs/by-name 62 - /pkgs/test/check-by-name @infinisil 63 - /pkgs/by-name/README.md @infinisil 64 - /pkgs/top-level/by-name-overlay.nix @infinisil 65 - /.github/workflows/check-by-name.yml @infinisil 66 - 67 63 # Nixpkgs build-support 68 64 /pkgs/build-support/writers @lassulus @Profpatsch 69 65 ··· 91 87 /doc/README.md @infinisil 92 88 /nixos/README.md @infinisil 93 89 /pkgs/README.md @infinisil 90 + /pkgs/by-name/README.md @infinisil 94 91 /maintainers/README.md @infinisil 95 92 96 93 # User-facing development documentation
+25 -36
.github/workflows/check-by-name.yml .github/workflows/nixpkgs-vet.yml
··· 1 - # Checks pkgs/by-name (see pkgs/by-name/README.md) 2 - # using the nixpkgs-check-by-name tool (see https://github.com/NixOS/nixpkgs-check-by-name) 3 - # 4 - # When you make changes to this workflow, also update pkgs/test/check-by-name/run-local.sh adequately 1 + # Checks pkgs/by-name (see pkgs/by-name/README.md) using the `nixpkgs-vet` tool (see https://github.com/NixOS/nixpkgs-vet) 2 + # When you make changes to this workflow, please also update `ci/nixpkgs-vet.sh` to reflect the impact of your work to the CI. 5 3 name: Check pkgs/by-name 6 4 7 5 on: 8 - # Using pull_request_target instead of pull_request avoids having to approve first time contributors 6 + # Using pull_request_target instead of pull_request avoids having to approve first time contributors. 9 7 pull_request_target: 10 - # This workflow depends on the base branch of the PR, 11 - # but changing the base branch is not included in the default trigger events, 12 - # which would be `opened`, `synchronize` or `reopened`. 13 - # Instead it causes an `edited` event, so we need to add it explicitly here 14 - # While `edited` is also triggered when the PR title/body is changed, 15 - # this PR action is fairly quick, and PR's don't get edited that often, 16 - # so it shouldn't be a problem 17 - # There is a feature request for adding a `base_changed` event: 18 - # https://github.com/orgs/community/discussions/35058 8 + # This workflow depends on the base branch of the PR, but changing the base branch is not included in the default trigger events, which would be `opened`, `synchronize` or `reopened`. 9 + # Instead it causes an `edited` event, so we need to add it explicitly here. 10 + # While `edited` is also triggered when the PR title/body is changed, this PR action is fairly quick, and PRs don't get edited **that** often, so it shouldn't be a problem. 11 + # There is a feature request for adding a `base_changed` event: https://github.com/orgs/community/discussions/35058 19 12 types: [opened, synchronize, reopened, edited] 20 13 21 14 permissions: {} 22 15 23 - # We don't use a concurrency group here, because the action is triggered quite often (due to the PR edit 24 - # trigger), and contributers would get notified on any canceled run. 25 - # There is a feature request for supressing notifications on concurrency-canceled runs: 26 - # https://github.com/orgs/community/discussions/13015 16 + # We don't use a concurrency group here, because the action is triggered quite often (due to the PR edit trigger), and contributors would get notified on any canceled run. 17 + # There is a feature request for suppressing notifications on concurrency-canceled runs: https://github.com/orgs/community/discussions/13015 27 18 28 19 jobs: 29 20 check: 30 21 name: pkgs-by-name-check 31 - # This needs to be x86_64-linux, because we depend on the tooling being pre-built in the GitHub releases 22 + # This needs to be x86_64-linux, because we depend on the tooling being pre-built in the GitHub releases. 32 23 runs-on: ubuntu-latest 33 - # This should take 1 minute at most, but let's be generous. 34 - # The default of 6 hours is definitely too long 24 + # This should take 1 minute at most, but let's be generous. The default of 6 hours is definitely too long. 35 25 timeout-minutes: 10 36 26 steps: 37 - # This step has to be in this file, 38 - # because it's needed to determine which revision of the repository to fetch, 39 - # and we can only use other files from the repository once it's fetched. 27 + # This step has to be in this file, because it's needed to determine which revision of the repository to fetch, and we can only use other files from the repository once it's fetched. 40 28 - name: Resolving the merge commit 41 29 env: 42 30 GH_TOKEN: ${{ github.token }} ··· 99 87 if: env.mergedSha 100 88 - name: Fetching the pinned tool 101 89 if: env.mergedSha 102 - # Update the pinned version using pkgs/test/check-by-name/update-pinned-tool.sh 90 + # Update the pinned version using ci/nixpkgs-vet/update-pinned-tool.sh 103 91 run: | 104 - # The pinned version of the tooling to use 105 - toolVersion=$(<pkgs/test/check-by-name/pinned-version.txt) 106 - # Fetch the x86_64-linux-specific release artifact containing the Gzipped NAR of the pre-built tool 107 - toolPath=$(curl -sSfL https://github.com/NixOS/nixpkgs-check-by-name/releases/download/"$toolVersion"/x86_64-linux.nar.gz \ 92 + # The pinned version of the tooling to use. 93 + toolVersion=$(<ci/nixpkgs-vet/pinned-version.txt) 94 + 95 + # Fetch the x86_64-linux-specific release artifact containing the gzipped NAR of the pre-built tool. 96 + toolPath=$(curl -sSfL https://github.com/NixOS/nixpkgs-vet/releases/download/"$toolVersion"/x86_64-linux.nar.gz \ 108 97 | gzip -cd | nix-store --import | tail -1) 109 - # Adds a result symlink as a GC root 98 + 99 + # Adds a result symlink as a GC root. 110 100 nix-store --realise "$toolPath" --add-root result 111 - - name: Running nixpkgs-check-by-name 101 + - name: Running nixpkgs-vet 112 102 if: env.mergedSha 113 103 env: 114 - # Force terminal colors to be enabled. The library that 115 - # nixpkgs-check-by-name uses respects: https://bixense.com/clicolors/ 104 + # Force terminal colors to be enabled. The library that `nixpkgs-vet` uses respects https://bixense.com/clicolors/ 116 105 CLICOLOR_FORCE: 1 117 106 run: | 118 - if result/bin/nixpkgs-check-by-name --base "$base" .; then 107 + if result/bin/nixpkgs-vet --base "$base" .; then 119 108 exit 0 120 109 else 121 110 exitCode=$? 122 - echo "To run locally: ./maintainers/scripts/check-by-name.sh $GITHUB_BASE_REF https://github.com/$GITHUB_REPOSITORY.git" 123 - echo "If you're having trouble, ping @NixOS/nixpkgs-check-by-name" 111 + echo "To run locally: ./ci/nixpkgs-vet.sh $GITHUB_BASE_REF https://github.com/$GITHUB_REPOSITORY.git" 112 + echo "If you're having trouble, ping @NixOS/nixpkgs-vet" 124 113 exit "$exitCode" 125 114 fi
+1 -1
.github/workflows/check-nix-format.yml
··· 7 7 8 8 on: 9 9 pull_request_target: 10 - # See the comment at the same location in ./check-by-name.yml 10 + # See the comment at the same location in ./nixpkgs-vet.yml 11 11 types: [opened, synchronize, reopened, edited] 12 12 permissions: 13 13 contents: read
+31
ci/README.md
··· 10 10 [`pinned-nixpkgs.json`](./pinned-nixpkgs.json) contains a pinned Nixpkgs version tested by Hydra. 11 11 12 12 Run [`update-pinned-nixpkgs.sh`](./update-pinned-nixpkgs.sh) to update it. 13 + 14 + ## `ci/nixpkgs-vet.sh BASE_BRANCH [REPOSITORY]` 15 + 16 + Runs the [`nixpkgs-vet` tool](https://github.com/NixOS/nixpkgs-vet) on the HEAD commit, closely matching what CI does. This can't do exactly the same as CI, because CI needs to rely on GitHub's server-side Git history to compute the mergeability of PRs before the check can be started. 17 + In turn, when contributors are running this tool locally, we don't want to have to push commits to test them, and we can also rely on the local Git history to do the mergeability check. 18 + 19 + Arguments: 20 + 21 + - `BASE_BRANCH`: The base branch to use, e.g. master or release-24.05 22 + - `REPOSITORY`: The repository from which to fetch the base branch. Defaults to <https://github.com/NixOS/nixpkgs.git>. 23 + 24 + ## `ci/nixpkgs-vet` 25 + 26 + This directory contains scripts and files used and related to [`nixpkgs-vet`](https://github.com/NixOS/nixpkgs-vet/), which the CI uses to implement `pkgs/by-name` checks, along with many other Nixpkgs architecture rules. 27 + See also the [CI GitHub Action](../.github/workflows/nixpkgs-vet.yml). 28 + 29 + ## `ci/nixpkgs-vet/update-pinned-tool.sh` 30 + 31 + Updates the pinned [`nixpkgs-vet` tool](https://github.com/NixOS/nixpkgs-vet) in [`ci/nixpkgs-vet/pinned-version.txt`](./nixpkgs-vet/pinned-version.txt) to the latest [release](https://github.com/NixOS/nixpkgs-vet/releases). 32 + 33 + Each release contains a pre-built `x86_64-linux` version of the tool which is used by CI. 34 + 35 + This script currently needs to be called manually when the CI tooling needs to be updated. 36 + 37 + Why not just build the tooling right from the PRs Nixpkgs version? 38 + 39 + - Because it allows CI to check all PRs, even if they would break the CI tooling. 40 + - Because it makes the CI check very fast, since no Nix builds need to be done, even for mass rebuilds. 41 + - Because it improves security, since we don't have to build potentially untrusted code from PRs. 42 + The tool only needs a very minimal Nix evaluation at runtime, which can work with [readonly-mode](https://nixos.org/manual/nix/stable/command-ref/opt-common.html#opt-readonly-mode) and [restrict-eval](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-restrict-eval). 43 +
+71
ci/nixpkgs-vet.sh
··· 1 + #!/usr/bin/env nix-shell 2 + #!nix-shell -i bash -p jq 3 + 4 + set -o pipefail -o errexit -o nounset 5 + 6 + trace() { echo >&2 "$@"; } 7 + 8 + tmp=$(mktemp -d) 9 + cleanup() { 10 + # Don't exit early if anything fails to cleanup 11 + set +o errexit 12 + 13 + trace -n "Cleaning up.. " 14 + 15 + [[ -e "$tmp/base" ]] && git worktree remove --force "$tmp/base" 16 + [[ -e "$tmp/merged" ]] && git worktree remove --force "$tmp/merged" 17 + 18 + rm -rf "$tmp" 19 + 20 + trace "Done" 21 + } 22 + trap cleanup exit 23 + 24 + 25 + repo=https://github.com/NixOS/nixpkgs.git 26 + 27 + if (( $# != 0 )); then 28 + baseBranch=$1 29 + shift 30 + else 31 + trace "Usage: $0 BASE_BRANCH [REPOSITORY]" 32 + trace "BASE_BRANCH: The base branch to use, e.g. master or release-23.11" 33 + trace "REPOSITORY: The repository to fetch the base branch from, defaults to $repo" 34 + exit 1 35 + fi 36 + 37 + if (( $# != 0 )); then 38 + repo=$1 39 + shift 40 + fi 41 + 42 + if [[ -n "$(git status --porcelain)" ]]; then 43 + trace -e "\e[33mWarning: Dirty tree, uncommitted changes won't be taken into account\e[0m" 44 + fi 45 + headSha=$(git rev-parse HEAD) 46 + trace -e "Using HEAD commit \e[34m$headSha\e[0m" 47 + 48 + trace -n "Creating Git worktree for the HEAD commit in $tmp/merged.. " 49 + git worktree add --detach -q "$tmp/merged" HEAD 50 + trace "Done" 51 + 52 + trace -n "Fetching base branch $baseBranch to compare against.. " 53 + git fetch -q "$repo" refs/heads/"$baseBranch" 54 + baseSha=$(git rev-parse FETCH_HEAD) 55 + trace -e "\e[34m$baseSha\e[0m" 56 + 57 + trace -n "Creating Git worktree for the base branch in $tmp/base.. " 58 + git worktree add -q "$tmp/base" "$baseSha" 59 + trace "Done" 60 + 61 + trace -n "Merging base branch into the HEAD commit in $tmp/merged.. " 62 + git -C "$tmp/merged" merge -q --no-edit "$baseSha" 63 + trace -e "\e[34m$(git -C "$tmp/merged" rev-parse HEAD)\e[0m" 64 + trace -n "Reading pinned nixpkgs-vet version from pinned-version.txt.. " 65 + toolVersion=$(<"$tmp/merged/ci/nixpkgs-vet/pinned-version.txt") 66 + trace -e "\e[34m$toolVersion\e[0m" 67 + 68 + trace -n "Building tool.. " 69 + nix-build https://github.com/NixOS/nixpkgs-vet/tarball/"$toolVersion" -o "$tmp/tool" -A build 70 + trace "Running nixpkgs-vet.." 71 + "$tmp/tool/bin/nixpkgs-vet" --base "$tmp/base" "$tmp/merged"
+1
ci/nixpkgs-vet/pinned-version.txt
··· 1 + 0.1.4
+22
ci/nixpkgs-vet/update-pinned-tool.sh
··· 1 + #!/usr/bin/env nix-shell 2 + #!nix-shell -i bash -p jq curl 3 + 4 + set -o pipefail -o errexit -o nounset 5 + 6 + trace() { echo >&2 "$@"; } 7 + 8 + SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) 9 + 10 + repository=NixOS/nixpkgs-vet 11 + pin_file=$SCRIPT_DIR/pinned-version.txt 12 + 13 + trace -n "Fetching latest release of $repository.. " 14 + latestRelease=$(curl -sSfL \ 15 + -H "Accept: application/vnd.github+json" \ 16 + -H "X-GitHub-Api-Version: 2022-11-28" \ 17 + https://api.github.com/repos/"$repository"/releases/latest) 18 + latestVersion=$(jq .tag_name -r <<< "$latestRelease") 19 + trace "$latestVersion" 20 + 21 + trace "Updating $pin_file" 22 + echo "$latestVersion" > "$pin_file"
-4
maintainers/scripts/README.md
··· 9 9 10 10 ## Metadata 11 11 12 - ### `check-by-name.sh` 13 - 14 - An alias for `pkgs/test/check-by-name/run-local.sh`, see [documentation](../../pkgs/test/check-by-name/README.md). 15 - 16 12 ### `get-maintainer.sh` 17 13 18 14 `get-maintainer.sh [selector] value` returns a JSON object describing
+1 -1
maintainers/scripts/check-by-name.sh
··· 1 - ../../pkgs/test/check-by-name/run-local.sh 1 + ../../ci/nixpkgs-vet.sh
+4 -4
pkgs/by-name/README.md
··· 110 110 111 111 ## Validation 112 112 113 - CI performs [certain checks](https://github.com/NixOS/nixpkgs-check-by-name?tab=readme-ov-file#validity-checks) on the `pkgs/by-name` structure. 114 - This is done using the [`nixpkgs-check-by-name` tool](https://github.com/NixOS/nixpkgs-check-by-name). 113 + CI performs [certain checks](https://github.com/NixOS/nixpkgs-vet?tab=readme-ov-file#validity-checks) on the `pkgs/by-name` structure. 114 + This is done using the [`nixpkgs-vet` tool](https://github.com/NixOS/nixpkgs-vet). 115 115 116 116 You can locally emulate the CI check using 117 117 118 118 ``` 119 - $ ./maintainers/scripts/check-by-name.sh master 119 + $ ./ci/nixpkgs-vet.sh master 120 120 ``` 121 121 122 - See [here](../../.github/workflows/check-by-name.yml) for more info. 122 + See [here](../../.github/workflows/nixpkgs-vet.yml) for more info. 123 123 124 124 ## Recommendation for new packages with multiple versions 125 125
-31
pkgs/test/check-by-name/README.md
··· 1 - # `pkgs/by-name` check CI scripts 2 - 3 - This directory contains scripts and files used and related to the CI running the `pkgs/by-name` checks in Nixpkgs. 4 - See also the [CI GitHub Action](../../../.github/workflows/check-by-name.yml). 5 - 6 - ## `./run-local.sh BASE_BRANCH [REPOSITORY]` 7 - 8 - Runs the `pkgs/by-name` check on the HEAD commit, closely matching what CI does. 9 - 10 - Note that this can't do exactly the same as CI, 11 - because CI needs to rely on GitHub's server-side Git history to compute the mergeability of PRs before the check can be started. 12 - In turn when running locally, we don't want to have to push commits to test them, 13 - and we can also rely on the local Git history to do the mergeability check. 14 - 15 - Arguments: 16 - - `BASE_BRANCH`: The base branch to use, e.g. master or release-24.05 17 - - `REPOSITORY`: The repository to fetch the base branch from, defaults to https://github.com/NixOS/nixpkgs.git 18 - 19 - ## `./update-pinned-tool.sh` 20 - 21 - Updates the pinned [nixpkgs-check-by-name tool](https://github.com/NixOS/nixpkgs-check-by-name) in [`./pinned-version.txt`](./pinned-version.txt) to the latest [release](https://github.com/NixOS/nixpkgs-check-by-name/releases). 22 - Each release contains a pre-built x86_64-linux version of the tool which is used by CI. 23 - 24 - This script currently needs to be called manually when the CI tooling needs to be updated. 25 - 26 - Why not just build the tooling right from the PRs Nixpkgs version? 27 - - Because it allows CI to check all PRs, even if they would break the CI tooling. 28 - - Because it makes the CI check very fast, since no Nix builds need to be done, even for mass rebuilds. 29 - - Because it improves security, since we don't have to build potentially untrusted code from PRs. 30 - The tool only needs a very minimal Nix evaluation at runtime, which can work with [readonly-mode](https://nixos.org/manual/nix/stable/command-ref/opt-common.html#opt-readonly-mode) and [restrict-eval](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-restrict-eval). 31 -
-73
pkgs/test/check-by-name/run-local.sh
··· 1 - #!/usr/bin/env nix-shell 2 - #!nix-shell -i bash -p jq 3 - 4 - set -o pipefail -o errexit -o nounset 5 - 6 - trace() { echo >&2 "$@"; } 7 - 8 - tmp=$(mktemp -d) 9 - cleanup() { 10 - # Don't exit early if anything fails to cleanup 11 - set +o errexit 12 - 13 - trace -n "Cleaning up.. " 14 - 15 - [[ -e "$tmp/base" ]] && git worktree remove --force "$tmp/base" 16 - [[ -e "$tmp/merged" ]] && git worktree remove --force "$tmp/merged" 17 - 18 - rm -rf "$tmp" 19 - 20 - trace "Done" 21 - } 22 - trap cleanup exit 23 - 24 - 25 - repo=https://github.com/NixOS/nixpkgs.git 26 - 27 - if (( $# != 0 )); then 28 - baseBranch=$1 29 - shift 30 - else 31 - trace "Usage: $0 BASE_BRANCH [REPOSITORY]" 32 - trace "BASE_BRANCH: The base branch to use, e.g. master or release-23.11" 33 - trace "REPOSITORY: The repository to fetch the base branch from, defaults to $repo" 34 - exit 1 35 - fi 36 - 37 - if (( $# != 0 )); then 38 - repo=$1 39 - shift 40 - fi 41 - 42 - if [[ -n "$(git status --porcelain)" ]]; then 43 - trace -e "\e[33mWarning: Dirty tree, uncommitted changes won't be taken into account\e[0m" 44 - fi 45 - headSha=$(git rev-parse HEAD) 46 - trace -e "Using HEAD commit \e[34m$headSha\e[0m" 47 - 48 - trace -n "Creating Git worktree for the HEAD commit in $tmp/merged.. " 49 - git worktree add --detach -q "$tmp/merged" HEAD 50 - trace "Done" 51 - 52 - trace -n "Fetching base branch $baseBranch to compare against.. " 53 - git fetch -q "$repo" refs/heads/"$baseBranch" 54 - baseSha=$(git rev-parse FETCH_HEAD) 55 - trace -e "\e[34m$baseSha\e[0m" 56 - 57 - trace -n "Creating Git worktree for the base branch in $tmp/base.. " 58 - git worktree add -q "$tmp/base" "$baseSha" 59 - trace "Done" 60 - 61 - trace -n "Merging base branch into the HEAD commit in $tmp/merged.. " 62 - git -C "$tmp/merged" merge -q --no-edit "$baseSha" 63 - trace -e "\e[34m$(git -C "$tmp/merged" rev-parse HEAD)\e[0m" 64 - 65 - trace -n "Reading pinned nixpkgs-check-by-name version from pinned-version.txt.. " 66 - toolVersion=$(<"$tmp/merged/pkgs/test/check-by-name/pinned-version.txt") 67 - trace -e "\e[34m$toolVersion\e[0m" 68 - 69 - trace -n "Building tool.. " 70 - nix-build https://github.com/NixOS/nixpkgs-check-by-name/tarball/"$toolVersion" -o "$tmp/tool" -A build 71 - 72 - trace "Running nixpkgs-check-by-name.." 73 - "$tmp/tool/bin/nixpkgs-check-by-name" --base "$tmp/base" "$tmp/merged"
-22
pkgs/test/check-by-name/update-pinned-tool.sh
··· 1 - #!/usr/bin/env nix-shell 2 - #!nix-shell -i bash -p jq curl 3 - 4 - set -o pipefail -o errexit -o nounset 5 - 6 - trace() { echo >&2 "$@"; } 7 - 8 - SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) 9 - 10 - repository=NixOS/nixpkgs-check-by-name 11 - pin_file=$SCRIPT_DIR/pinned-version.txt 12 - 13 - trace -n "Fetching latest release of $repository.. " 14 - latestRelease=$(curl -sSfL \ 15 - -H "Accept: application/vnd.github+json" \ 16 - -H "X-GitHub-Api-Version: 2022-11-28" \ 17 - https://api.github.com/repos/"$repository"/releases/latest) 18 - latestVersion=$(jq .tag_name -r <<< "$latestRelease") 19 - trace "$latestVersion" 20 - 21 - trace "Updating $pin_file" 22 - echo "$latestVersion" > "$pin_file"