···1+# Path library
2+3+This document explains why the `lib.path` library is designed the way it is.
4+5+The purpose of this library is to process [filesystem paths]. It does not read files from the filesystem.
6+It exists to support the native Nix [path value type] with extra functionality.
7+8+[filesystem paths]: https://en.m.wikipedia.org/wiki/Path_(computing)
9+[path value type]: https://nixos.org/manual/nix/stable/language/values.html#type-path
10+11+As an extension of the path value type, it inherits the same intended use cases and limitations:
12+- Only use paths to access files at evaluation time, such as the local project source.
13+- Paths cannot point to derivations, so they are unfit to represent dependencies.
14+- A path implicitly imports the referenced files into the Nix store when interpolated to a string. Therefore paths are not suitable to access files at build- or run-time, as you risk importing the path from the evaluation system instead.
15+16+Overall, this library works with two types of paths:
17+- Absolute paths are represented with the Nix [path value type]. Nix automatically normalises these paths.
18+- Subpaths are represented with the [string value type] since path value types don't support relative paths. This library normalises these paths as safely as possible. Absolute paths in strings are not supported.
19+20+ A subpath refers to a specific file or directory within an absolute base directory.
21+ It is a stricter form of a relative path, notably [without support for `..` components][parents] since those could escape the base directory.
22+23+[string value type]: https://nixos.org/manual/nix/stable/language/values.html#type-string
24+25+This library is designed to be as safe and intuitive as possible, throwing errors when operations are attempted that would produce surprising results, and giving the expected result otherwise.
26+27+This library is designed to work well as a dependency for the `lib.filesystem` and `lib.sources` library components. Contrary to these library components, `lib.path` does not read any paths from the filesystem.
28+29+This library makes only these assumptions about paths and no others:
30+- `dirOf path` returns the path to the parent directory of `path`, unless `path` is the filesystem root, in which case `path` is returned.
31+ - There can be multiple filesystem roots: `p == dirOf p` and `q == dirOf q` does not imply `p == q`.
32+ - While there's only a single filesystem root in stable Nix, the [lazy trees feature](https://github.com/NixOS/nix/pull/6530) introduces [additional filesystem roots](https://github.com/NixOS/nix/pull/6530#discussion_r1041442173).
33+- `path + ("/" + string)` returns the path to the `string` subdirectory in `path`.
34+ - If `string` contains no `/` characters, then `dirOf (path + ("/" + string)) == path`.
35+ - If `string` contains no `/` characters, then `baseNameOf (path + ("/" + string)) == string`.
36+- `path1 == path2` returns `true` only if `path1` points to the same filesystem path as `path2`.
37+38+Notably we do not make the assumption that we can turn paths into strings using `toString path`.
39+40+## Design decisions
41+42+Each subsection here contains a decision along with arguments and counter-arguments for (+) and against (-) that decision.
43+44+### Leading dots for relative paths
45+[leading-dots]: #leading-dots-for-relative-paths
46+47+Observing: Since subpaths are a form of relative paths, they can have a leading `./` to indicate it being a relative path, this is generally not necessary for tools though.
48+49+Considering: Paths should be as explicit, consistent and unambiguous as possible.
50+51+Decision: Returned subpaths should always have a leading `./`.
52+53+<details>
54+<summary>Arguments</summary>
55+56+- (+) In shells, just running `foo` as a command wouldn't execute the file `foo`, whereas `./foo` would execute the file. In contrast, `foo/bar` does execute that file without the need for `./`. This can lead to confusion about when a `./` needs to be prefixed. If a `./` is always included, this becomes a non-issue. This effectively then means that paths don't overlap with command names.
57+- (+) Prepending with `./` makes the subpaths always valid as relative Nix path expressions.
58+- (+) Using paths in command line arguments could give problems if not escaped properly, e.g. if a path was `--version`. This is not a problem with `./--version`. This effectively then means that paths don't overlap with GNU-style command line options.
59+- (-) `./` is not required to resolve relative paths, resolution always has an implicit `./` as prefix.
60+- (-) It's less noisy without the `./`, e.g. in error messages.
61+ - (+) But similarly, it could be confusing whether something was even a path.
62+ e.g. `foo` could be anything, but `./foo` is more clearly a path.
63+- (+) Makes it more uniform with absolute paths (those always start with `/`).
64+ - (-) That is not relevant for practical purposes.
65+- (+) `find` also outputs results with `./`.
66+ - (-) But only if you give it an argument of `.`. If you give it the argument `some-directory`, it won't prefix that.
67+- (-) `realpath --relative-to` doesn't prefix relative paths with `./`.
68+ - (+) There is no need to return the same result as `realpath`.
69+70+</details>
71+72+### Representation of the current directory
73+[curdir]: #representation-of-the-current-directory
74+75+Observing: The subpath that produces the base directory can be represented with `.` or `./` or `./.`.
76+77+Considering: Paths should be as consistent and unambiguous as possible.
78+79+Decision: It should be `./.`.
80+81+<details>
82+<summary>Arguments</summary>
83+84+- (+) `./` would be inconsistent with [the decision to not persist trailing slashes][trailing-slashes].
85+- (-) `.` is how `realpath` normalises paths.
86+- (+) `.` can be interpreted as a shell command (it's a builtin for sourcing files in `bash` and `zsh`).
87+- (+) `.` would be the only path without a `/`. It could not be used as a Nix path expression, since those require at least one `/` to be parsed as such.
88+- (-) `./.` is rather long.
89+ - (-) We don't require users to type this though, as it's only output by the library.
90+ As inputs all three variants are supported for subpaths (and we can't do anything about absolute paths)
91+- (-) `builtins.dirOf "foo" == "."`, so `.` would be consistent with that.
92+- (+) `./.` is consistent with the [decision to have leading `./`][leading-dots].
93+- (+) `./.` is a valid Nix path expression, although this property does not hold for every relative path or subpath.
94+95+</details>
96+97+### Subpath representation
98+[relrepr]: #subpath-representation
99+100+Observing: Subpaths such as `foo/bar` can be represented in various ways:
101+- string: `"foo/bar"`
102+- list with all the components: `[ "foo" "bar" ]`
103+- attribute set: `{ type = "relative-path"; components = [ "foo" "bar" ]; }`
104+105+Considering: Paths should be as safe to use as possible. We should generate string outputs in the library and not encourage users to do that themselves.
106+107+Decision: Paths are represented as strings.
108+109+<details>
110+<summary>Arguments</summary>
111+112+- (+) It's simpler for the users of the library. One doesn't have to convert a path a string before it can be used.
113+ - (+) Naively converting the list representation to a string with `concatStringsSep "/"` would break for `[]`, requiring library users to be more careful.
114+- (+) It doesn't encourage people to do their own path processing and instead use the library.
115+ With a list representation it would seem easy to just use `lib.lists.init` to get the parent directory, but then it breaks for `.`, which would be represented as `[ ]`.
116+- (+) `+` is convenient and doesn't work on lists and attribute sets.
117+ - (-) Shouldn't use `+` anyways, we export safer functions for path manipulation.
118+119+</details>
120+121+### Parent directory
122+[parents]: #parent-directory
123+124+Observing: Relative paths can have `..` components, which refer to the parent directory.
125+126+Considering: Paths should be as safe and unambiguous as possible.
127+128+Decision: `..` path components in string paths are not supported, neither as inputs nor as outputs. Hence, string paths are called subpaths, rather than relative paths.
129+130+<details>
131+<summary>Arguments</summary>
132+133+- (+) If we wanted relative paths to behave according to the "physical" interpretation (as a directory tree with relations between nodes), it would require resolving symlinks, since e.g. `foo/..` would not be the same as `.` if `foo` is a symlink.
134+ - (-) The "logical" interpretation is also valid (treating paths as a sequence of names), and is used by some software. It is simpler, and not using symlinks at all is safer.
135+ - (+) Mixing both models can lead to surprises.
136+ - (+) We can't resolve symlinks without filesystem access.
137+ - (+) Nix also doesn't support reading symlinks at evaluation time.
138+ - (-) We could just not handle such cases, e.g. `equals "foo" "foo/bar/.. == false`. The paths are different, we don't need to check whether the paths point to the same thing.
139+ - (+) Assume we said `relativeTo /foo /bar == "../bar"`. If this is used like `/bar/../foo` in the end, and `bar` turns out to be a symlink to somewhere else, this won't be accurate.
140+ - (-) We could decide to not support such ambiguous operations, or mark them as such, e.g. the normal `relativeTo` will error on such a case, but there could be `extendedRelativeTo` supporting that.
141+- (-) `..` are a part of paths, a path library should therefore support it.
142+ - (+) If we can convincingly argue that all such use cases are better done e.g. with runtime tools, the library not supporting it can nudge people towards using those.
143+- (-) We could allow "..", but only in the prefix.
144+ - (+) Then we'd have to throw an error for doing `append /some/path "../foo"`, making it non-composable.
145+ - (+) The same is for returning paths with `..`: `relativeTo /foo /bar => "../bar"` would produce a non-composable path.
146+- (+) We argue that `..` is not needed at the Nix evaluation level, since we'd always start evaluation from the project root and don't go up from there.
147+ - (+) `..` is supported in Nix paths, turning them into absolute paths.
148+ - (-) This is ambiguous in the presence of symlinks.
149+- (+) If you need `..` for building or runtime, you can use build-/run-time tooling to create those (e.g. `realpath` with `--relative-to`), or use absolute paths instead.
150+ This also gives you the ability to correctly handle symlinks.
151+152+</details>
153+154+### Trailing slashes
155+[trailing-slashes]: #trailing-slashes
156+157+Observing: Subpaths can contain trailing slashes, like `foo/`, indicating that the path points to a directory and not a file.
158+159+Considering: Paths should be as consistent as possible, there should only be a single normalisation for the same path.
160+161+Decision: All functions remove trailing slashes in their results.
162+163+<details>
164+<summary>Arguments</summary>
165+166+- (+) It allows normalisations to be unique, in that there's only a single normalisation for the same path. If trailing slashes were preserved, both `foo/bar` and `foo/bar/` would be valid but different normalisations for the same path.
167+- Comparison to other frameworks to figure out the least surprising behavior:
168+ - (+) Nix itself doesn't support trailing slashes when parsing and doesn't preserve them when appending paths.
169+ - (-) [Rust's std::path](https://doc.rust-lang.org/std/path/index.html) does preserve them during [construction](https://doc.rust-lang.org/std/path/struct.Path.html#method.new).
170+ - (+) Doesn't preserve them when returning individual [components](https://doc.rust-lang.org/std/path/struct.Path.html#method.components).
171+ - (+) Doesn't preserve them when [canonicalizing](https://doc.rust-lang.org/std/path/struct.Path.html#method.canonicalize).
172+ - (+) [Python 3's pathlib](https://docs.python.org/3/library/pathlib.html#module-pathlib) doesn't preserve them during [construction](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath).
173+ - Notably it represents the individual components as a list internally.
174+ - (-) [Haskell's filepath](https://hackage.haskell.org/package/filepath-1.4.100.0) has [explicit support](https://hackage.haskell.org/package/filepath-1.4.100.0/docs/System-FilePath.html#g:6) for handling trailing slashes.
175+ - (-) Does preserve them for [normalisation](https://hackage.haskell.org/package/filepath-1.4.100.0/docs/System-FilePath.html#v:normalise).
176+ - (-) [NodeJS's Path library](https://nodejs.org/api/path.html) preserves trailing slashes for [normalisation](https://nodejs.org/api/path.html#pathnormalizepath).
177+ - (+) For [parsing a path](https://nodejs.org/api/path.html#pathparsepath) into its significant elements, trailing slashes are not preserved.
178+- (+) Nix's builtin function `dirOf` gives an unexpected result for paths with trailing slashes: `dirOf "foo/bar/" == "foo/bar"`.
179+ Inconsistently, `baseNameOf` works correctly though: `baseNameOf "foo/bar/" == "bar"`.
180+ - (-) We are writing a path library to improve handling of paths though, so we shouldn't use these functions and discourage their use.
181+- (-) Unexpected result when normalising intermediate paths, like `relative.normalise ("foo" + "/") + "bar" == "foobar"`.
182+ - (+) This is not a practical use case though.
183+ - (+) Don't use `+` to append paths, this library has a `join` function for that.
184+ - (-) Users might use `+` out of habit though.
185+- (+) The `realpath` command also removes trailing slashes.
186+- (+) Even with a trailing slash, the path is the same, it's only an indication that it's a directory.
187+188+</details>
189+190+## Other implementations and references
191+192+- [Rust](https://doc.rust-lang.org/std/path/struct.Path.html)
193+- [Python](https://docs.python.org/3/library/pathlib.html)
194+- [Haskell](https://hackage.haskell.org/package/filepath-1.4.100.0/docs/System-FilePath.html)
195+- [Nodejs](https://nodejs.org/api/path.html)
196+- [POSIX.1-2017](https://pubs.opengroup.org/onlinepubs/9699919799/nframe.html)
···1+# Functions for working with paths, see ./path.md
2+{ lib }:
3+let
4+5+ inherit (builtins)
6+ isString
7+ split
8+ match
9+ ;
10+11+ inherit (lib.lists)
12+ length
13+ head
14+ last
15+ genList
16+ elemAt
17+ ;
18+19+ inherit (lib.strings)
20+ concatStringsSep
21+ substring
22+ ;
23+24+ inherit (lib.asserts)
25+ assertMsg
26+ ;
27+28+ # Return the reason why a subpath is invalid, or `null` if it's valid
29+ subpathInvalidReason = value:
30+ if ! isString value then
31+ "The given value is of type ${builtins.typeOf value}, but a string was expected"
32+ else if value == "" then
33+ "The given string is empty"
34+ else if substring 0 1 value == "/" then
35+ "The given string \"${value}\" starts with a `/`, representing an absolute path"
36+ # We don't support ".." components, see ./path.md#parent-directory
37+ else if match "(.*/)?\\.\\.(/.*)?" value != null then
38+ "The given string \"${value}\" contains a `..` component, which is not allowed in subpaths"
39+ else null;
40+41+ # Split and normalise a relative path string into its components.
42+ # Error for ".." components and doesn't include "." components
43+ splitRelPath = path:
44+ let
45+ # Split the string into its parts using regex for efficiency. This regex
46+ # matches patterns like "/", "/./", "/././", with arbitrarily many "/"s
47+ # together. These are the main special cases:
48+ # - Leading "./" gets split into a leading "." part
49+ # - Trailing "/." or "/" get split into a trailing "." or ""
50+ # part respectively
51+ #
52+ # These are the only cases where "." and "" parts can occur
53+ parts = split "/+(\\./+)*" path;
54+55+ # `split` creates a list of 2 * k + 1 elements, containing the k +
56+ # 1 parts, interleaved with k matches where k is the number of
57+ # (non-overlapping) matches. This calculation here gets the number of parts
58+ # back from the list length
59+ # floor( (2 * k + 1) / 2 ) + 1 == floor( k + 1/2 ) + 1 == k + 1
60+ partCount = length parts / 2 + 1;
61+62+ # To assemble the final list of components we want to:
63+ # - Skip a potential leading ".", normalising "./foo" to "foo"
64+ # - Skip a potential trailing "." or "", normalising "foo/" and "foo/." to
65+ # "foo". See ./path.md#trailing-slashes
66+ skipStart = if head parts == "." then 1 else 0;
67+ skipEnd = if last parts == "." || last parts == "" then 1 else 0;
68+69+ # We can now know the length of the result by removing the number of
70+ # skipped parts from the total number
71+ componentCount = partCount - skipEnd - skipStart;
72+73+ in
74+ # Special case of a single "." path component. Such a case leaves a
75+ # componentCount of -1 due to the skipStart/skipEnd not verifying that
76+ # they don't refer to the same character
77+ if path == "." then []
78+79+ # Generate the result list directly. This is more efficient than a
80+ # combination of `filter`, `init` and `tail`, because here we don't
81+ # allocate any intermediate lists
82+ else genList (index:
83+ # To get to the element we need to add the number of parts we skip and
84+ # multiply by two due to the interleaved layout of `parts`
85+ elemAt parts ((skipStart + index) * 2)
86+ ) componentCount;
87+88+ # Join relative path components together
89+ joinRelPath = components:
90+ # Always return relative paths with `./` as a prefix (./path.md#leading-dots-for-relative-paths)
91+ "./" +
92+ # An empty string is not a valid relative path, so we need to return a `.` when we have no components
93+ (if components == [] then "." else concatStringsSep "/" components);
94+95+in /* No rec! Add dependencies on this file at the top. */ {
96+97+98+ /* Whether a value is a valid subpath string.
99+100+ - The value is a string
101+102+ - The string is not empty
103+104+ - The string doesn't start with a `/`
105+106+ - The string doesn't contain any `..` path components
107+108+ Type:
109+ subpath.isValid :: String -> Bool
110+111+ Example:
112+ # Not a string
113+ subpath.isValid null
114+ => false
115+116+ # Empty string
117+ subpath.isValid ""
118+ => false
119+120+ # Absolute path
121+ subpath.isValid "/foo"
122+ => false
123+124+ # Contains a `..` path component
125+ subpath.isValid "../foo"
126+ => false
127+128+ # Valid subpath
129+ subpath.isValid "foo/bar"
130+ => true
131+132+ # Doesn't need to be normalised
133+ subpath.isValid "./foo//bar/"
134+ => true
135+ */
136+ subpath.isValid = value:
137+ subpathInvalidReason value == null;
138+139+140+ /* Normalise a subpath. Throw an error if the subpath isn't valid, see
141+ `lib.path.subpath.isValid`
142+143+ - Limit repeating `/` to a single one
144+145+ - Remove redundant `.` components
146+147+ - Remove trailing `/` and `/.`
148+149+ - Add leading `./`
150+151+ Laws:
152+153+ - (Idempotency) Normalising multiple times gives the same result:
154+155+ subpath.normalise (subpath.normalise p) == subpath.normalise p
156+157+ - (Uniqueness) There's only a single normalisation for the paths that lead to the same file system node:
158+159+ subpath.normalise p != subpath.normalise q -> $(realpath ${p}) != $(realpath ${q})
160+161+ - Don't change the result when appended to a Nix path value:
162+163+ base + ("/" + p) == base + ("/" + subpath.normalise p)
164+165+ - Don't change the path according to `realpath`:
166+167+ $(realpath ${p}) == $(realpath ${subpath.normalise p})
168+169+ - Only error on invalid subpaths:
170+171+ builtins.tryEval (subpath.normalise p)).success == subpath.isValid p
172+173+ Type:
174+ subpath.normalise :: String -> String
175+176+ Example:
177+ # limit repeating `/` to a single one
178+ subpath.normalise "foo//bar"
179+ => "./foo/bar"
180+181+ # remove redundant `.` components
182+ subpath.normalise "foo/./bar"
183+ => "./foo/bar"
184+185+ # add leading `./`
186+ subpath.normalise "foo/bar"
187+ => "./foo/bar"
188+189+ # remove trailing `/`
190+ subpath.normalise "foo/bar/"
191+ => "./foo/bar"
192+193+ # remove trailing `/.`
194+ subpath.normalise "foo/bar/."
195+ => "./foo/bar"
196+197+ # Return the current directory as `./.`
198+ subpath.normalise "."
199+ => "./."
200+201+ # error on `..` path components
202+ subpath.normalise "foo/../bar"
203+ => <error>
204+205+ # error on empty string
206+ subpath.normalise ""
207+ => <error>
208+209+ # error on absolute path
210+ subpath.normalise "/foo"
211+ => <error>
212+ */
213+ subpath.normalise = path:
214+ assert assertMsg (subpathInvalidReason path == null)
215+ "lib.path.subpath.normalise: Argument is not a valid subpath string: ${subpathInvalidReason path}";
216+ joinRelPath (splitRelPath path);
217+218+}
···1+# Generate random path-like strings, separated by null characters.
2+#
3+# Invocation:
4+#
5+# awk -f ./generate.awk -v <variable>=<value> | tr '\0' '\n'
6+#
7+# Customizable variables (all default to 0):
8+# - seed: Deterministic random seed to use for generation
9+# - count: Number of paths to generate
10+# - extradotweight: Give extra weight to dots being generated
11+# - extraslashweight: Give extra weight to slashes being generated
12+# - extranullweight: Give extra weight to null being generated, making paths shorter
13+BEGIN {
14+ # Random seed, passed explicitly for reproducibility
15+ srand(seed)
16+17+ # Don't include special characters below 32
18+ minascii = 32
19+ # Don't include DEL at 128
20+ maxascii = 127
21+ upperascii = maxascii - minascii
22+23+ # add extra weight for ., in addition to the one weight from the ascii range
24+ upperdot = upperascii + extradotweight
25+26+ # add extra weight for /, in addition to the one weight from the ascii range
27+ upperslash = upperdot + extraslashweight
28+29+ # add extra weight for null, indicating the end of the string
30+ # Must be at least 1 to have strings end at all
31+ total = upperslash + 1 + extranullweight
32+33+ # new=1 indicates that it's a new string
34+ new=1
35+ while (count > 0) {
36+37+ # Random integer between [0, total)
38+ value = int(rand() * total)
39+40+ if (value < upperascii) {
41+ # Ascii range
42+ printf("%c", value + minascii)
43+ new=0
44+45+ } else if (value < upperdot) {
46+ # Dot range
47+ printf "."
48+ new=0
49+50+ } else if (value < upperslash) {
51+ # If it's the start of a new path, only generate a / in 10% of cases
52+ # This is always an invalid subpath, which is not a very interesting case
53+ if (new && rand() > 0.1) continue
54+ printf "/"
55+56+ } else {
57+ # Do not generate empty strings
58+ if (new) continue
59+ printf "\x00"
60+ count--
61+ new=1
62+ }
63+ }
64+}
···1+# Given a list of path-like strings, check some properties of the path library
2+# using those paths and return a list of attribute sets of the following form:
3+#
4+# { <string> = <lib.path.subpath.normalise string>; }
5+#
6+# If `normalise` fails to evaluate, the attribute value is set to `""`.
7+# If not, the resulting value is normalised again and an appropriate attribute set added to the output list.
8+{
9+ # The path to the nixpkgs lib to use
10+ libpath,
11+ # A flat directory containing files with randomly-generated
12+ # path-like values
13+ dir,
14+}:
15+let
16+ lib = import libpath;
17+18+ # read each file into a string
19+ strings = map (name:
20+ builtins.readFile (dir + "/${name}")
21+ ) (builtins.attrNames (builtins.readDir dir));
22+23+ inherit (lib.path.subpath) normalise isValid;
24+ inherit (lib.asserts) assertMsg;
25+26+ normaliseAndCheck = str:
27+ let
28+ originalValid = isValid str;
29+30+ tryOnce = builtins.tryEval (normalise str);
31+ tryTwice = builtins.tryEval (normalise tryOnce.value);
32+33+ absConcatOrig = /. + ("/" + str);
34+ absConcatNormalised = /. + ("/" + tryOnce.value);
35+ in
36+ # Check the lib.path.subpath.normalise property to only error on invalid subpaths
37+ assert assertMsg
38+ (originalValid -> tryOnce.success)
39+ "Even though string \"${str}\" is valid as a subpath, the normalisation for it failed";
40+ assert assertMsg
41+ (! originalValid -> ! tryOnce.success)
42+ "Even though string \"${str}\" is invalid as a subpath, the normalisation for it succeeded";
43+44+ # Check normalisation idempotency
45+ assert assertMsg
46+ (originalValid -> tryTwice.success)
47+ "For valid subpath \"${str}\", the normalisation \"${tryOnce.value}\" was not a valid subpath";
48+ assert assertMsg
49+ (originalValid -> tryOnce.value == tryTwice.value)
50+ "For valid subpath \"${str}\", normalising it once gives \"${tryOnce.value}\" but normalising it twice gives a different result: \"${tryTwice.value}\"";
51+52+ # Check that normalisation doesn't change a string when appended to an absolute Nix path value
53+ assert assertMsg
54+ (originalValid -> absConcatOrig == absConcatNormalised)
55+ "For valid subpath \"${str}\", appending to an absolute Nix path value gives \"${absConcatOrig}\", but appending the normalised result \"${tryOnce.value}\" gives a different value \"${absConcatNormalised}\"";
56+57+ # Return an empty string when failed
58+ if tryOnce.success then tryOnce.value else "";
59+60+in lib.genAttrs strings normaliseAndCheck
···1+#!/usr/bin/env bash
2+3+# Property tests for the `lib.path` library
4+#
5+# It generates random path-like strings and runs the functions on
6+# them, checking that the expected laws of the functions hold
7+8+set -euo pipefail
9+shopt -s inherit_errexit
10+11+# https://stackoverflow.com/a/246128
12+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
13+14+if test -z "${TEST_LIB:-}"; then
15+ TEST_LIB=$SCRIPT_DIR/../..
16+fi
17+18+tmp="$(mktemp -d)"
19+clean_up() {
20+ rm -rf "$tmp"
21+}
22+trap clean_up EXIT
23+mkdir -p "$tmp/work"
24+cd "$tmp/work"
25+26+# Defaulting to a random seed but the first argument can override this
27+seed=${1:-$RANDOM}
28+echo >&2 "Using seed $seed, use \`lib/path/tests/prop.sh $seed\` to reproduce this result"
29+30+# The number of random paths to generate. This specific number was chosen to
31+# be fast enough while still generating enough variety to detect bugs.
32+count=500
33+34+debug=0
35+# debug=1 # print some extra info
36+# debug=2 # print generated values
37+38+# Fine tuning parameters to balance the number of generated invalid paths
39+# to the variance in generated paths.
40+extradotweight=64 # Larger value: more dots
41+extraslashweight=64 # Larger value: more slashes
42+extranullweight=16 # Larger value: shorter strings
43+44+die() {
45+ echo >&2 "test case failed: " "$@"
46+ exit 1
47+}
48+49+if [[ "$debug" -ge 1 ]]; then
50+ echo >&2 "Generating $count random path-like strings"
51+fi
52+53+# Read stream of null-terminated strings entry-by-entry into bash,
54+# write it to a file and the `strings` array.
55+declare -a strings=()
56+mkdir -p "$tmp/strings"
57+while IFS= read -r -d $'\0' str; do
58+ echo -n "$str" > "$tmp/strings/${#strings[@]}"
59+ strings+=("$str")
60+done < <(awk \
61+ -f "$SCRIPT_DIR"/generate.awk \
62+ -v seed="$seed" \
63+ -v count="$count" \
64+ -v extradotweight="$extradotweight" \
65+ -v extraslashweight="$extraslashweight" \
66+ -v extranullweight="$extranullweight")
67+68+if [[ "$debug" -ge 1 ]]; then
69+ echo >&2 "Trying to normalise the generated path-like strings with Nix"
70+fi
71+72+# Precalculate all normalisations with a single Nix call. Calling Nix for each
73+# string individually would take way too long
74+nix-instantiate --eval --strict --json \
75+ --argstr libpath "$TEST_LIB" \
76+ --argstr dir "$tmp/strings" \
77+ "$SCRIPT_DIR"/prop.nix \
78+ >"$tmp/result.json"
79+80+# Uses some jq magic to turn the resulting attribute set into an associative
81+# bash array assignment
82+declare -A normalised_result="($(jq '
83+ to_entries
84+ | map("[\(.key | @sh)]=\(.value | @sh)")
85+ | join(" \n")' -r < "$tmp/result.json"))"
86+87+# Looks up a normalisation result for a string
88+# Checks that the normalisation is only failing iff it's an invalid subpath
89+# For valid subpaths, returns 0 and prints the normalisation result
90+# For invalid subpaths, returns 1
91+normalise() {
92+ local str=$1
93+ # Uses the same check for validity as in the library implementation
94+ if [[ "$str" == "" || "$str" == /* || "$str" =~ ^(.*/)?\.\.(/.*)?$ ]]; then
95+ valid=
96+ else
97+ valid=1
98+ fi
99+100+ normalised=${normalised_result[$str]}
101+ # An empty string indicates failure, this is encoded in ./prop.nix
102+ if [[ -n "$normalised" ]]; then
103+ if [[ -n "$valid" ]]; then
104+ echo "$normalised"
105+ else
106+ die "For invalid subpath \"$str\", lib.path.subpath.normalise returned this result: \"$normalised\""
107+ fi
108+ else
109+ if [[ -n "$valid" ]]; then
110+ die "For valid subpath \"$str\", lib.path.subpath.normalise failed"
111+ else
112+ if [[ "$debug" -ge 2 ]]; then
113+ echo >&2 "String \"$str\" is not a valid subpath"
114+ fi
115+ # Invalid and it correctly failed, we let the caller continue if they catch the exit code
116+ return 1
117+ fi
118+ fi
119+}
120+121+# Intermediate result populated by test_idempotency_realpath
122+# and used in test_normalise_uniqueness
123+#
124+# Contains a mapping from a normalised subpath to the realpath result it represents
125+declare -A norm_to_real
126+127+test_idempotency_realpath() {
128+ if [[ "$debug" -ge 1 ]]; then
129+ echo >&2 "Checking idempotency of each result and making sure the realpath result isn't changed"
130+ fi
131+132+ # Count invalid subpaths to display stats
133+ invalid=0
134+ for str in "${strings[@]}"; do
135+ if ! result=$(normalise "$str"); then
136+ ((invalid++)) || true
137+ continue
138+ fi
139+140+ # Check the law that it doesn't change the result of a realpath
141+ mkdir -p -- "$str" "$result"
142+ real_orig=$(realpath -- "$str")
143+ real_norm=$(realpath -- "$result")
144+145+ if [[ "$real_orig" != "$real_norm" ]]; then
146+ die "realpath of the original string \"$str\" (\"$real_orig\") is not the same as realpath of the normalisation \"$result\" (\"$real_norm\")"
147+ fi
148+149+ if [[ "$debug" -ge 2 ]]; then
150+ echo >&2 "String \"$str\" gets normalised to \"$result\" and file path \"$real_orig\""
151+ fi
152+ norm_to_real["$result"]="$real_orig"
153+ done
154+ if [[ "$debug" -ge 1 ]]; then
155+ echo >&2 "$(bc <<< "scale=1; 100 / $count * $invalid")% of the total $count generated strings were invalid subpath strings, and were therefore ignored"
156+ fi
157+}
158+159+test_normalise_uniqueness() {
160+ if [[ "$debug" -ge 1 ]]; then
161+ echo >&2 "Checking for the uniqueness law"
162+ fi
163+164+ for norm_p in "${!norm_to_real[@]}"; do
165+ real_p=${norm_to_real["$norm_p"]}
166+ for norm_q in "${!norm_to_real[@]}"; do
167+ real_q=${norm_to_real["$norm_q"]}
168+ # Checks normalisation uniqueness law for each pair of values
169+ if [[ "$norm_p" != "$norm_q" && "$real_p" == "$real_q" ]]; then
170+ die "Normalisations \"$norm_p\" and \"$norm_q\" are different, but the realpath of them is the same: \"$real_p\""
171+ fi
172+ done
173+ done
174+}
175+176+test_idempotency_realpath
177+test_normalise_uniqueness
178+179+echo >&2 tests ok
···360 </listitem>
361 <listitem>
362 <para>
0000000363 The module <literal>services.headscale</literal> was
364 refactored to be compliant with
365 <link xlink:href="https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md">RFC
···360 </listitem>
361 <listitem>
362 <para>
363+ <literal>services.chronyd</literal> is now started with
364+ additional systemd sandbox/hardening options for better
365+ security.
366+ </para>
367+ </listitem>
368+ <listitem>
369+ <para>
370 The module <literal>services.headscale</literal> was
371 refactored to be compliant with
372 <link xlink:href="https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md">RFC
+2
nixos/doc/manual/release-notes/rl-2305.section.md
···9899 And backup your data.
10000101- The module `services.headscale` was refactored to be compliant with [RFC 0042](https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md). To be precise, this means that the following things have changed:
102103 - Most settings has been migrated under [services.headscale.settings](#opt-services.headscale.settings) which is an attribute-set that
···9899 And backup your data.
100101+- `services.chronyd` is now started with additional systemd sandbox/hardening options for better security.
102+103- The module `services.headscale` was refactored to be compliant with [RFC 0042](https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md). To be precise, this means that the following things have changed:
104105 - Most settings has been migrated under [services.headscale.settings](#opt-services.headscale.settings) which is an attribute-set that
+1-1
nixos/modules/programs/gnupg.nix
···135 # The SSH agent protocol doesn't have support for changing TTYs; however we
136 # can simulate this with the `exec` feature of openssh (see ssh_config(5))
137 # that hooks a command to the shell currently running the ssh program.
138- Match host * exec "${cfg.package}/bin/gpg-connect-agent --quiet updatestartuptty /bye >/dev/null 2>&1"
139 '';
140141 environment.extraInit = mkIf cfg.agent.enableSSHSupport ''
···135 # The SSH agent protocol doesn't have support for changing TTYs; however we
136 # can simulate this with the `exec` feature of openssh (see ssh_config(5))
137 # that hooks a command to the shell currently running the ssh program.
138+ Match host * exec "${pkgs.runtimeShell} -c '${cfg.package}/bin/gpg-connect-agent --quiet updatestartuptty /bye >/dev/null 2>&1'"
139 '';
140141 environment.extraInit = mkIf cfg.agent.enableSSHSupport ''
···1-{ fetchFromGitHub, lib, i3 }:
2-3-i3.overrideAttrs (oldAttrs : rec {
4- pname = "i3-gaps";
5- version = "4.21.1";
6-7- src = fetchFromGitHub {
8- owner = "Airblader";
9- repo = "i3";
10- rev = version;
11- sha256 = "sha256-+JxJjvzEuAA4CH+gufzAzIqd5BSvHtPvLm2zTfXc/xk=";
12- };
13-14- meta = with lib; {
15- description = "A fork of the i3 tiling window manager with some additional features";
16- homepage = "https://github.com/Airblader/i3";
17- maintainers = with maintainers; [ fmthoma ];
18- license = licenses.bsd3;
19- platforms = platforms.linux ++ platforms.netbsd ++ platforms.openbsd;
20-21- longDescription = ''
22- Fork of i3wm, a tiling window manager primarily targeted at advanced users
23- and developers. Based on a tree as data structure, supports tiling,
24- stacking, and tabbing layouts, handled dynamically, as well as floating
25- windows. This fork adds a few features such as gaps between windows.
26- Configured via plain text file. Multi-monitor. UTF-8 clean.
27- '';
28- };
29-})
···00000000000000000000000000000
+115-76
pkgs/data/fonts/noto-fonts/default.nix
···11, imagemagick
12, zopfli
13, buildPackages
014}:
15-16let
17- mkNoto = { pname, weights }:
18- stdenvNoCC.mkDerivation {
0000000000000000000019 inherit pname;
20- version = "2020-01-23";
2122 src = fetchFromGitHub {
23 owner = "googlefonts";
24 repo = "noto-fonts";
25- rev = "f4726a2ec36169abd02a6d8abe67c8ff0236f6d8";
26- sha256 = "0zc1r7zph62qmvzxqfflsprazjf6x1qnwc2ma27kyzh6v36gaykw";
27 };
002829 installPhase = ''
30 # We copy in reverse preference order -- unhinted first, then
···33 #
34 # TODO: install OpenType, variable versions?
35 local out_ttf=$out/share/fonts/truetype/noto
36- install -m444 -Dt $out_ttf phaseIII_only/unhinted/ttf/*/*-${weights}.ttf
37- install -m444 -Dt $out_ttf phaseIII_only/hinted/ttf/*/*-${weights}.ttf
38- install -m444 -Dt $out_ttf unhinted/*/*-${weights}.ttf
39- install -m444 -Dt $out_ttf hinted/*/*-${weights}.ttf
40- '';
000000004142 meta = with lib; {
43 description = "Beautiful and free fonts for many languages";
44 homepage = "https://www.google.com/get/noto/";
45- longDescription =
46- ''
47- When text is rendered by a computer, sometimes characters are
48- displayed as “tofu”. They are little boxes to indicate your device
49- doesn’t have a font to display the text.
50-51- Google has been developing a font family called Noto, which aims to
52- support all languages with a harmonious look and feel. Noto is
53- Google’s answer to tofu. The name noto is to convey the idea that
54- Google’s goal is to see “no more tofu”. Noto has multiple styles and
55- weights, and freely available to all.
56-57- This package also includes the Arimo, Cousine, and Tinos fonts.
58- '';
59 license = licenses.ofl;
60 platforms = platforms.all;
61 maintainers = with maintainers; [ mathnerd314 emily ];
···100 maintainers = with maintainers; [ mathnerd314 emily ];
101 };
102 };
103-in
104105-{
106 noto-fonts = mkNoto {
107 pname = "noto-fonts";
108 weights = "{Regular,Bold,Light,Italic,BoldItalic,LightItalic}";
109 };
1100000000000000000000000111 noto-fonts-extra = mkNoto {
112 pname = "noto-fonts-extra";
113 weights = "{Black,Condensed,Extra,Medium,Semi,Thin}*";
···127 sha256 = "sha256-1w66Ge7DZjbONGhxSz69uFhfsjMsDiDkrGl6NsoB7dY=";
128 };
129130- noto-fonts-emoji = let
131- version = "2.038";
132- emojiPythonEnv =
133- buildPackages.python3.withPackages (p: with p; [ fonttools nototools ]);
134- in stdenvNoCC.mkDerivation {
135- pname = "noto-fonts-emoji";
136- inherit version;
00137138- src = fetchFromGitHub {
139- owner = "googlefonts";
140- repo = "noto-emoji";
141- rev = "v${version}";
142- sha256 = "1rgmcc6nqq805iqr8kvxxlk5cf50q714xaxk3ld6rjrd69kb8ix9";
143- };
144145- depsBuildBuild = [
146- buildPackages.stdenv.cc
147- pkg-config
148- cairo
149- ];
150151- nativeBuildInputs = [
152- imagemagick
153- zopfli
154- pngquant
155- which
156- emojiPythonEnv
157- ];
158159- postPatch = ''
160- patchShebangs *.py
161- patchShebangs third_party/color_emoji/*.py
162- # remove check for virtualenv, since we handle
163- # python requirements using python.withPackages
164- sed -i '/ifndef VIRTUAL_ENV/,+2d' Makefile
165166- # Make the build verbose so it won't get culled by Hydra thinking that
167- # it somehow got stuck doing nothing.
168- sed -i 's;\t@;\t;' Makefile
169- '';
170171- enableParallelBuilding = true;
172173- installPhase = ''
174- runHook preInstall
175- mkdir -p $out/share/fonts/noto
176- cp NotoColorEmoji.ttf $out/share/fonts/noto
177- runHook postInstall
178- '';
179180- meta = with lib; {
181- description = "Color and Black-and-White emoji fonts";
182- homepage = "https://github.com/googlefonts/noto-emoji";
183- license = with licenses; [ ofl asl20 ];
184- platforms = platforms.all;
185- maintainers = with maintainers; [ mathnerd314 sternenseemann ];
0186 };
187- };
188189 noto-fonts-emoji-blob-bin =
190 let
···11, imagemagick
12, zopfli
13, buildPackages
14+, variants ? [ ]
15}:
016let
17+ notoLongDescription = ''
18+ When text is rendered by a computer, sometimes characters are
19+ displayed as “tofu”. They are little boxes to indicate your device
20+ doesn’t have a font to display the text.
21+22+ Google has been developing a font family called Noto, which aims to
23+ support all languages with a harmonious look and feel. Noto is
24+ Google’s answer to tofu. The name noto is to convey the idea that
25+ Google’s goal is to see “no more tofu”. Noto has multiple styles and
26+ weights, and freely available to all.
27+28+ This package also includes the Arimo, Cousine, and Tinos fonts.
29+ '';
30+in
31+rec {
32+ mkNoto =
33+ { pname
34+ , weights
35+ , variants ? [ ]
36+ , longDescription ? notoLongDescription
37+ }:
38+ stdenvNoCC.mkDerivation rec {
39 inherit pname;
40+ version = "20201206-phase3";
4142 src = fetchFromGitHub {
43 owner = "googlefonts";
44 repo = "noto-fonts";
45+ rev = "v${version}";
46+ hash = "sha256-x60RvCRFLoGe0CNvswROnDkIsUFbWH+/laN8q2qkUPk=";
47 };
48+49+ _variants = map (variant: builtins.replaceStrings [ " " ] [ "" ] variant) variants;
5051 installPhase = ''
52 # We copy in reverse preference order -- unhinted first, then
···55 #
56 # TODO: install OpenType, variable versions?
57 local out_ttf=$out/share/fonts/truetype/noto
58+ '' + (if _variants == [ ] then ''
59+ install -m444 -Dt $out_ttf archive/unhinted/*/*-${weights}.ttf
60+ install -m444 -Dt $out_ttf archive/hinted/*/*-${weights}.ttf
61+ install -m444 -Dt $out_ttf unhinted/*/*/*-${weights}.ttf
62+ install -m444 -Dt $out_ttf hinted/*/*/*-${weights}.ttf
63+ '' else ''
64+ for variant in $_variants; do
65+ install -m444 -Dt $out_ttf archive/unhinted/$variant/*-${weights}.ttf
66+ install -m444 -Dt $out_ttf archive/hinted/$variant/*-${weights}.ttf
67+ install -m444 -Dt $out_ttf unhinted/*/$variant/*-${weights}.ttf
68+ install -m444 -Dt $out_ttf hinted/*/$variant/*-${weights}.ttf
69+ done
70+ '');
7172 meta = with lib; {
73 description = "Beautiful and free fonts for many languages";
74 homepage = "https://www.google.com/get/noto/";
75+ inherit longDescription;
000000000000076 license = licenses.ofl;
77 platforms = platforms.all;
78 maintainers = with maintainers; [ mathnerd314 emily ];
···117 maintainers = with maintainers; [ mathnerd314 emily ];
118 };
119 };
01200121 noto-fonts = mkNoto {
122 pname = "noto-fonts";
123 weights = "{Regular,Bold,Light,Italic,BoldItalic,LightItalic}";
124 };
125126+ noto-fonts-lgc-plus = mkNoto {
127+ pname = "noto-fonts-lgc-plus";
128+ weights = "{Regular,Bold,Light,Italic,BoldItalic,LightItalic}";
129+ variants = [
130+ "Noto Sans"
131+ "Noto Serif"
132+ "Noto Sans Display"
133+ "Noto Serif Display"
134+ "Noto Sans Mono"
135+ "Noto Music"
136+ "Noto Sans Symbols"
137+ "Noto Sans Symbols 2"
138+ "Noto Sans Math"
139+ ];
140+ longDescription = ''
141+ This package provides the Noto Fonts, but only for latin, greek
142+ and cyrillic scripts, as well as some extra fonts. To create a
143+ custom Noto package with custom variants, see the `mkNoto`
144+ helper function.
145+ '';
146+ };
147+148 noto-fonts-extra = mkNoto {
149 pname = "noto-fonts-extra";
150 weights = "{Black,Condensed,Extra,Medium,Semi,Thin}*";
···164 sha256 = "sha256-1w66Ge7DZjbONGhxSz69uFhfsjMsDiDkrGl6NsoB7dY=";
165 };
166167+ noto-fonts-emoji =
168+ let
169+ version = "2.038";
170+ emojiPythonEnv =
171+ buildPackages.python3.withPackages (p: with p; [ fonttools nototools ]);
172+ in
173+ stdenvNoCC.mkDerivation {
174+ pname = "noto-fonts-emoji";
175+ inherit version;
176177+ src = fetchFromGitHub {
178+ owner = "googlefonts";
179+ repo = "noto-emoji";
180+ rev = "v${version}";
181+ sha256 = "1rgmcc6nqq805iqr8kvxxlk5cf50q714xaxk3ld6rjrd69kb8ix9";
182+ };
183184+ depsBuildBuild = [
185+ buildPackages.stdenv.cc
186+ pkg-config
187+ cairo
188+ ];
189190+ nativeBuildInputs = [
191+ imagemagick
192+ zopfli
193+ pngquant
194+ which
195+ emojiPythonEnv
196+ ];
197198+ postPatch = ''
199+ patchShebangs *.py
200+ patchShebangs third_party/color_emoji/*.py
201+ # remove check for virtualenv, since we handle
202+ # python requirements using python.withPackages
203+ sed -i '/ifndef VIRTUAL_ENV/,+2d' Makefile
204205+ # Make the build verbose so it won't get culled by Hydra thinking that
206+ # it somehow got stuck doing nothing.
207+ sed -i 's;\t@;\t;' Makefile
208+ '';
209210+ enableParallelBuilding = true;
211212+ installPhase = ''
213+ runHook preInstall
214+ mkdir -p $out/share/fonts/noto
215+ cp NotoColorEmoji.ttf $out/share/fonts/noto
216+ runHook postInstall
217+ '';
218219+ meta = with lib; {
220+ description = "Color and Black-and-White emoji fonts";
221+ homepage = "https://github.com/googlefonts/noto-emoji";
222+ license = with licenses; [ ofl asl20 ];
223+ platforms = platforms.all;
224+ maintainers = with maintainers; [ mathnerd314 sternenseemann ];
225+ };
226 };
0227228 noto-fonts-emoji-blob-bin =
229 let
···633634 ### I ###
6350636 i3cat = throw "i3cat has been dropped due to the lack of maintanence from upstream since 2016"; # Added 2022-06-02
637 iana_etc = throw "'iana_etc' has been renamed to/replaced by 'iana-etc'"; # Converted to throw 2022-02-22
638 iasl = throw "iasl has been removed, use acpica-tools instead"; # Added 2021-08-08
···633634 ### I ###
635636+ i3-gaps = i3; # Added 2023-01-03
637 i3cat = throw "i3cat has been dropped due to the lack of maintanence from upstream since 2016"; # Added 2022-06-02
638 iana_etc = throw "'iana_etc' has been renamed to/replaced by 'iana-etc'"; # Converted to throw 2022-02-22
639 iasl = throw "iasl has been removed, use acpica-tools instead"; # Added 2021-08-08