···11+# Path library
22+33+This document explains why the `lib.path` library is designed the way it is.
44+55+The purpose of this library is to process [filesystem paths]. It does not read files from the filesystem.
66+It exists to support the native Nix [path value type] with extra functionality.
77+88+[filesystem paths]: https://en.m.wikipedia.org/wiki/Path_(computing)
99+[path value type]: https://nixos.org/manual/nix/stable/language/values.html#type-path
1010+1111+As an extension of the path value type, it inherits the same intended use cases and limitations:
1212+- Only use paths to access files at evaluation time, such as the local project source.
1313+- Paths cannot point to derivations, so they are unfit to represent dependencies.
1414+- A path implicitly imports the referenced files into the Nix store when interpolated to a string. Therefore paths are not suitable to access files at build- or run-time, as you risk importing the path from the evaluation system instead.
1515+1616+Overall, this library works with two types of paths:
1717+- Absolute paths are represented with the Nix [path value type]. Nix automatically normalises these paths.
1818+- Subpaths are represented with the [string value type] since path value types don't support relative paths. This library normalises these paths as safely as possible. Absolute paths in strings are not supported.
1919+2020+ A subpath refers to a specific file or directory within an absolute base directory.
2121+ It is a stricter form of a relative path, notably [without support for `..` components][parents] since those could escape the base directory.
2222+2323+[string value type]: https://nixos.org/manual/nix/stable/language/values.html#type-string
2424+2525+This library is designed to be as safe and intuitive as possible, throwing errors when operations are attempted that would produce surprising results, and giving the expected result otherwise.
2626+2727+This library is designed to work well as a dependency for the `lib.filesystem` and `lib.sources` library components. Contrary to these library components, `lib.path` does not read any paths from the filesystem.
2828+2929+This library makes only these assumptions about paths and no others:
3030+- `dirOf path` returns the path to the parent directory of `path`, unless `path` is the filesystem root, in which case `path` is returned.
3131+ - There can be multiple filesystem roots: `p == dirOf p` and `q == dirOf q` does not imply `p == q`.
3232+ - While there's only a single filesystem root in stable Nix, the [lazy trees feature](https://github.com/NixOS/nix/pull/6530) introduces [additional filesystem roots](https://github.com/NixOS/nix/pull/6530#discussion_r1041442173).
3333+- `path + ("/" + string)` returns the path to the `string` subdirectory in `path`.
3434+ - If `string` contains no `/` characters, then `dirOf (path + ("/" + string)) == path`.
3535+ - If `string` contains no `/` characters, then `baseNameOf (path + ("/" + string)) == string`.
3636+- `path1 == path2` returns `true` only if `path1` points to the same filesystem path as `path2`.
3737+3838+Notably we do not make the assumption that we can turn paths into strings using `toString path`.
3939+4040+## Design decisions
4141+4242+Each subsection here contains a decision along with arguments and counter-arguments for (+) and against (-) that decision.
4343+4444+### Leading dots for relative paths
4545+[leading-dots]: #leading-dots-for-relative-paths
4646+4747+Observing: Since subpaths are a form of relative paths, they can have a leading `./` to indicate it being a relative path, this is generally not necessary for tools though.
4848+4949+Considering: Paths should be as explicit, consistent and unambiguous as possible.
5050+5151+Decision: Returned subpaths should always have a leading `./`.
5252+5353+<details>
5454+<summary>Arguments</summary>
5555+5656+- (+) In shells, just running `foo` as a command wouldn't execute the file `foo`, whereas `./foo` would execute the file. In contrast, `foo/bar` does execute that file without the need for `./`. This can lead to confusion about when a `./` needs to be prefixed. If a `./` is always included, this becomes a non-issue. This effectively then means that paths don't overlap with command names.
5757+- (+) Prepending with `./` makes the subpaths always valid as relative Nix path expressions.
5858+- (+) Using paths in command line arguments could give problems if not escaped properly, e.g. if a path was `--version`. This is not a problem with `./--version`. This effectively then means that paths don't overlap with GNU-style command line options.
5959+- (-) `./` is not required to resolve relative paths, resolution always has an implicit `./` as prefix.
6060+- (-) It's less noisy without the `./`, e.g. in error messages.
6161+ - (+) But similarly, it could be confusing whether something was even a path.
6262+ e.g. `foo` could be anything, but `./foo` is more clearly a path.
6363+- (+) Makes it more uniform with absolute paths (those always start with `/`).
6464+ - (-) That is not relevant for practical purposes.
6565+- (+) `find` also outputs results with `./`.
6666+ - (-) But only if you give it an argument of `.`. If you give it the argument `some-directory`, it won't prefix that.
6767+- (-) `realpath --relative-to` doesn't prefix relative paths with `./`.
6868+ - (+) There is no need to return the same result as `realpath`.
6969+7070+</details>
7171+7272+### Representation of the current directory
7373+[curdir]: #representation-of-the-current-directory
7474+7575+Observing: The subpath that produces the base directory can be represented with `.` or `./` or `./.`.
7676+7777+Considering: Paths should be as consistent and unambiguous as possible.
7878+7979+Decision: It should be `./.`.
8080+8181+<details>
8282+<summary>Arguments</summary>
8383+8484+- (+) `./` would be inconsistent with [the decision to not persist trailing slashes][trailing-slashes].
8585+- (-) `.` is how `realpath` normalises paths.
8686+- (+) `.` can be interpreted as a shell command (it's a builtin for sourcing files in `bash` and `zsh`).
8787+- (+) `.` would be the only path without a `/`. It could not be used as a Nix path expression, since those require at least one `/` to be parsed as such.
8888+- (-) `./.` is rather long.
8989+ - (-) We don't require users to type this though, as it's only output by the library.
9090+ As inputs all three variants are supported for subpaths (and we can't do anything about absolute paths)
9191+- (-) `builtins.dirOf "foo" == "."`, so `.` would be consistent with that.
9292+- (+) `./.` is consistent with the [decision to have leading `./`][leading-dots].
9393+- (+) `./.` is a valid Nix path expression, although this property does not hold for every relative path or subpath.
9494+9595+</details>
9696+9797+### Subpath representation
9898+[relrepr]: #subpath-representation
9999+100100+Observing: Subpaths such as `foo/bar` can be represented in various ways:
101101+- string: `"foo/bar"`
102102+- list with all the components: `[ "foo" "bar" ]`
103103+- attribute set: `{ type = "relative-path"; components = [ "foo" "bar" ]; }`
104104+105105+Considering: Paths should be as safe to use as possible. We should generate string outputs in the library and not encourage users to do that themselves.
106106+107107+Decision: Paths are represented as strings.
108108+109109+<details>
110110+<summary>Arguments</summary>
111111+112112+- (+) It's simpler for the users of the library. One doesn't have to convert a path a string before it can be used.
113113+ - (+) Naively converting the list representation to a string with `concatStringsSep "/"` would break for `[]`, requiring library users to be more careful.
114114+- (+) It doesn't encourage people to do their own path processing and instead use the library.
115115+ With a list representation it would seem easy to just use `lib.lists.init` to get the parent directory, but then it breaks for `.`, which would be represented as `[ ]`.
116116+- (+) `+` is convenient and doesn't work on lists and attribute sets.
117117+ - (-) Shouldn't use `+` anyways, we export safer functions for path manipulation.
118118+119119+</details>
120120+121121+### Parent directory
122122+[parents]: #parent-directory
123123+124124+Observing: Relative paths can have `..` components, which refer to the parent directory.
125125+126126+Considering: Paths should be as safe and unambiguous as possible.
127127+128128+Decision: `..` path components in string paths are not supported, neither as inputs nor as outputs. Hence, string paths are called subpaths, rather than relative paths.
129129+130130+<details>
131131+<summary>Arguments</summary>
132132+133133+- (+) If we wanted relative paths to behave according to the "physical" interpretation (as a directory tree with relations between nodes), it would require resolving symlinks, since e.g. `foo/..` would not be the same as `.` if `foo` is a symlink.
134134+ - (-) The "logical" interpretation is also valid (treating paths as a sequence of names), and is used by some software. It is simpler, and not using symlinks at all is safer.
135135+ - (+) Mixing both models can lead to surprises.
136136+ - (+) We can't resolve symlinks without filesystem access.
137137+ - (+) Nix also doesn't support reading symlinks at evaluation time.
138138+ - (-) We could just not handle such cases, e.g. `equals "foo" "foo/bar/.. == false`. The paths are different, we don't need to check whether the paths point to the same thing.
139139+ - (+) Assume we said `relativeTo /foo /bar == "../bar"`. If this is used like `/bar/../foo` in the end, and `bar` turns out to be a symlink to somewhere else, this won't be accurate.
140140+ - (-) We could decide to not support such ambiguous operations, or mark them as such, e.g. the normal `relativeTo` will error on such a case, but there could be `extendedRelativeTo` supporting that.
141141+- (-) `..` are a part of paths, a path library should therefore support it.
142142+ - (+) If we can convincingly argue that all such use cases are better done e.g. with runtime tools, the library not supporting it can nudge people towards using those.
143143+- (-) We could allow "..", but only in the prefix.
144144+ - (+) Then we'd have to throw an error for doing `append /some/path "../foo"`, making it non-composable.
145145+ - (+) The same is for returning paths with `..`: `relativeTo /foo /bar => "../bar"` would produce a non-composable path.
146146+- (+) We argue that `..` is not needed at the Nix evaluation level, since we'd always start evaluation from the project root and don't go up from there.
147147+ - (+) `..` is supported in Nix paths, turning them into absolute paths.
148148+ - (-) This is ambiguous in the presence of symlinks.
149149+- (+) If you need `..` for building or runtime, you can use build-/run-time tooling to create those (e.g. `realpath` with `--relative-to`), or use absolute paths instead.
150150+ This also gives you the ability to correctly handle symlinks.
151151+152152+</details>
153153+154154+### Trailing slashes
155155+[trailing-slashes]: #trailing-slashes
156156+157157+Observing: Subpaths can contain trailing slashes, like `foo/`, indicating that the path points to a directory and not a file.
158158+159159+Considering: Paths should be as consistent as possible, there should only be a single normalisation for the same path.
160160+161161+Decision: All functions remove trailing slashes in their results.
162162+163163+<details>
164164+<summary>Arguments</summary>
165165+166166+- (+) It allows normalisations to be unique, in that there's only a single normalisation for the same path. If trailing slashes were preserved, both `foo/bar` and `foo/bar/` would be valid but different normalisations for the same path.
167167+- Comparison to other frameworks to figure out the least surprising behavior:
168168+ - (+) Nix itself doesn't support trailing slashes when parsing and doesn't preserve them when appending paths.
169169+ - (-) [Rust's std::path](https://doc.rust-lang.org/std/path/index.html) does preserve them during [construction](https://doc.rust-lang.org/std/path/struct.Path.html#method.new).
170170+ - (+) Doesn't preserve them when returning individual [components](https://doc.rust-lang.org/std/path/struct.Path.html#method.components).
171171+ - (+) Doesn't preserve them when [canonicalizing](https://doc.rust-lang.org/std/path/struct.Path.html#method.canonicalize).
172172+ - (+) [Python 3's pathlib](https://docs.python.org/3/library/pathlib.html#module-pathlib) doesn't preserve them during [construction](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath).
173173+ - Notably it represents the individual components as a list internally.
174174+ - (-) [Haskell's filepath](https://hackage.haskell.org/package/filepath-1.4.100.0) has [explicit support](https://hackage.haskell.org/package/filepath-1.4.100.0/docs/System-FilePath.html#g:6) for handling trailing slashes.
175175+ - (-) Does preserve them for [normalisation](https://hackage.haskell.org/package/filepath-1.4.100.0/docs/System-FilePath.html#v:normalise).
176176+ - (-) [NodeJS's Path library](https://nodejs.org/api/path.html) preserves trailing slashes for [normalisation](https://nodejs.org/api/path.html#pathnormalizepath).
177177+ - (+) For [parsing a path](https://nodejs.org/api/path.html#pathparsepath) into its significant elements, trailing slashes are not preserved.
178178+- (+) Nix's builtin function `dirOf` gives an unexpected result for paths with trailing slashes: `dirOf "foo/bar/" == "foo/bar"`.
179179+ Inconsistently, `baseNameOf` works correctly though: `baseNameOf "foo/bar/" == "bar"`.
180180+ - (-) We are writing a path library to improve handling of paths though, so we shouldn't use these functions and discourage their use.
181181+- (-) Unexpected result when normalising intermediate paths, like `relative.normalise ("foo" + "/") + "bar" == "foobar"`.
182182+ - (+) This is not a practical use case though.
183183+ - (+) Don't use `+` to append paths, this library has a `join` function for that.
184184+ - (-) Users might use `+` out of habit though.
185185+- (+) The `realpath` command also removes trailing slashes.
186186+- (+) Even with a trailing slash, the path is the same, it's only an indication that it's a directory.
187187+188188+</details>
189189+190190+## Other implementations and references
191191+192192+- [Rust](https://doc.rust-lang.org/std/path/struct.Path.html)
193193+- [Python](https://docs.python.org/3/library/pathlib.html)
194194+- [Haskell](https://hackage.haskell.org/package/filepath-1.4.100.0/docs/System-FilePath.html)
195195+- [Nodejs](https://nodejs.org/api/path.html)
196196+- [POSIX.1-2017](https://pubs.opengroup.org/onlinepubs/9699919799/nframe.html)
+218
lib/path/default.nix
···11+# Functions for working with paths, see ./path.md
22+{ lib }:
33+let
44+55+ inherit (builtins)
66+ isString
77+ split
88+ match
99+ ;
1010+1111+ inherit (lib.lists)
1212+ length
1313+ head
1414+ last
1515+ genList
1616+ elemAt
1717+ ;
1818+1919+ inherit (lib.strings)
2020+ concatStringsSep
2121+ substring
2222+ ;
2323+2424+ inherit (lib.asserts)
2525+ assertMsg
2626+ ;
2727+2828+ # Return the reason why a subpath is invalid, or `null` if it's valid
2929+ subpathInvalidReason = value:
3030+ if ! isString value then
3131+ "The given value is of type ${builtins.typeOf value}, but a string was expected"
3232+ else if value == "" then
3333+ "The given string is empty"
3434+ else if substring 0 1 value == "/" then
3535+ "The given string \"${value}\" starts with a `/`, representing an absolute path"
3636+ # We don't support ".." components, see ./path.md#parent-directory
3737+ else if match "(.*/)?\\.\\.(/.*)?" value != null then
3838+ "The given string \"${value}\" contains a `..` component, which is not allowed in subpaths"
3939+ else null;
4040+4141+ # Split and normalise a relative path string into its components.
4242+ # Error for ".." components and doesn't include "." components
4343+ splitRelPath = path:
4444+ let
4545+ # Split the string into its parts using regex for efficiency. This regex
4646+ # matches patterns like "/", "/./", "/././", with arbitrarily many "/"s
4747+ # together. These are the main special cases:
4848+ # - Leading "./" gets split into a leading "." part
4949+ # - Trailing "/." or "/" get split into a trailing "." or ""
5050+ # part respectively
5151+ #
5252+ # These are the only cases where "." and "" parts can occur
5353+ parts = split "/+(\\./+)*" path;
5454+5555+ # `split` creates a list of 2 * k + 1 elements, containing the k +
5656+ # 1 parts, interleaved with k matches where k is the number of
5757+ # (non-overlapping) matches. This calculation here gets the number of parts
5858+ # back from the list length
5959+ # floor( (2 * k + 1) / 2 ) + 1 == floor( k + 1/2 ) + 1 == k + 1
6060+ partCount = length parts / 2 + 1;
6161+6262+ # To assemble the final list of components we want to:
6363+ # - Skip a potential leading ".", normalising "./foo" to "foo"
6464+ # - Skip a potential trailing "." or "", normalising "foo/" and "foo/." to
6565+ # "foo". See ./path.md#trailing-slashes
6666+ skipStart = if head parts == "." then 1 else 0;
6767+ skipEnd = if last parts == "." || last parts == "" then 1 else 0;
6868+6969+ # We can now know the length of the result by removing the number of
7070+ # skipped parts from the total number
7171+ componentCount = partCount - skipEnd - skipStart;
7272+7373+ in
7474+ # Special case of a single "." path component. Such a case leaves a
7575+ # componentCount of -1 due to the skipStart/skipEnd not verifying that
7676+ # they don't refer to the same character
7777+ if path == "." then []
7878+7979+ # Generate the result list directly. This is more efficient than a
8080+ # combination of `filter`, `init` and `tail`, because here we don't
8181+ # allocate any intermediate lists
8282+ else genList (index:
8383+ # To get to the element we need to add the number of parts we skip and
8484+ # multiply by two due to the interleaved layout of `parts`
8585+ elemAt parts ((skipStart + index) * 2)
8686+ ) componentCount;
8787+8888+ # Join relative path components together
8989+ joinRelPath = components:
9090+ # Always return relative paths with `./` as a prefix (./path.md#leading-dots-for-relative-paths)
9191+ "./" +
9292+ # An empty string is not a valid relative path, so we need to return a `.` when we have no components
9393+ (if components == [] then "." else concatStringsSep "/" components);
9494+9595+in /* No rec! Add dependencies on this file at the top. */ {
9696+9797+9898+ /* Whether a value is a valid subpath string.
9999+100100+ - The value is a string
101101+102102+ - The string is not empty
103103+104104+ - The string doesn't start with a `/`
105105+106106+ - The string doesn't contain any `..` path components
107107+108108+ Type:
109109+ subpath.isValid :: String -> Bool
110110+111111+ Example:
112112+ # Not a string
113113+ subpath.isValid null
114114+ => false
115115+116116+ # Empty string
117117+ subpath.isValid ""
118118+ => false
119119+120120+ # Absolute path
121121+ subpath.isValid "/foo"
122122+ => false
123123+124124+ # Contains a `..` path component
125125+ subpath.isValid "../foo"
126126+ => false
127127+128128+ # Valid subpath
129129+ subpath.isValid "foo/bar"
130130+ => true
131131+132132+ # Doesn't need to be normalised
133133+ subpath.isValid "./foo//bar/"
134134+ => true
135135+ */
136136+ subpath.isValid = value:
137137+ subpathInvalidReason value == null;
138138+139139+140140+ /* Normalise a subpath. Throw an error if the subpath isn't valid, see
141141+ `lib.path.subpath.isValid`
142142+143143+ - Limit repeating `/` to a single one
144144+145145+ - Remove redundant `.` components
146146+147147+ - Remove trailing `/` and `/.`
148148+149149+ - Add leading `./`
150150+151151+ Laws:
152152+153153+ - (Idempotency) Normalising multiple times gives the same result:
154154+155155+ subpath.normalise (subpath.normalise p) == subpath.normalise p
156156+157157+ - (Uniqueness) There's only a single normalisation for the paths that lead to the same file system node:
158158+159159+ subpath.normalise p != subpath.normalise q -> $(realpath ${p}) != $(realpath ${q})
160160+161161+ - Don't change the result when appended to a Nix path value:
162162+163163+ base + ("/" + p) == base + ("/" + subpath.normalise p)
164164+165165+ - Don't change the path according to `realpath`:
166166+167167+ $(realpath ${p}) == $(realpath ${subpath.normalise p})
168168+169169+ - Only error on invalid subpaths:
170170+171171+ builtins.tryEval (subpath.normalise p)).success == subpath.isValid p
172172+173173+ Type:
174174+ subpath.normalise :: String -> String
175175+176176+ Example:
177177+ # limit repeating `/` to a single one
178178+ subpath.normalise "foo//bar"
179179+ => "./foo/bar"
180180+181181+ # remove redundant `.` components
182182+ subpath.normalise "foo/./bar"
183183+ => "./foo/bar"
184184+185185+ # add leading `./`
186186+ subpath.normalise "foo/bar"
187187+ => "./foo/bar"
188188+189189+ # remove trailing `/`
190190+ subpath.normalise "foo/bar/"
191191+ => "./foo/bar"
192192+193193+ # remove trailing `/.`
194194+ subpath.normalise "foo/bar/."
195195+ => "./foo/bar"
196196+197197+ # Return the current directory as `./.`
198198+ subpath.normalise "."
199199+ => "./."
200200+201201+ # error on `..` path components
202202+ subpath.normalise "foo/../bar"
203203+ => <error>
204204+205205+ # error on empty string
206206+ subpath.normalise ""
207207+ => <error>
208208+209209+ # error on absolute path
210210+ subpath.normalise "/foo"
211211+ => <error>
212212+ */
213213+ subpath.normalise = path:
214214+ assert assertMsg (subpathInvalidReason path == null)
215215+ "lib.path.subpath.normalise: Argument is not a valid subpath string: ${subpathInvalidReason path}";
216216+ joinRelPath (splitRelPath path);
217217+218218+}
···11+# Generate random path-like strings, separated by null characters.
22+#
33+# Invocation:
44+#
55+# awk -f ./generate.awk -v <variable>=<value> | tr '\0' '\n'
66+#
77+# Customizable variables (all default to 0):
88+# - seed: Deterministic random seed to use for generation
99+# - count: Number of paths to generate
1010+# - extradotweight: Give extra weight to dots being generated
1111+# - extraslashweight: Give extra weight to slashes being generated
1212+# - extranullweight: Give extra weight to null being generated, making paths shorter
1313+BEGIN {
1414+ # Random seed, passed explicitly for reproducibility
1515+ srand(seed)
1616+1717+ # Don't include special characters below 32
1818+ minascii = 32
1919+ # Don't include DEL at 128
2020+ maxascii = 127
2121+ upperascii = maxascii - minascii
2222+2323+ # add extra weight for ., in addition to the one weight from the ascii range
2424+ upperdot = upperascii + extradotweight
2525+2626+ # add extra weight for /, in addition to the one weight from the ascii range
2727+ upperslash = upperdot + extraslashweight
2828+2929+ # add extra weight for null, indicating the end of the string
3030+ # Must be at least 1 to have strings end at all
3131+ total = upperslash + 1 + extranullweight
3232+3333+ # new=1 indicates that it's a new string
3434+ new=1
3535+ while (count > 0) {
3636+3737+ # Random integer between [0, total)
3838+ value = int(rand() * total)
3939+4040+ if (value < upperascii) {
4141+ # Ascii range
4242+ printf("%c", value + minascii)
4343+ new=0
4444+4545+ } else if (value < upperdot) {
4646+ # Dot range
4747+ printf "."
4848+ new=0
4949+5050+ } else if (value < upperslash) {
5151+ # If it's the start of a new path, only generate a / in 10% of cases
5252+ # This is always an invalid subpath, which is not a very interesting case
5353+ if (new && rand() > 0.1) continue
5454+ printf "/"
5555+5656+ } else {
5757+ # Do not generate empty strings
5858+ if (new) continue
5959+ printf "\x00"
6060+ count--
6161+ new=1
6262+ }
6363+ }
6464+}
+60
lib/path/tests/prop.nix
···11+# Given a list of path-like strings, check some properties of the path library
22+# using those paths and return a list of attribute sets of the following form:
33+#
44+# { <string> = <lib.path.subpath.normalise string>; }
55+#
66+# If `normalise` fails to evaluate, the attribute value is set to `""`.
77+# If not, the resulting value is normalised again and an appropriate attribute set added to the output list.
88+{
99+ # The path to the nixpkgs lib to use
1010+ libpath,
1111+ # A flat directory containing files with randomly-generated
1212+ # path-like values
1313+ dir,
1414+}:
1515+let
1616+ lib = import libpath;
1717+1818+ # read each file into a string
1919+ strings = map (name:
2020+ builtins.readFile (dir + "/${name}")
2121+ ) (builtins.attrNames (builtins.readDir dir));
2222+2323+ inherit (lib.path.subpath) normalise isValid;
2424+ inherit (lib.asserts) assertMsg;
2525+2626+ normaliseAndCheck = str:
2727+ let
2828+ originalValid = isValid str;
2929+3030+ tryOnce = builtins.tryEval (normalise str);
3131+ tryTwice = builtins.tryEval (normalise tryOnce.value);
3232+3333+ absConcatOrig = /. + ("/" + str);
3434+ absConcatNormalised = /. + ("/" + tryOnce.value);
3535+ in
3636+ # Check the lib.path.subpath.normalise property to only error on invalid subpaths
3737+ assert assertMsg
3838+ (originalValid -> tryOnce.success)
3939+ "Even though string \"${str}\" is valid as a subpath, the normalisation for it failed";
4040+ assert assertMsg
4141+ (! originalValid -> ! tryOnce.success)
4242+ "Even though string \"${str}\" is invalid as a subpath, the normalisation for it succeeded";
4343+4444+ # Check normalisation idempotency
4545+ assert assertMsg
4646+ (originalValid -> tryTwice.success)
4747+ "For valid subpath \"${str}\", the normalisation \"${tryOnce.value}\" was not a valid subpath";
4848+ assert assertMsg
4949+ (originalValid -> tryOnce.value == tryTwice.value)
5050+ "For valid subpath \"${str}\", normalising it once gives \"${tryOnce.value}\" but normalising it twice gives a different result: \"${tryTwice.value}\"";
5151+5252+ # Check that normalisation doesn't change a string when appended to an absolute Nix path value
5353+ assert assertMsg
5454+ (originalValid -> absConcatOrig == absConcatNormalised)
5555+ "For valid subpath \"${str}\", appending to an absolute Nix path value gives \"${absConcatOrig}\", but appending the normalised result \"${tryOnce.value}\" gives a different value \"${absConcatNormalised}\"";
5656+5757+ # Return an empty string when failed
5858+ if tryOnce.success then tryOnce.value else "";
5959+6060+in lib.genAttrs strings normaliseAndCheck
+179
lib/path/tests/prop.sh
···11+#!/usr/bin/env bash
22+33+# Property tests for the `lib.path` library
44+#
55+# It generates random path-like strings and runs the functions on
66+# them, checking that the expected laws of the functions hold
77+88+set -euo pipefail
99+shopt -s inherit_errexit
1010+1111+# https://stackoverflow.com/a/246128
1212+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
1313+1414+if test -z "${TEST_LIB:-}"; then
1515+ TEST_LIB=$SCRIPT_DIR/../..
1616+fi
1717+1818+tmp="$(mktemp -d)"
1919+clean_up() {
2020+ rm -rf "$tmp"
2121+}
2222+trap clean_up EXIT
2323+mkdir -p "$tmp/work"
2424+cd "$tmp/work"
2525+2626+# Defaulting to a random seed but the first argument can override this
2727+seed=${1:-$RANDOM}
2828+echo >&2 "Using seed $seed, use \`lib/path/tests/prop.sh $seed\` to reproduce this result"
2929+3030+# The number of random paths to generate. This specific number was chosen to
3131+# be fast enough while still generating enough variety to detect bugs.
3232+count=500
3333+3434+debug=0
3535+# debug=1 # print some extra info
3636+# debug=2 # print generated values
3737+3838+# Fine tuning parameters to balance the number of generated invalid paths
3939+# to the variance in generated paths.
4040+extradotweight=64 # Larger value: more dots
4141+extraslashweight=64 # Larger value: more slashes
4242+extranullweight=16 # Larger value: shorter strings
4343+4444+die() {
4545+ echo >&2 "test case failed: " "$@"
4646+ exit 1
4747+}
4848+4949+if [[ "$debug" -ge 1 ]]; then
5050+ echo >&2 "Generating $count random path-like strings"
5151+fi
5252+5353+# Read stream of null-terminated strings entry-by-entry into bash,
5454+# write it to a file and the `strings` array.
5555+declare -a strings=()
5656+mkdir -p "$tmp/strings"
5757+while IFS= read -r -d $'\0' str; do
5858+ echo -n "$str" > "$tmp/strings/${#strings[@]}"
5959+ strings+=("$str")
6060+done < <(awk \
6161+ -f "$SCRIPT_DIR"/generate.awk \
6262+ -v seed="$seed" \
6363+ -v count="$count" \
6464+ -v extradotweight="$extradotweight" \
6565+ -v extraslashweight="$extraslashweight" \
6666+ -v extranullweight="$extranullweight")
6767+6868+if [[ "$debug" -ge 1 ]]; then
6969+ echo >&2 "Trying to normalise the generated path-like strings with Nix"
7070+fi
7171+7272+# Precalculate all normalisations with a single Nix call. Calling Nix for each
7373+# string individually would take way too long
7474+nix-instantiate --eval --strict --json \
7575+ --argstr libpath "$TEST_LIB" \
7676+ --argstr dir "$tmp/strings" \
7777+ "$SCRIPT_DIR"/prop.nix \
7878+ >"$tmp/result.json"
7979+8080+# Uses some jq magic to turn the resulting attribute set into an associative
8181+# bash array assignment
8282+declare -A normalised_result="($(jq '
8383+ to_entries
8484+ | map("[\(.key | @sh)]=\(.value | @sh)")
8585+ | join(" \n")' -r < "$tmp/result.json"))"
8686+8787+# Looks up a normalisation result for a string
8888+# Checks that the normalisation is only failing iff it's an invalid subpath
8989+# For valid subpaths, returns 0 and prints the normalisation result
9090+# For invalid subpaths, returns 1
9191+normalise() {
9292+ local str=$1
9393+ # Uses the same check for validity as in the library implementation
9494+ if [[ "$str" == "" || "$str" == /* || "$str" =~ ^(.*/)?\.\.(/.*)?$ ]]; then
9595+ valid=
9696+ else
9797+ valid=1
9898+ fi
9999+100100+ normalised=${normalised_result[$str]}
101101+ # An empty string indicates failure, this is encoded in ./prop.nix
102102+ if [[ -n "$normalised" ]]; then
103103+ if [[ -n "$valid" ]]; then
104104+ echo "$normalised"
105105+ else
106106+ die "For invalid subpath \"$str\", lib.path.subpath.normalise returned this result: \"$normalised\""
107107+ fi
108108+ else
109109+ if [[ -n "$valid" ]]; then
110110+ die "For valid subpath \"$str\", lib.path.subpath.normalise failed"
111111+ else
112112+ if [[ "$debug" -ge 2 ]]; then
113113+ echo >&2 "String \"$str\" is not a valid subpath"
114114+ fi
115115+ # Invalid and it correctly failed, we let the caller continue if they catch the exit code
116116+ return 1
117117+ fi
118118+ fi
119119+}
120120+121121+# Intermediate result populated by test_idempotency_realpath
122122+# and used in test_normalise_uniqueness
123123+#
124124+# Contains a mapping from a normalised subpath to the realpath result it represents
125125+declare -A norm_to_real
126126+127127+test_idempotency_realpath() {
128128+ if [[ "$debug" -ge 1 ]]; then
129129+ echo >&2 "Checking idempotency of each result and making sure the realpath result isn't changed"
130130+ fi
131131+132132+ # Count invalid subpaths to display stats
133133+ invalid=0
134134+ for str in "${strings[@]}"; do
135135+ if ! result=$(normalise "$str"); then
136136+ ((invalid++)) || true
137137+ continue
138138+ fi
139139+140140+ # Check the law that it doesn't change the result of a realpath
141141+ mkdir -p -- "$str" "$result"
142142+ real_orig=$(realpath -- "$str")
143143+ real_norm=$(realpath -- "$result")
144144+145145+ if [[ "$real_orig" != "$real_norm" ]]; then
146146+ die "realpath of the original string \"$str\" (\"$real_orig\") is not the same as realpath of the normalisation \"$result\" (\"$real_norm\")"
147147+ fi
148148+149149+ if [[ "$debug" -ge 2 ]]; then
150150+ echo >&2 "String \"$str\" gets normalised to \"$result\" and file path \"$real_orig\""
151151+ fi
152152+ norm_to_real["$result"]="$real_orig"
153153+ done
154154+ if [[ "$debug" -ge 1 ]]; then
155155+ echo >&2 "$(bc <<< "scale=1; 100 / $count * $invalid")% of the total $count generated strings were invalid subpath strings, and were therefore ignored"
156156+ fi
157157+}
158158+159159+test_normalise_uniqueness() {
160160+ if [[ "$debug" -ge 1 ]]; then
161161+ echo >&2 "Checking for the uniqueness law"
162162+ fi
163163+164164+ for norm_p in "${!norm_to_real[@]}"; do
165165+ real_p=${norm_to_real["$norm_p"]}
166166+ for norm_q in "${!norm_to_real[@]}"; do
167167+ real_q=${norm_to_real["$norm_q"]}
168168+ # Checks normalisation uniqueness law for each pair of values
169169+ if [[ "$norm_p" != "$norm_q" && "$real_p" == "$real_q" ]]; then
170170+ die "Normalisations \"$norm_p\" and \"$norm_q\" are different, but the realpath of them is the same: \"$real_p\""
171171+ fi
172172+ done
173173+ done
174174+}
175175+176176+test_idempotency_realpath
177177+test_normalise_uniqueness
178178+179179+echo >&2 tests ok
···360360 </listitem>
361361 <listitem>
362362 <para>
363363+ <literal>services.chronyd</literal> is now started with
364364+ additional systemd sandbox/hardening options for better
365365+ security.
366366+ </para>
367367+ </listitem>
368368+ <listitem>
369369+ <para>
363370 The module <literal>services.headscale</literal> was
364371 refactored to be compliant with
365372 <link xlink:href="https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md">RFC
+2
nixos/doc/manual/release-notes/rl-2305.section.md
···98989999 And backup your data.
100100101101+- `services.chronyd` is now started with additional systemd sandbox/hardening options for better security.
102102+101103- The module `services.headscale` was refactored to be compliant with [RFC 0042](https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md). To be precise, this means that the following things have changed:
102104103105 - Most settings has been migrated under [services.headscale.settings](#opt-services.headscale.settings) which is an attribute-set that
+1-1
nixos/modules/programs/gnupg.nix
···135135 # The SSH agent protocol doesn't have support for changing TTYs; however we
136136 # can simulate this with the `exec` feature of openssh (see ssh_config(5))
137137 # that hooks a command to the shell currently running the ssh program.
138138- Match host * exec "${cfg.package}/bin/gpg-connect-agent --quiet updatestartuptty /bye >/dev/null 2>&1"
138138+ Match host * exec "${pkgs.runtimeShell} -c '${cfg.package}/bin/gpg-connect-agent --quiet updatestartuptty /bye >/dev/null 2>&1'"
139139 '';
140140141141 environment.extraInit = mkIf cfg.agent.enableSSHSupport ''
···11-{ fetchFromGitHub, lib, i3 }:
22-33-i3.overrideAttrs (oldAttrs : rec {
44- pname = "i3-gaps";
55- version = "4.21.1";
66-77- src = fetchFromGitHub {
88- owner = "Airblader";
99- repo = "i3";
1010- rev = version;
1111- sha256 = "sha256-+JxJjvzEuAA4CH+gufzAzIqd5BSvHtPvLm2zTfXc/xk=";
1212- };
1313-1414- meta = with lib; {
1515- description = "A fork of the i3 tiling window manager with some additional features";
1616- homepage = "https://github.com/Airblader/i3";
1717- maintainers = with maintainers; [ fmthoma ];
1818- license = licenses.bsd3;
1919- platforms = platforms.linux ++ platforms.netbsd ++ platforms.openbsd;
2020-2121- longDescription = ''
2222- Fork of i3wm, a tiling window manager primarily targeted at advanced users
2323- and developers. Based on a tree as data structure, supports tiling,
2424- stacking, and tabbing layouts, handled dynamically, as well as floating
2525- windows. This fork adds a few features such as gaps between windows.
2626- Configured via plain text file. Multi-monitor. UTF-8 clean.
2727- '';
2828- };
2929-})
+115-76
pkgs/data/fonts/noto-fonts/default.nix
···1111, imagemagick
1212, zopfli
1313, buildPackages
1414+, variants ? [ ]
1415}:
1515-1616let
1717- mkNoto = { pname, weights }:
1818- stdenvNoCC.mkDerivation {
1717+ notoLongDescription = ''
1818+ When text is rendered by a computer, sometimes characters are
1919+ displayed as “tofu”. They are little boxes to indicate your device
2020+ doesn’t have a font to display the text.
2121+2222+ Google has been developing a font family called Noto, which aims to
2323+ support all languages with a harmonious look and feel. Noto is
2424+ Google’s answer to tofu. The name noto is to convey the idea that
2525+ Google’s goal is to see “no more tofu”. Noto has multiple styles and
2626+ weights, and freely available to all.
2727+2828+ This package also includes the Arimo, Cousine, and Tinos fonts.
2929+ '';
3030+in
3131+rec {
3232+ mkNoto =
3333+ { pname
3434+ , weights
3535+ , variants ? [ ]
3636+ , longDescription ? notoLongDescription
3737+ }:
3838+ stdenvNoCC.mkDerivation rec {
1939 inherit pname;
2020- version = "2020-01-23";
4040+ version = "20201206-phase3";
21412242 src = fetchFromGitHub {
2343 owner = "googlefonts";
2444 repo = "noto-fonts";
2525- rev = "f4726a2ec36169abd02a6d8abe67c8ff0236f6d8";
2626- sha256 = "0zc1r7zph62qmvzxqfflsprazjf6x1qnwc2ma27kyzh6v36gaykw";
4545+ rev = "v${version}";
4646+ hash = "sha256-x60RvCRFLoGe0CNvswROnDkIsUFbWH+/laN8q2qkUPk=";
2747 };
4848+4949+ _variants = map (variant: builtins.replaceStrings [ " " ] [ "" ] variant) variants;
28502951 installPhase = ''
3052 # We copy in reverse preference order -- unhinted first, then
···3355 #
3456 # TODO: install OpenType, variable versions?
3557 local out_ttf=$out/share/fonts/truetype/noto
3636- install -m444 -Dt $out_ttf phaseIII_only/unhinted/ttf/*/*-${weights}.ttf
3737- install -m444 -Dt $out_ttf phaseIII_only/hinted/ttf/*/*-${weights}.ttf
3838- install -m444 -Dt $out_ttf unhinted/*/*-${weights}.ttf
3939- install -m444 -Dt $out_ttf hinted/*/*-${weights}.ttf
4040- '';
5858+ '' + (if _variants == [ ] then ''
5959+ install -m444 -Dt $out_ttf archive/unhinted/*/*-${weights}.ttf
6060+ install -m444 -Dt $out_ttf archive/hinted/*/*-${weights}.ttf
6161+ install -m444 -Dt $out_ttf unhinted/*/*/*-${weights}.ttf
6262+ install -m444 -Dt $out_ttf hinted/*/*/*-${weights}.ttf
6363+ '' else ''
6464+ for variant in $_variants; do
6565+ install -m444 -Dt $out_ttf archive/unhinted/$variant/*-${weights}.ttf
6666+ install -m444 -Dt $out_ttf archive/hinted/$variant/*-${weights}.ttf
6767+ install -m444 -Dt $out_ttf unhinted/*/$variant/*-${weights}.ttf
6868+ install -m444 -Dt $out_ttf hinted/*/$variant/*-${weights}.ttf
6969+ done
7070+ '');
41714272 meta = with lib; {
4373 description = "Beautiful and free fonts for many languages";
4474 homepage = "https://www.google.com/get/noto/";
4545- longDescription =
4646- ''
4747- When text is rendered by a computer, sometimes characters are
4848- displayed as “tofu”. They are little boxes to indicate your device
4949- doesn’t have a font to display the text.
5050-5151- Google has been developing a font family called Noto, which aims to
5252- support all languages with a harmonious look and feel. Noto is
5353- Google’s answer to tofu. The name noto is to convey the idea that
5454- Google’s goal is to see “no more tofu”. Noto has multiple styles and
5555- weights, and freely available to all.
5656-5757- This package also includes the Arimo, Cousine, and Tinos fonts.
5858- '';
7575+ inherit longDescription;
5976 license = licenses.ofl;
6077 platforms = platforms.all;
6178 maintainers = with maintainers; [ mathnerd314 emily ];
···100117 maintainers = with maintainers; [ mathnerd314 emily ];
101118 };
102119 };
103103-in
104120105105-{
106121 noto-fonts = mkNoto {
107122 pname = "noto-fonts";
108123 weights = "{Regular,Bold,Light,Italic,BoldItalic,LightItalic}";
109124 };
110125126126+ noto-fonts-lgc-plus = mkNoto {
127127+ pname = "noto-fonts-lgc-plus";
128128+ weights = "{Regular,Bold,Light,Italic,BoldItalic,LightItalic}";
129129+ variants = [
130130+ "Noto Sans"
131131+ "Noto Serif"
132132+ "Noto Sans Display"
133133+ "Noto Serif Display"
134134+ "Noto Sans Mono"
135135+ "Noto Music"
136136+ "Noto Sans Symbols"
137137+ "Noto Sans Symbols 2"
138138+ "Noto Sans Math"
139139+ ];
140140+ longDescription = ''
141141+ This package provides the Noto Fonts, but only for latin, greek
142142+ and cyrillic scripts, as well as some extra fonts. To create a
143143+ custom Noto package with custom variants, see the `mkNoto`
144144+ helper function.
145145+ '';
146146+ };
147147+111148 noto-fonts-extra = mkNoto {
112149 pname = "noto-fonts-extra";
113150 weights = "{Black,Condensed,Extra,Medium,Semi,Thin}*";
···127164 sha256 = "sha256-1w66Ge7DZjbONGhxSz69uFhfsjMsDiDkrGl6NsoB7dY=";
128165 };
129166130130- noto-fonts-emoji = let
131131- version = "2.038";
132132- emojiPythonEnv =
133133- buildPackages.python3.withPackages (p: with p; [ fonttools nototools ]);
134134- in stdenvNoCC.mkDerivation {
135135- pname = "noto-fonts-emoji";
136136- inherit version;
167167+ noto-fonts-emoji =
168168+ let
169169+ version = "2.038";
170170+ emojiPythonEnv =
171171+ buildPackages.python3.withPackages (p: with p; [ fonttools nototools ]);
172172+ in
173173+ stdenvNoCC.mkDerivation {
174174+ pname = "noto-fonts-emoji";
175175+ inherit version;
137176138138- src = fetchFromGitHub {
139139- owner = "googlefonts";
140140- repo = "noto-emoji";
141141- rev = "v${version}";
142142- sha256 = "1rgmcc6nqq805iqr8kvxxlk5cf50q714xaxk3ld6rjrd69kb8ix9";
143143- };
177177+ src = fetchFromGitHub {
178178+ owner = "googlefonts";
179179+ repo = "noto-emoji";
180180+ rev = "v${version}";
181181+ sha256 = "1rgmcc6nqq805iqr8kvxxlk5cf50q714xaxk3ld6rjrd69kb8ix9";
182182+ };
144183145145- depsBuildBuild = [
146146- buildPackages.stdenv.cc
147147- pkg-config
148148- cairo
149149- ];
184184+ depsBuildBuild = [
185185+ buildPackages.stdenv.cc
186186+ pkg-config
187187+ cairo
188188+ ];
150189151151- nativeBuildInputs = [
152152- imagemagick
153153- zopfli
154154- pngquant
155155- which
156156- emojiPythonEnv
157157- ];
190190+ nativeBuildInputs = [
191191+ imagemagick
192192+ zopfli
193193+ pngquant
194194+ which
195195+ emojiPythonEnv
196196+ ];
158197159159- postPatch = ''
160160- patchShebangs *.py
161161- patchShebangs third_party/color_emoji/*.py
162162- # remove check for virtualenv, since we handle
163163- # python requirements using python.withPackages
164164- sed -i '/ifndef VIRTUAL_ENV/,+2d' Makefile
198198+ postPatch = ''
199199+ patchShebangs *.py
200200+ patchShebangs third_party/color_emoji/*.py
201201+ # remove check for virtualenv, since we handle
202202+ # python requirements using python.withPackages
203203+ sed -i '/ifndef VIRTUAL_ENV/,+2d' Makefile
165204166166- # Make the build verbose so it won't get culled by Hydra thinking that
167167- # it somehow got stuck doing nothing.
168168- sed -i 's;\t@;\t;' Makefile
169169- '';
205205+ # Make the build verbose so it won't get culled by Hydra thinking that
206206+ # it somehow got stuck doing nothing.
207207+ sed -i 's;\t@;\t;' Makefile
208208+ '';
170209171171- enableParallelBuilding = true;
210210+ enableParallelBuilding = true;
172211173173- installPhase = ''
174174- runHook preInstall
175175- mkdir -p $out/share/fonts/noto
176176- cp NotoColorEmoji.ttf $out/share/fonts/noto
177177- runHook postInstall
178178- '';
212212+ installPhase = ''
213213+ runHook preInstall
214214+ mkdir -p $out/share/fonts/noto
215215+ cp NotoColorEmoji.ttf $out/share/fonts/noto
216216+ runHook postInstall
217217+ '';
179218180180- meta = with lib; {
181181- description = "Color and Black-and-White emoji fonts";
182182- homepage = "https://github.com/googlefonts/noto-emoji";
183183- license = with licenses; [ ofl asl20 ];
184184- platforms = platforms.all;
185185- maintainers = with maintainers; [ mathnerd314 sternenseemann ];
219219+ meta = with lib; {
220220+ description = "Color and Black-and-White emoji fonts";
221221+ homepage = "https://github.com/googlefonts/noto-emoji";
222222+ license = with licenses; [ ofl asl20 ];
223223+ platforms = platforms.all;
224224+ maintainers = with maintainers; [ mathnerd314 sternenseemann ];
225225+ };
186226 };
187187- };
188227189228 noto-fonts-emoji-blob-bin =
190229 let
···633633634634 ### I ###
635635636636+ i3-gaps = i3; # Added 2023-01-03
636637 i3cat = throw "i3cat has been dropped due to the lack of maintanence from upstream since 2016"; # Added 2022-06-02
637638 iana_etc = throw "'iana_etc' has been renamed to/replaced by 'iana-etc'"; # Converted to throw 2022-02-22
638639 iasl = throw "iasl has been removed, use acpica-tools instead"; # Added 2021-08-08