tjh.dev
nixpkgs mirror (for testing)
github.com/NixOS/nixpkgs
nix
1# K3s Usage
2
3## Single Node
4
5```
6{
7 networking.firewall.allowedTCPPorts = [
8 6443 # k3s: required so that pods can reach the API server (running on port 6443 by default)
9 # 2379 # k3s, etcd clients: required if using a "High Availability Embedded etcd" configuration
10 # 2380 # k3s, etcd peers: required if using a "High Availability Embedded etcd" configuration
11 ];
12 networking.firewall.allowedUDPPorts = [
13 # 8472 # k3s, flannel: required if using multi-node for inter-node networking
14 ];
15 services.k3s.enable = true;
16 services.k3s.role = "server";
17 services.k3s.extraFlags = toString [
18 # "--debug" # Optionally add additional args to k3s
19 ];
20}
21```
22
23Once the above changes are active, you can access your cluster through `sudo k3s kubectl` (e.g. `sudo k3s kubectl cluster-info`) or by using the generated kubeconfig file in `/etc/rancher/k3s/k3s.yaml`.
24Multi-node setup
25
26## Multi-Node
27
28it is simple to create a cluster of multiple nodes in a highly available setup (all nodes are in the control-plane and are a part of the etcd cluster).
29
30The first node is configured like this:
31
32```
33{
34 services.k3s = {
35 enable = true;
36 role = "server";
37 token = "<randomized common secret>";
38 clusterInit = true;
39 };
40}
41```
42
43Any other subsequent nodes can be added with a slightly different config:
44
45```
46{
47 services.k3s = {
48 enable = true;
49 role = "server"; # Or "agent" for worker only nodes
50 token = "<randomized common secret>";
51 serverAddr = "https://<ip of first node>:6443";
52 };
53}
54```
55
56For this to work you need to open the aforementioned API, etcd, and flannel ports in the firewall. Official documentation on what ports need to be opened for specific use cases can be found on [k3s' documentation site](https://docs.k3s.io/installation/requirements#inbound-rules-for-k3s-nodes). Note that it is [recommended](https://etcd.io/docs/v3.3/faq/#why-an-odd-number-of-cluster-members) to use an odd number of nodes in such a cluster.
57
58Tip: If you run into connectivity issues between nodes for specific applications (e.g. ingress controller), please verify the firewall settings you have enabled (example under [Single Node](#single-node)) against the documentation for that specific application. In the ingress controller example, you may want to open 443 or 80 depending on your use case.
59
60## Quirks
61
62### `prefer-bundled-bin`
63
64K3s has a config setting `prefer-bundled-bin` (and CLI flag `--prefer-bundled-bin`) that makes k3s use binaries from the `/var/lib/rancher/k3s/data/current/bin/aux/` directory, as unpacked by the k3s binary, before the system `$PATH`.
65This works with the official distribution of k3s but not with the package from nixpkgs, as it does not bundle the upstream binaries from [`k3s-root`](https://github.com/k3s-io/k3s-root) into the k3s binary.
66Thus the `prefer-bundled-bin` setting **cannot** be used to work around issues (like [this `mount` regression](https://github.com/util-linux/util-linux/issues/3474)) with binaries used/called by the kubelet.
67
68### Building from a different source
69
70Because the package is split into multiple derivations and the build process is generally more complex, it is not very obvious how to build k3s from a different source (fork or arbitrary commit).
71
72To build k3s from a different source, you must use `.override` together with `overrideBundleAttrs` (for the k3sBundle derivation) and another `.overrideAttrs` (for the final derivation):
73
74```nix
75{ fetchgit, k3s }:
76let
77 k3sRepo = fetchgit {
78 url = "https://github.com/k3s-io/k3s";
79 rev = "99d91538b1327da933356c318dc8040335fbb66c";
80 hash = "sha256-vVqZzVp0Tea27s8HDVq4SgqlbHBdZcFzNKmPFi0Yktk=";
81 };
82 vendorHash = "sha256-jrPVY+FVZV9wlbik/I35W8ChcLrHlYbLAwUYU16mJLM=";
83in
84(k3s.override {
85 overrideBundleAttrs = {
86 src = k3sRepo;
87 inherit vendorHash;
88 };
89}).overrideAttrs
90 {
91 src = k3sRepo;
92 inherit vendorHash;
93 }
94```
95
96- Additionally to `overrideBundleAttrs` there are also: `overrideCniPluginsAttrs` and `overrideContainerdAttrs`.
97- `k3s --version` still prints the commit SHA (`k3sCommit` passed into `builder.nix`) from the "base" package instead of the actually used `rev`.
98- Depending on the changes made in the fork / commit, the `k3s.override` (without the `overrideAttrs` of the final derivation) might already be enough.
99- If the commit is for a different version of k3s, make sure to use the correct "base" package, e.g., `k3s_1_31.override`. Otherwise the build fails with `Tagged version 'v1.33.1+k3s1' does not match expected version 'v1.31.9[+-]*'`
100- When adding an entirely new k3s version by calling `builder.nix`, keep in mind that the `k3sCommit` parameter is not used as the `k3sRepo` `rev` (it uses `v${k3sVersion}`). Therefore, you additionally must override the package, as shown above.