Something went wrong. Try again.
simple and declarative nixos router
Something went wrong. Try again.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645// FIXME: text too dense// spend more time on what nixos can bring?// e.g. benifets of using nixos for routing// the code is kinda useless with all the explanation?// memory consumption of BIRD, maybe add why the global routing table is// needed and when default routes is necessary// figures?// more background on routing basica?// clarify how this talk is related to nix (very first slide?) up until p.10 its not like a nix talk// bring page 7 to the top? be clear on what we want to achive with nix// explain what bird is (background)// add motivation for the first slide or second?// comparison with ubuntu?// why use nix for managing config? bird is text only// add story, what is the debug chanllange, how i fixed it// after motivation add challnge// less implementation details?// mention that options added are close to 1-1 match with bird official docs//// for challenges e.g. why not exporting routes directly to kernel//// 1. motivation// 2. some info about networking (if necessary to introduce following challenges)// 3. challenges// 4. overview of solutions// 5. solution 1// 6. solution 2// 7. ...// 8. summary (can be the same as overview of solutions)// kazuki says:1. motivation// 2. some info about networking which are necessary to introduce challenges// 3. challenges <<<<<<// 4. some info about networking which are necessary to introduce solutions// 5. overview of solutions// 6. solution 1// 7. solution 2// 8. ...// last slide. summary (can be the same as overview of solutions)// motivation here is "want to use Nix as a templating engine for bird config"// do it more adhoc? instead of dump background beforehand//// adrien:// give a little introduction about myself at the beginning// involvement in community, motivation for the project, etc.// missing main topic? what am i trying to do here// its too network oriented, some audience might not// slide 23 what's NAT// add my motivation in lessons learedn (like entirely possible to run router with nixos)// when introducing new terms at very beginning, add a small explanation when its mentioned down the line// slide 9 maybe give a recap on what bgp is// what's the exact contribution? make it clear (i made a new module to configure bird)// good outcode: bird module exist, current way hard to use, my module makes it easier, audience would want to try
#import "@preview/muchpdf:0.1.2": muchpdf#import "@preview/polylux:0.4.0": *
#let title = "Internet scale routing with NixOS"#let author = "Yifei Sun"#let date = datetime(year: 2025, month: 9, day: 6)#let conference = "NixCon 2025"
#set document(title: title, author: author, date: date)
#set text(size: 20pt)#set page( paper: "presentation-16-9", margin: 2cm, footer: context[ #set text(size: 12pt) #set align(horizon) #if here().page() == 1 [ #conference $dot.c$ #date.display("[month repr:long] [day padding:none], [year]") ] else if here().page() == counter(page).final().at(0) [ ] else [ #author $dot.c$ #conference $dot.c$ #date.display("[month repr:short]. [day padding:none], [year]") #h(1fr) #toolbox.slide-number / #toolbox.last-slide-number ] ],)
#slide[ = #title
#v(1em) #set text(size: 20pt)
Yifei Sun
Inria, ENS de Lyon, Université Grenoble Alpes
#box(image("inria.png", height: 11%)) #h(1em) #box(move(dx: 0pt, dy: 15pt, image("ensl.png", height: 12.5%))) #h(1em) #box(move(dx: 0pt, dy: 9pt, image("uga.png", height: 11.5%)))]
#slide[ == Why?
#v(1fr) #set align(center) // awesome use of free will *(O\_O)* #v(1fr)]
#slide[ == Routing daemon
Software that implements *routing protocols* to *exchange routing information* with other routers and *maintain routing tables*.
#v(2em)
*BIRD*: BIRD Internet Routing Daemon // recursive acronym]
#slide[== Problem
NixOS module option `services.bird.config` is text only
#v(2em)
#toolbox.side-by-side[ - Maintenance burden - Hard to read - Hard to write - Hard to debug
- Hard to compose and reuse][ #image("meme.png")]]
#slide[ == Solution
#set text(size: 28pt) #set align(center) #v(4em) *Parameterizing BIRD configuration with NixOS options*]
#slide[ == Prerequisites
- Nix and NixOS
#v(1em)
- Some netwoking knowledge - systemd-networkd - nftables #footnote[Linux Kernel packet classification framework]
#v(1em)
- BIRD - Tailscale #footnote[Open source mesh VPN software]]
#slide[ == Background
#set text(size: 28pt) #set align(center) #v(3em) *How the internet works*
#set text(size: 22pt) #v(2em) Resource acquisition
Routing security
Finding an upstream
Pingable!]
#slide[ == Acquisition #footnote[Or dn42, ...]
#toolbox.side-by-side[ *ASN*: // unique identifier for each autonomous system (simply put, an AS is a network or a group of networks under a unified routing policy) - RIR #footnote[nro.net/about/rirs] direct assignment or LIR sponsorship ][ *IP address*: // identifier for each network interface - Direct assignment - Lease/purchase from third-party ]
#set align(center) #image("rir.jpg", width: 49.90%)]
#slide[ == Routing security
#toolbox.side-by-side[ *RPKI*: signed authorization objects hosted by registries
*ROA* - Which AS can announce the prefix under some max length - Only registries can host ROAs - X.509 cert signed objects ][ *IRR*: database of routing policies, hosted by registries and other entities
*IRR objects* - Mostly who can announce the prefix, who is customer, who is provider, etc. - Queryable - Aside from registries, known entities can also host IRR (NTT, RADB, etc.) ] #set align(center) Add objects to for your ASN & prefix
ROA
ROUTE/ROUTE6, AS-SET, etc.]
#slide[ == Getting connected
#toolbox.side-by-side[ - Physical presence in datacenter - Hurricane Electric - Cogent - Equinix - ... ][ - Virtual presence at cloud provider that provide IP transit - Vultr, V.PS, Neptune Networks, etc. - https://bgp.services
#v(3em)
- Any VPS + virtual IX or transit providers - Cloudflare Magic Transit - https://bgp.exchange - https://route64.org ]]
#slide[ == Going live
- Setting up BGP session - Get routes from upstream - Default route (0.0.0.0/0, ::/0) - Full table (usually not necessary) - BIRD: 250MB+ for \~1M IPv4 routes + \~230K IPv6 routes) - May be smaller or larger depending on routing daemon
- Routing policies - Import/export - Filtering - ...
- Add address within announced prefix to interface]
#slide[ == Implementation
#set text(size: 28pt) #set align(center) #v(3em) *Parameterizing BIRD configuration*
#set text(size: 22pt) #v(2em) RPKI and filters
Export routes to kernel
Static routes and BGP sessions
Internal routing with Tailscale + bonus ;)]
#slide[== RPKI and filters
#toolbox.side-by-side[*RPKI*- BIRD `rpki` protocol and `roa` table- Defining `options.router.rpki` - v4/v6 table and filter names - retry, refresh, expiration times - list of validators][ *Pre-defined filters* - Renamable ROA filters based on route RPIKI status - Can be referred later in import/export filters - Will support arbitrary filters in the future]
#set align(center)```nixrouter.rpki.validators = [{ id = 0; remote = "rtr.rpki.cloudflare.com"; port = 8282;}];```]
#slide[== RPKI and filters
```nixservices.bird.config = lib.mkOrder <int> '' ${lib.concatMapStringsSep "\n\n" (validator: '' protocol rpki rpki${toString validator.id} { remote "${validator.remote}" port ${toString validator.port};
retry keep ${toString cfg.router.rpki.retry}; refresh keep ${toString cfg.router.rpki.refresh}; expire ${toString cfg.router.rpki.expire}; }'') cfg.router.rpki.validators}'';```]
#slide[== RPKI and filters
#set text(size: 14pt)
```birdroa6 table trpki6;
protocol rpki rpki0 { roa6 { table trpki6; }; remote "rtr.rpki.cloudflare.com" port 8282; retry ...; refresh ...; expire ...;}
protocol rpki rpki1 { roa6 { table trpki6; }; remote "r3k.zrh2.v.rpki.daknob.net" port 3323; retry ...; refresh ...; expire ...;}
filter validated6 { if (roa_check(trpki6, net, bgp_path.last) = ROA_INVALID) then { print "Ignore RPKI invalid ", net, " for ASN ", bgp_path.last; reject; } accept;}```]
#slide[== Kernel protocol
#toolbox.side-by-side[Export routes to kernel- Default route: most common- Full table: 250MB+ from BIRD, and another copy in kernel- Manual: don't export routes but set `networking.*` options// can work but might break][```nixoptions.router.kernel = { ipv4 = { import = lib.mkOption { ... }; export = lib.mkOption { ... }; }; ipv6 = { import = lib.mkOption { ... }; export = lib.mkOption { ... }; };};```]]
#slide[== Static routes
#toolbox.side-by-side[Manually configured routes
#set text(size: 14pt)```nixoptions.router.static.ipv4.routes = lib.mkOption { type = lib.types.listOf (lib.types.submodule { options = { prefix = ...; option = ...; }; });};```][#set text(size: 15pt)```nixrouter.static = { ipv4.routes = [ { prefix = "0.0.0.0/0"; option = "via 198.51.100.1"; } { prefix = "203.0.113.0/24"; option = "blackhole"; } ]; ipv6.routes = [{ prefix = "2001:db8::/32"; option = "reject"; }];};
# config.services.bird.configlib.concatMapStringsSep "\n " (r: "route ${r.prefix} ${r.option};") cfg.router.static.ipv4.routes```]]
#slide[== BPG sessions
#toolbox.side-by-side[- The usual: 1 session per protocol- MP-BGP: 1 session for both v4 and v6
#set text(size: 15pt)```nixoptions.router.sessions = lib.mkOption { type = lib.types.listOf (lib.types.submodule { options = { name = ...; password = ...; # null for don't validate type = ...; # disable, direct, multihop mp = ...; # null, v4 over v6, v6 over v4 neighbor = ...; # ASN, IPv4, IPv6 import = ...; # IPv4/IPv6 import filter export = ...; # IPv4/IPv6 export filter ... }; });};```][#set text(size: 15pt)```nix# examplerouter.sessions = [{ name = "somebody"; password = null; type.ipv4 = "disabled"; type.ipv6 = "multihop"; mp = "v4 over v6"; neighbor = { asn = 65536; ipv4 = null; ipv6 = "2001:db8::1"; }; import.ipv4 = "import none;"; import.ipv6 = "import none;"; export.ipv4 = "export all;"; export.ipv6 = "export all;";}];```]]
#slide[== Adding announced prefixes to interfaces
#toolbox.side-by-side[- Enable forwarding- Use `dummy` interface (or use the main interface or whatever)- Disable `ManageForeignRoutes` in systemd-networkd (will delete routes exported by BIRD)][#set text(size: 15pt)```nix boot.kernelModules = [ "dummy" ];
boot.kernel.sysctl = { "net.ipv4.conf.default.forwarding" = 1; "net.ipv6.conf.default.forwarding" = 1; };
systemd.network.config = { networkConfig.ManageForeignRoutes = false; };```]]
#slide[== Adding announced prefixes to interfaces
#toolbox.side-by-side[- Use `systemd.network.netdevs` to configure virtual interfaces- Use `systemd.network.networks` to configure addresses and routing policies - `routingPolicyRules` is only needed when the outbound gateway of the announced prefixes is different from the default gateway of the main interface][#set text(size: 15pt)```nix systemd.network.netdevs = { "40-dummy0".netdevConfig = { Kind = "dummy"; Name = "dummy0"; }; };
systemd.network.networks = { "40-dummy0" = { name = "dummy0"; address = ipv4.addresses ++ ipv6.addresses; routingPolicyRules = ...; }; };```]]
#slide[== Immediate benefit
*Parametericity*
#set text(size: 18pt)```nixconfig = lib.mkIf (options?router && lib.elem config.a.b.c.bind.v4 config.services.router.local.ipv4.addresses && lib.elem config.a.b.c.bind.v6 config.services.router.local.ipv6.addresses) { ... };```]
#slide[ == Multiple upstreams?
#set align(center) #muchpdf(read("ir.pdf", encoding: none), scale: 2.6)]
#slide[== Internal routing with Tailscale #footnote[tailscale.com/blog/how-tailscale-works]
#toolbox.side-by-side[ #set align(center) #muchpdf(read("ts1.pdf", encoding: none), scale: 0.205) #muchpdf(read("ts2.pdf", encoding: none), scale: 0.255)][#set text(size: 15pt)```nixservices.tailscale.extraSetFlags =let v4s = lib.concatStringsSep "," ipv4.addresses; v6s = lib.concatStringsSep "," ipv6.addresses; addresses = if v4s == "" then v6s else if v6s == "" then v4s else v4s + "," + v6s;in[ "--accept-routes" "--advertise-routes=${addresses}" "--snat-subnet-routes=false" ];```
#v(4em)\*Other tunneling protocols will also work]]
#slide[ == Anycast
#set align(center) #muchpdf(read("anycast.pdf", encoding: none))
#set align(left) Announcing the same prefixes, and use same address(es) on multiple machines. This is what Cloudflare, GitHub Pages, and many other providers do.]
#slide[== Lessons learned
- Use IPv6 when possible
- Host a looking glass - `bird-lg` or other looking glass tools - Feed full table to BGP.Tools, NLNOG, etc.- `tcpdump`, `mtr`, `ping`, `traceroute`, `dig`, etc. #footnote[Also ping.sx is pretty cool] are your friends
- Tailscale is good but... - Eat the entirety of CGNAT address space (100.64.0.0/10) - `nodeAttrs` -> `ipPool` is half-baked, outstanding issue since Feb 2021 (tailscale\#1381) - My workaround: set lookup rules with very specific priorities]
#slide[== Closing remarks
- Possible to run your own validator (ROA based and IRR based)
- Plan to write a more generic module to replace `services.bird` and upstream it - Proof of concept: `github:stepbrobd/router#nixosModules.alpha` - Idea from: `github:NuschtOS/bird.nix` - Support all protocols - Integrate flow exporter for better observability]
#slide[#set align(center + horizon)== Questions?
#v(2em)
ysun\@hey.com $dot.c$ `github:stepbrobd/router`
#v(2em)
Special thanks to Nick Cao (`github:NickCao`)
#v(1fr)
#box(image("france2030.png", height: 15%))#h(1em)#box(image("numpex.png", height: 15%))
#box(image("cf_blk.png", width: 25%)) #h(0.25em) #text(size: 16pt, weight: "bold")[Project Alexandria]]