nixos-pvm
Star 0
Fork 0
Watch
5 1 0
Files
.tangled
modules
pkgs
.gitignore
default.nix
flake.lock
flake.nix
README.md
README.md

nixos-pvm #

PVM (pagetable-based virtual machine) kernels and NixOS modules. PVM is a KVM implementation that needs no hardware virtualization extensions, so cloud VMs that do not expose /dev/kvm can still run KVM-accelerated microVMs (QEMU, Cloud Hypervisor).

  • Host kernel (linux-pvm): its kvm-pvm.ko registers /dev/kvm without VT-x/SVM.
  • Guest kernel (linux-pvm-guest): guests of a PVM host must boot it; stock kernels cannot boot under PVM.
  • NixOS modules host and guest, which do nothing until enabled.

Both kernels are Linux 6.12.33 from virt-pvm/linux (pvm-612, pinned by commit in pkgs/linux-pvm.nix), built with nixpkgs' buildLinux and its common config, like nixpkgs' own kernels. The host adds CONFIG_KVM_PVM=m; the guest adds CONFIG_X86_PIE=y and CONFIG_PVM_GUEST=y. Host and guest share PVM's hypercall interface, so both come from the same commit.

Before you start #

  • You build the kernels yourself. There is no binary cache. Each kernel takes about an hour to build, and a host with PVM guests needs both. So does every update of this repo that changes them.
  • Do not make this repo's nixpkgs follow yours. The modules use kernels built from the nixpkgs pinned in this repo's flake.lock, whatever nixpkgs your system uses, because that is what the kernels are tested against. With inputs.nixos-pvm.inputs.nixpkgs.follows = "nixpkgs" you get kernels built from your nixpkgs instead, which may not build.
  • boot.kernelPatches means your own kernel. Extra kernel options go through the standard boot.kernelPatches; that changes the kernel derivation, so you build and maintain that kernel. Options whose dependencies are unmet fail the build rather than being dropped silently.
  • The fork is behind 6.12 stable. virt-pvm/linux is based on 6.12.33, while nixpkgs' 6.12 is at 6.12.112: about 80 stable releases of fixes, security fixes included, are missing. Building with buildLinux does not change that; only a rebase of virt-pvm/linux does.
  • No Rust. 6.12.33's Rust support does not build with the pinned rustc, so the kernels are built without it. pkgs/linux-pvm.nix lists this and the other differences from nixpkgs' common config.
  • x86_64 only.

Usage #

Flakes #

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
  # Not `follows`; see above.
  inputs.nixos-pvm.url = "git+https://tangled.org/jacobs.land/nixos-pvm";

  outputs =
    { nixpkgs, nixos-pvm, ... }:
    {
      nixosConfigurations.my-host = nixpkgs.lib.nixosSystem {
        modules = [
          ./configuration.nix
          nixos-pvm.nixosModules.host
          { pvm.host.enable = true; }
        ];
      };
    };
}

Plain Nix #

let
  nixos-pvm = import (builtins.fetchGit {
    url = "https://tangled.org/jacobs.land/nixos-pvm";
    rev = "...";
  }) { };
in
{
  imports = [ nixos-pvm.nixosModules.host ];
  pvm.host.enable = true;
}

default.nix is the source of truth; flake.nix wraps it. It takes optional nixpkgs and pkgs arguments, which default to the pin in flake.lock.

The modules #

pvm.host.enable #

Runs the system as a PVM host:

  • boot.kernelPackages defaults (mkDefault) to the host kernel;
  • loads kvm-pvm;
  • blacklists kvm-intel and kvm-amd, which cannot coexist with kvm-pvm;
  • boots with pti=off nokaslr, because kvm-pvm refuses to load with page table isolation or KASLR active. The kernel only enables PTI on CPUs affected by Meltdown; on those, pti=off removes that mitigation. Check /sys/devices/system/cpu/vulnerabilities/meltdown.

Nothing else changes: the kernel has nixpkgs' common config, so the default initrd modules, the firewall (nf_tables) and Docker work as on a stock kernel.

A new kernel only takes effect at the next boot, and nixos-rebuild switch does not replace the running one. Deploy kernel changes with nixos-rebuild boot, reboot, and keep the previous generation in the boot menu in case the new kernel does not come back.

pvm.guest.enable #

Runs the system as a PVM guest: boot.kernelPackages defaults to the guest kernel, and it boots with pti=off nokaslr. The guest kernel is modular, like nixpkgs' own kernels.

Packages #

Output Contents
packages.x86_64-linux.linux-pvm host kernel
packages.x86_64-linux.linux-pvm-guest guest kernel (bzImage for direct boot)
nixosModules.host, nixosModules.guest the modules above

From plain Nix, (import ./. { }) has kernels.{host,guest}, linuxPackages.{host,guest} and nixosModules.{host,guest}.

To build the kernels from another nixpkgs on purpose, for example to try a nixpkgs bump, pass it as pkgs: (import ./. { pkgs = …; }).kernels. That is untested. The modules ignore it and always use the pin.

Known limitations #

  • One kernel cannot be both host and guest: kvm-pvm.ko does not build in a PVM_GUEST kernel (see pkgs/linux-pvm.nix).
  • The guest kernel has no KVM PTP clock (ptp_kvm): as a module it does not link, because the fork does not export pvm_hypercall.
  • kvm-pvm refuses to load on hosts with 5-level paging (LA57), virt-pvm/linux#6. The workaround is X86_5LEVEL = no through boot.kernelPatches.
  • QEMU: SeaBIOS does not work under PVM. Use direct kernel boot with the microvm machine (qboot by default) or -bios qboot.rom.
  • Cloud Hypervisor supports PVM from v35. Avoid a virtio-pmem root filesystem, virt-pvm/linux#1.
  • The host logs kvm_pvm: Inject event in non-PVM mode and a KVM PFN-cache warning while guests boot, virt-pvm/linux#21. Guests still run.
  • No out-of-tree kvm-pvm for nixpkgs' kernels. virt-pvm's pvm-6.12-host-oot branch builds against nixpkgs' 6.12 with small fixes, but keeps 6.12.33's KVM code and can break on any 6.12.y bump, so the host boots the fork; guests need it regardless. virt-pvm/linux#26 proposes a supported out-of-tree mode.

Development #

nix flake check            # builds both kernels (about an hour each)
nix flake check --no-build # evaluation only
nix fmt

The checks build both kernels, and evaluate a minimal NixOS system with each module and one with neither enabled. They also check that the modules boot the same kernel derivations as the packages, so one build serves both.

CI (.tangled/workflows/eval.yml) only evaluates: a kernel build does not fit a workflow VM. It prints both kernels' derivation paths, which show when a change means rebuilding them.

References #