cmpunlocker
asm64-hooligan's cmpunlocker + 610.57.04 support
Star
Fork
Watch
cmpunlocker docs ARCHITECTURE.md
12 kB · 164 lines
Markdown
at main

Architecture #

Overview #

The CMP 170HX is a full GA100 die — the same silicon as the A100 — with SM throttle, memory and PCIe link speed clamped in firmware. The clamps are enforced by PLM (Priv Level Mask) gates: registers that reject writes coming from the host, no matter how privileged the driver is.

cmpunlocker lifts those clamps during driver initialisation, before GSP boots. Nothing is flashed and no hardware is modified; the unlock is re-applied from scratch on every driver load.

Stock Unlocked
Memory (0x20C2) 8 GB 64 GB
Memory (0x2082) 10 GB 40 GB
SM speed select clamped full
PCIe Gen1 Gen2
GPU-to-GPU P2P blocked working, opt-in (--p2p)

The SEC2 Booter Trick #

The host cannot write a PLM-gated register, but the SEC2 Booter can — it runs signed and is trusted by the PLM logic.

During GSP bootstrap the driver allocates a signature buffer and hands its physical address to the Booter. cmpunlocker enlarges that buffer to 0xf800 bytes and fills it with a crafted Booter payload instead of the real signature. The payload makes SEC2 perform exactly one 32-bit register write, with the address and value taken from two slots inside it (0xf76c and 0xf754).

One Booter Load = one gated write. Rewriting those two slots and running Booter Load again gives the next write, so the gates are opened one at a time in a loop. When the loop is done the real signature is put back and GSP boots on an already-unlocked GPU, seeing nothing unusual.

The payload is loaded from /lib/firmware/nvidia/ga100/gsp/dmem.bin when present, otherwise it is generated in place — both produce the same image.

Code Layout #

All logic lives in one source file that is dropped into the stock tree. The patches only add the calls that reach it, which is why they are a few lines each.

driver/
  src/
    cmpunlock.c          all unlock logic
    cmpunlock.h          the hook interface
    cmpunlock_config.h   build-time options (regenerated by build.sh)
  patches/
    0001-gsp-hooks.patch            kernel_gsp.c        6 hooks
    0002-tu102-hook.patch           kernel_gsp_tu102.c  1 hook
    0003-osinit-late-pma-hook.patch osinit.c            1 hook
    0004-memmgr-quirks.patch        mem_mgr*, mem_scrub 3 conditionals
    0005-bar0-pramin-clamp.patch    kern_bus_gm107.c    1 conditional
    0006-persistent-sw-state.patch  nv.c                1 conditional
    0007-p2p-caps-hook.patch        gpu.c               1 hook

build.sh copies driver/src/* into src/nvidia/{src,inc}/kernel/gpu/cmpunlock/, appends one SRCS += line to src/nvidia/srcs.mk, then applies the patches.

Every hook starts with cmpUnlockIsTarget(), which matches PCI device ID 0x20C2 or 0x2082. On any other GPU each hook is a no-op, so the patched modules are safe on mixed systems. Keeping the logic outside the patches means moving to a new nvidia-open release only requires re-checking the insertion points instead of re-merging large diffs.

Boot Sequence #

Where Hook What it does
_kgspCreateSignatureMemdesc() cmpUnlockSignatureSize
cmpUnlockFillSignature		 Enlarge the signature buffer, stash the real signature aside, write the Booter payload in its place
_kgspBootGspRm(), after kgspPrepareForBootstrap_HAL() cmpUnlockPreBoot The unlock itself (below)
kgspBootstrap_TU102(), after Booter Load cmpUnlockPostBooterLoad Verify the gated registers survived; first half of the HBM overclock
kgspInitRm_IMPL(), after static config fetch cmpUnlockFixStaticInfo Widen the FB geometry GSP reported back to the unlocked size
kgspInitRm_IMPL(), after kgspStartLogPolling() cmpUnlockMclkPostGsp Second half of the HBM overclock
_gpuInitPcieP2PCapability() cmpUnlockForceP2PCaps Override the GSP P2P capability response (only with --p2p)
RmInitNvDevice() cmpUnlockLateExtendPma Hand the memory above the stock limit to PMA

cmpUnlockPreBoot() runs in the window after the bootstrap is prepared and before GSP is released:

  1. Save the WPR2 window (0x1fa824 / 0x1fa828) — Booter Load clobbers it and it is restored before every round trip.
  2. Walk the PLM table: 17 entries, each one Booter Load that opens a gate, retried once if the register does not read back as expected.
  3. With the gates open, write the SM speed select and memory geometry registers.
  4. Retrain the PCIe link at Gen2.
  5. Restore the stock signature, then call kgspPopulateWprMeta_HAL() — the framebuffer size changed underneath, so the WPR layout has to be recomputed.

Patch 0001 also stops the stock "WPR2 already up, cannot proceed" bail-out from firing on a CMP, because cmpunlocker recovers from that state instead of treating it as fatal.

Register Map #

Written once the gates are open:

Register Address Value
SM speed select 0 0x0082381c 0x88888888
SM speed select 1 0x00823820 0x00000008
FBPA CFG1 0x009a0204 0x02779000 (8GB) / 0x02669000 (10GB)
MMU LMR 0x00100ce0 0x0000020B (8GB) / 0x0000028A (10GB)

CFG1 sets the memory address mapping and bank layout; LMR selects the capacity. 8GB vs 10GB is decided at runtime from the device ID, so one build covers both variants and mixed systems.

PCIe Gen2 #

The XP registers that control link speed are PLM-gated, so they are only writable inside the unlock window. The retrain clears the CYA bits pinning the link to Gen1, selects Gen2 in XP_CFG0 and XP_LCTRL2, sets the root port's target speed via its LnkCtl2 config word (the link will not come up at Gen2 unless both ends agree), then triggers a directed speed change and polls the LTSSM until it settles.

GPU-to-GPU P2P #

GSP firmware reports pcieP2PReadCaps / pcieP2PWriteCaps as NOT_SUPPORTED for CMP device IDs. The driver stores that verdict and p2pGetCapsStatus() then refuses every PCIe P2P path, so cudaDeviceEnablePeerAccess() fails and nvidia-smi topo -m shows GNS.

On the GPU side that restriction is only the reported capability — the GA100 mailbox P2P hardware works. cmpUnlockForceP2PCaps() overwrites both fields with OK right after the RPC returns, and the driver falls into P2P_CONNECTIVITY_PCIE_PROPRIETARY: the mailbox protocol, which does not need a large BAR1 (the CMP's is 64 MiB and locked by VBIOS).

Where the path works, throughput follows the PCIe topology — roughly 1.7 GB/s between GPUs behind one switch at Gen2 x4, less across a host bridge or NUMA node.

This is opt-in (--p2p) and compiled out by default. The override changes what the driver advertises, not whether the host can carry the traffic. When the topology cannot route peer-to-peer — ACS in the way, no IOMMU passthrough, GPUs on separate root complexes — the honest NOT_SUPPORTED is what makes CUDA stage through system memory instead. Forcing the caps removes that fallback: cudaDeviceEnablePeerAccess() succeeds, nvidia-smi topo -p2p r reports OK, and the transfers then time out. A wrong answer that looks right is worse than a correct refusal, which is why it is not the default.

HBM Memory Clock #

Compiled out entirely unless --mclk-ndiv=N is passed; build.sh then generates cmpunlock_config.h with CMPUNLOCK_MCLK_NDIV. The clock is N * 27 MHz.

The sequence is VBIOS-agnostic: the stock NDIV is read out of the PLL rather than assumed, and every COEFF write is a read-modify-write that preserves the MDIV/PDIV the VBIOS programmed. It runs in two halves:

  • After Booter Load — assert MEMCLK_CHANGE_ALERT, enter HBM self-refresh, cycle the PLL on each of the 12 FBPAs (drop the enable bits, write the new NDIV, re-enable, poll for lock), exit self-refresh, run a DDLL calibration, clear the alert.
  • After GSP is up — one multicast COEFF write across all FBPAs, followed by a PRI fence and a lock poll.

Supporting Changes #

Not part of the unlock proper, but required for the unlocked geometry to be stable:

  • Memory manager quirks (0004) — CE virtual mode is disabled for the scrubber and CeUtils, and the PTE kind is forced to GENERIC_MEMORY instead of the PLC-disable variant. Compression is unreliable on the widened geometry.
  • BAR0 PRAMIN clamp (0005) — the BAR0 window only decodes the stock 8 GB, so PRAMIN is pinned where it would have landed before the unlock widened fbAddrSpaceSizeMb.
  • Persistent software state (0006) — NV_FLAG_PERSISTENT_SW_STATE is set for CMP device IDs. The unlock happens once during GSP bootstrap, so the software state has to survive teardown rather than being rebuilt.
  • Late PMA extension (0003) — GSP hands back a heap sized for the stock capacity. The memory above it is present but parked in a reserved FB region; once the heap and PMA are fully initialised, that region is carved out and registered with PMA.

Persistence #

The unlock is applied by patched kernel modules, not a userspace daemon. build.sh installs nvidia.ko, nvidia-modeset.ko, nvidia-uvm.ko, nvidia-drm.ko and nvidia-peermem.ko into /lib/modules/$(uname -r)/updates/cmpunlocker/, which depmod prefers over the stock paths, then rebuilds the initramfs.

Every driver initialisation re-runs the sequence, so the unlock survives reboots until ./remove.sh is run. Nothing is stored on the card.

Verifying #

CMPUNLOCK: PLM[0] WPR_CFG(0x1fa7cc) attempt=0 status=0x0 reg=0xfffff0ff
CMPUNLOCK: PLMs: FEAT=0xffffffff FBPA=0xffffffff WPR=0xffffffff WPR_CFG=0xfffff0ff
CMPUNLOCK: POST-WRITE SS0=0x88888888 SS1=0x00000008 CFG1=0x02779000 LMR=0x0000020b (devId=0x20c2)
CMPUNLOCK: PCIe retrain done (polls=1): LinkCtrlStat=0x... speed=2 width=x16
CMPUNLOCK: PCIe P2P caps forced to OK (--p2p)

Non-zero Booter status codes during the early PLM rounds are common and harmless as long as the register reads back correctly and the boot completes — the table retries each gate anyway.

Limitations #

  • Secure Boot must be disabled — the patched modules are unsigned.
  • nvidia-open 610.57.04 or 610.43.0x only — the proprietary driver has a different GSP boot path and cannot be patched this way.
  • Linux only — the unlock lives in the Linux GSP bootstrap path.
  • Kernel headers required — modules are compiled against the running kernel.
  • ?BAR1 stays at 64 MiB — locked by VBIOS, so P2P goes through the mailbox protocol rather than direct BAR1 mapping.?