Something went wrong. Try again.
(I'm so sorry!) Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Something went wrong. Try again.
3.6 kB · 75 lines
at master
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576==============UALink Support==============
Overview========
Connected GPUs in a pod can directly access the remove memory on another GPUover UALink. Unlike RMDA, there is no copy involved; it is direct loads/storesover the fabric. Shared memory can only be accessed by a remote GPU if thememory was exported and the importer has been authorized. For the memory to beshared, it must be part of a unified physical address space shared betweennodes. This address space is called NPA (Nework Physical Address) space. Thisaddress space is partitioned between the GPUs so that each GPU has its ownsegment of the address space in which to export its memory. Each GPU maintainsa dedicated set of page tables for their NPA space similar to GPUVM. Note thatthis mechanism only allows for GPU access to remote memory. The remote memoryis not CPU accessible.
Exported memory is pinned. This is similar to how dma-bufs in VRAM are pinnedfor P2P access. Remote TLB shootdowns from the exporter are used when theexported memory is freed in order to remove access by remote GPUs.
To access remote memory, the driver can map NPA addresses into its per processGPUVM page tables just like local memory. Applications use opaque handles torepresent remote memory. GPUs in a pod communicate with eachother directly toexchange NPA addresses between importers and exporters. If a node goes offlineor is reset, their peers will clean up any remaining refrences that are lostwhen that happens. NPA addresses are exchanged directly between the kerneldrivers using the scale up fabric. NPA addresses are never exposed to userspace.
On the importer, the NPA space is like another physical address space. NPAaddresses can be used as physical addresses for GPUVM to provide GPU virtualaddresses to the memory for processes using the GPU.
On the exporter, the NPA space provides a way to expose discontiguous localmemory as a contiguous address range for remote GPUs. This allows the exporterto locally manage the pages mapped into the NPA space.
Remote NPAs are managed like another device specific TTM pool similar todoorbells or VRAM, however they cannot be CPU mapped.
User Interface==============Two IOCTLs are provided to export and import remote memory.
Export Memory-------------To export memory, a UALINK handle must be created for an allocation that can beshared with another node in the pod. To do this the exporter calls the GEMUALink IOCTL with the GEM handle to the buffer it wants to export. The IOCTLreturns a unique 128 bit handle which can be shared with the remote host.Calling export on the same GEM handle always returns the same UALink handle.The UALink handle is destroyed when the GEM object reference count reaches 0.
Import Memory-------------To import remote memory, the UALink handle from the remote node must beconverted from a UALink handle to a local GEM object which represents the localreference to the NPA space on the importer. If the memory has already beenimported, it just returns a new reference to the existing GEM object. If not,the importer queries the exporter to get the NPA address. Once it has that, theimporter can create the GEM to represent the NPA space used by the allocation.The GEM object is then exported to the caller as a dma-buf. The dma-buf isleveraged for dynamic attachment which provides the ability to revoke accesswhen necessary.
Device to Device Communications===============================
Devices communicate via a protocol implemented in firmware. Mesages sent to aremote node generate an interrupt on that node for servicing.