标签：#nvidiapeermem

nvidia-peermem "Invalid argument" on Ubuntu — Fix GPUDirect RDMA with DMA-BUF TL;DR: If modprobe nvidia-peermem fails with Invalid argument ( -EINVAL ) on a system using the inbox Ubuntu InfiniBand stack ( rdma-core ), the module is not broken and you do not need it. nvidia-peermem requires an API that only exists in MLNX_OFED. On Hopper/Blackwell GPUs with the NVIDIA open driver, use DMA-BUF instead — it does GPUDirect RDMA natively. The one gotcha: you must enable nvidia-drm modeset=1 . Applies to: Ubuntu 22.04 / 24.04, inbox rdma-core stack, NVIDIA open kernel driver, H100 / H200 / B200, ConnectX-6/7 (or any HCA with ODP support). The symptom $ sudo modprobe nvidia-peermem modprobe: ERROR: could not insert 'nvidia_peermem' : Invalid argument dmesg shows nvidia-peermem loaded but registered nothing, or the load returns -EINVAL . GPUDirect RDMA appears to be unavailable. Why this happens (and why it is not a bug) nvidia-peermem is the legacy path for GPUDirect RDMA. It registers GPU memory with the InfiniBand subsystem through a Mellanox-proprietary kernel API: ib_register_peer_memory_client () That symbol only exists in MLNX_OFED's build of ib_core . It is not in the mainline kernel, and it is not in rdma-core , which is the inbox InfiniBand stack on Ubuntu. If you are on the inbox stack, nvidia-peermem was compiled without that API present, so it can never bind and always returns Invalid argument . No module parameter or config change will fix it, because the thing it needs was never there. Do not install MLNX_OFED just to make nvidia-peermem load. That works, but it is the wrong fix — you would be adding a heavy proprietary stack to revive an obsolete module. There is a native path already in your kernel. The fix: use DMA-BUF On Hopper and newer with the open driver, GPUDirect RDMA works through DMA-BUF , a mainline Linux framework. No external module, no MLNX_OFED. Requirements (check these first) NVIDIA open kernel driver (not the proprietary build) nvidia-drm