DIY two-node AMD Strix Halo cluster runs big LLMs over RDMA at 5µs latency

A community toolbox documents how to lash two Framework Desktop boards—each built on AMD’s Ryzen AI MAX+ “Strix Halo” APU with 128GB of unified memory—into a single inference cluster for vLLM. The two nodes use Tensor Parallelism to split a model that won’t fit on one APU, with Ray handling orchestration and AMD’s RCCL library moving tensor data between machines. The interconnect is the trick: Intel E810 100GbE cards wired directly with a DAC cable (no switch) speaking RoCE v2, which lets RCCL write straight into the other node’s memory and bypass the kernel. That drops cross-node latency from the ~70-100µs of TCP/IP to roughly 5µs—the difference between sluggish and usable interactive token generation.

The setup is deeply hands-on. On Fedora 43, it relies on in-kernel ice/irdma drivers rather than proprietary Intel ones, plus a careful BIOS and kernel-parameter regimen: iGPU memory pinned to 512MB so system RAM can be dynamically mapped as GPU “VRAM” via GTT, IOMMU pass-through, disabled PCIe power management, jumbo frames, and a trusted-zone firewall for the RDMA link. Because the Framework’s PCIe slot is physically x4, a riser is needed to seat the x16 NIC—one test board was literally cut open with an ultrasonic knife, which the author explicitly warns against.

The most consequential piece is software: the toolbox container ships a custom-built librccl.so that adds gfx1151 (Strix Halo) RDMA support missing from upstream ROCm, compiled via a GitHub Action and bundled into the image. A refresh_toolbox.sh script auto-detects InfiniBand/RDMA devices and exposes them along with /dev/dri and /dev/kfd. It’s a sharp example of enthusiasts squeezing distributed, data-center-style inference out of consumer hardware—and of how far ahead community patches can run of vendor driver support.