Wafer runs GLM5.2 on AMD's MI355X at 2x lower cost than NVIDIA Blackwell

Inference provider Wafer served the GLM5.2 model on AMD’s Instinct MI355X and hit 2626 tokens/sec/node on a prefill-heavy workload (20k in / 1k out, 60% cache hit) and 213 tokens/sec in single-stream decode. That aggregate throughput is only about 80% of what they measured on an NVIDIA B200, but because MI355X GPUs run roughly 2.75x cheaper than a B300, the AMD setup wins decisively on performance per dollar. The pitch: with Blackwell supply tight and NVIDIA prices climbing, AMD’s comparable silicon is an underused escape hatch for cheap inference.

The engineering story is about closing NVIDIA’s software gap without writing custom kernels. Wafer quantized the bf16 model to MXFP4 using AMD Quark (lossless versus the official FP8 build on their benchmarks) and ran it on sglang after ruling out vLLM and ATOM. Getting speculative decoding to work on the ROCm image took two small but necessary patches: fixing a module-prefix mismatch that caused sglang to wrongly quantize the MTP head’s shared expert (a ~3x single-stream gain once fixed), and adding a single USE_ROCM guard around a CUDA-only kernel include so deep draft depths would compile. Prefill throughput then jumped after they hand-tuned MoE kernel selection for GLM’s FP4 shapes, since AMD’s aiter library only shipped tuned configs for the FP8 path.

The broader takeaway is that NVIDIA’s CUDA advantage increasingly comes down to day-0 model support rather than any fundamental software moat. As coding agents get better at kernel and framework optimization, the friction of bringing frontier models up on ROCm keeps shrinking — turning AMD’s cost edge into a practical option rather than a perpetual catch-up game.