Getting DeepSeek-V4-Flash Running on AMD's MI300X: A Worklog

AMD’s MI300X offers 192GB of HBM3 and roughly half the list price of NVIDIA’s H100, but software gaps keep it underused — especially on newer models. Doubleword documents what it took to bring DeepSeek-V4-Flash up on the chip via vLLM, where out-of-the-box support simply didn’t work as of May 2026. The core obstacle is that MI300X uses the losing side of the FP8 standards war: the fnuz dialect that AMD and Graphcore proposed, which shares bit layout with OCP FP8 but differs in exponent bias by one, silently producing values off by a factor of two when code assumes the wrong variant.

Beyond FP8, DeepSeek v4’s sparse attention exposed thin kernel coverage in AITER, AMD’s tuned-kernel library. Several paths — paged MQA logits, sparse MLA prefill and decode — are missing entirely on the gfx942 architecture, while others exist but are broken there, requiring platform guards that fall back to slower generic Triton. HIP graph capture added more constraints, since any host reads, ragged allocations, or mid-stream syncs get frozen at warmup, forcing rewrites of Triton metadata code to be capture-safe.

Smaller bugs piled on: an MoE routing mask gated on the wrong flag sent tokens to wrong experts, and a Triton kernel masked padded lanes against the wrong bound and corrupted the routing bitmatrix. Once correctness was achieved, profiling showed the expected hotspots in sparse MLA and MXFP4 MoE kernels, but a significant share of time still went to bookkeeping around them — ragged metadata rebuilds, redundant weight materialization, and untuned launch shapes — leaving clear room for optimization.