RC RANDOM CHAOS

llama-modchip: 284B Inference on 64GB RAM

· via Hacker News

Original source

Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

Hacker News →

llama-modchip is a set of kernel patches and runtime tools designed to run sparse Mixture-of-Experts models in system RAM on CPU processors. The project fits the 284-billion-parameter DeepSeek-V4-Flash model into 56.56 GiB, enabling it to operate on 64GB physical RAM. Standard quantization creates 76.87 GiB checkpoints that exceed available system memory; llama-modchip’s compression keeps the complete model and cache resident in RAM without memory paging.

Performance gains come from CPU-targeted optimizations. Multi-level cache tiling and SIMD kernel patches increase prefill throughput by 77% and decoding speed by 53%, reaching 44.36 tokens per second on 16-core Zen 5 processors compared to a 25.0 baseline. The project supports x86_64 processors with AVX-512 or AVX2 instructions and ARM64 architectures, including optimizations for the 30-billion-parameter Nemotron-3.5 model. Installation is available for Windows, macOS, and Linux systems.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.