Huawei Releases MACaQ: Multi-Modal Quantizer for Local Inference
Huawei open-sourced MACaQ, a quantization framework that compresses AI models for local deployment. The tool supports multiple modalities—text, images, videos, and audio—and integrates seamlessly with established inference stacks including llama.cpp, vLLM, Diffusers, and PyTorch. MACaQ can quantize a 122-billion-parameter model in 64 minutes using just 32.5 GB of memory. In quality benchmarks, the method achieves lower perplexity than competing approaches like MR-GPTQ and DuQuant++, while requiring roughly 11 times less quantization time.
The release includes MACaQ-R, a refinement system that improves the quality of existing quantized models and adapts them to specific workloads without breaking compatibility with standard llama.cpp. The framework supports 2, 3, and 4-bit quantization schemes, with W4A4 (4-bit weights and activations) as its primary mode. By open-sourcing the toolkit under Apache 2.0, Huawei makes high-performance model optimization accessible to developers building on-device or edge-based AI applications.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.