Huawei releases MACaQ, its internal quantizer for local inference
Repository · https://github.com/huawei-csl/macaq
First seen by Oriel on .
AI summary
MACaQ is a model-agnostic quantization toolkit for running text, image, video, and audio models through GGUF, llama.cpp, vLLM, Diffusers, and PyTorch.
- It quantizes a 122B-parameter model in 64 minutes using 32.5 GB of device memory.
- MACaQ supports Diffusers W4A4 generation, llama.cpp 2-, 3-, and 4-bit GGUF, vLLM W4A4 or weight-only checkpoints, and PyTorch W4A4 speech generation.
- For Qwen3.8-27B, MACaQ W4A4 reports 2.8994 perplexity, 2.33× BF16 prefill speed, and 2.58× BF16 decode speed in vLLM.
- The repository provides workflows for dense, MoE, hybrid, diffusion, speech, image, video, image-to-video, and audio models.
- Ascend support includes dense and grouped matmul kernels, but full-model execution requires integration with an Ascend inference runtime.
#model quantization #gguf #vllm #diffusion models
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 134, latest observation ; +128 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 32, latest observation ; +32 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50458166 | https://github.com/huawei-csl/MACaQ Read discussion |