Oriel — AI & technology worth noticing, every day.

Huawei releases MACaQ, its internal quantizer for local inference

Repository · https://github.com/huawei-csl/macaq

First seen by Oriel on .

AI summary

MACaQ is a model-agnostic quantization toolkit for running text, image, video, and audio models through GGUF, llama.cpp, vLLM, Diffusers, and PyTorch.

  • It quantizes a 122B-parameter model in 64 minutes using 32.5 GB of device memory.
  • MACaQ supports Diffusers W4A4 generation, llama.cpp 2-, 3-, and 4-bit GGUF, vLLM W4A4 or weight-only checkpoints, and PyTorch W4A4 speech generation.
  • For Qwen3.8-27B, MACaQ W4A4 reports 2.8994 perplexity, 2.33× BF16 prefill speed, and 2.58× BF16 decode speed in vLLM.
  • The repository provides workflows for dense, MoE, hybrid, diffusion, speech, image, video, image-to-video, and audio models.
  • Ascend support includes dense and grouped matmul kernels, but full-model execution requires integration with an Ascend inference runtime.

#model quantization #gguf #vllm #diffusion models

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 134, latest observation ; +128 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 32, latest observation ; +32 over observed period since

Aliases

SourceExternal IDURL
Hacker News50458166https://github.com/huawei-csl/MACaQ Read discussion