Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU
Repository · https://github.com/synchopate/llama-modchip
First seen by Oriel on .
AI summary
llama-modchip is a set of llama.cpp patches and tools that runs large sparse MoE models on CPU-only systems, including a 284B DeepSeek-V4-Flash checkpoint in 64 GB RAM.
- Its DeepSeek-V4-Flash GGUF is 56.56 GiB at 1.71 bpw, with reported peak memory of 57.61 GiB at 256K context and no swap.
- On a 16-core Zen 5 system, the project reports 43.16–44.36 tok/s prefill and 9.06–9.95 tok/s lossless decode; optional expert pruning trades small quality changes for faster decoding.
- Quantization preserves selected tensors and compresses routed experts, while AVX-512 multi-level tiling, FlashAttention, and non-temporal MoE streaming target CPU cache and memory behavior.
- For 30B MoE models, Fractal-64 is reported at 14.35 GiB and 25.0–60.4 tok/s on CPU, with a stated 100/100 quality gate.
- The tools include model conversion, benchmarking, a chat interface, and Claude Code integration; custom tensor type 40 requires the project’s patched runtime.
#llama.cpp #mixture of experts #model quantization #cpu inference
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 470, latest observation ; +465 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 204, latest observation ; +203 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50411961 | https://github.com/synchopate/llama-modchip Read discussion |