Oriel — AI & technology worth noticing, every day.

Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

Repository · https://github.com/synchopate/llama-modchip

First seen by Oriel on .

AI summary

llama-modchip is a set of llama.cpp patches and tools that runs large sparse MoE models on CPU-only systems, including a 284B DeepSeek-V4-Flash checkpoint in 64 GB RAM.

  • Its DeepSeek-V4-Flash GGUF is 56.56 GiB at 1.71 bpw, with reported peak memory of 57.61 GiB at 256K context and no swap.
  • On a 16-core Zen 5 system, the project reports 43.16–44.36 tok/s prefill and 9.06–9.95 tok/s lossless decode; optional expert pruning trades small quality changes for faster decoding.
  • Quantization preserves selected tensors and compresses routed experts, while AVX-512 multi-level tiling, FlashAttention, and non-temporal MoE streaming target CPU cache and memory behavior.
  • For 30B MoE models, Fractal-64 is reported at 14.35 GiB and 25.0–60.4 tok/s on CPU, with a stated 100/100 quality gate.
  • The tools include model conversion, benchmarking, a chat interface, and Claude Code integration; custom tensor type 40 requires the project’s patched runtime.

#llama.cpp #mixture of experts #model quantization #cpu inference

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 470, latest observation ; +465 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 204, latest observation ; +203 over observed period since

Aliases

SourceExternal IDURL
Hacker News50411961https://github.com/synchopate/llama-modchip Read discussion