I'm making LLMs work on small and old devices
Repository · https://github.com/hyperionaidev/smolmaxxing
First seen by Oriel on .
AI summary
smolmaxxing documents measured techniques for running increasingly large quantized and mixture-of-experts language models on severely memory-constrained devices, reaching 16B parameters on a Xiaomi TV Box and iPad Air.
- On the Xiaomi TV Box S, Qwen3.5-2B reaches 7.7 tok/s, Moon-Beam-8B-A1B reaches 7.2 tok/s, and Ling-mini-2.0 reaches 5.1 tok/s through expert paging.
- A 1-bit lookup-table kernel, headless execution, pruned heads, fused operations, and memory optimizations improve Bonsai-8B from 1.21 to 5.2 tok/s.
- Weight streaming and expert paging read model data from flash on demand, but flash bandwidth limits dense models and larger models that access more weights per token.
- On the jailbroken iPad Air, Bonsai-1.7B reaches 23.7–25.8 tok/s using base-NEON kernels, while Moon-Beam-8B-A1B reaches 3.5–3.6 tok/s.
- The repository records 160 experiments and emphasizes hardware measurements, with trade-offs including increased perplexity, slower long-context generation, and models unsuitable for always-on service.
#llm inference #quantization #arm neon #mixture-of-experts
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 477, latest observation ; +469 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 180, latest observation ; +178 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50468888 | https://github.com/hyperionaidev/smolmaxxing/ Read discussion |