Oriel — AI & technology worth noticing, every day.

I'm making LLMs work on small and old devices

Repository · https://github.com/hyperionaidev/smolmaxxing

First seen by Oriel on .

AI summary

smolmaxxing documents measured techniques for running increasingly large quantized and mixture-of-experts language models on severely memory-constrained devices, reaching 16B parameters on a Xiaomi TV Box and iPad Air.

  • On the Xiaomi TV Box S, Qwen3.5-2B reaches 7.7 tok/s, Moon-Beam-8B-A1B reaches 7.2 tok/s, and Ling-mini-2.0 reaches 5.1 tok/s through expert paging.
  • A 1-bit lookup-table kernel, headless execution, pruned heads, fused operations, and memory optimizations improve Bonsai-8B from 1.21 to 5.2 tok/s.
  • Weight streaming and expert paging read model data from flash on demand, but flash bandwidth limits dense models and larger models that access more weights per token.
  • On the jailbroken iPad Air, Bonsai-1.7B reaches 23.7–25.8 tok/s using base-NEON kernels, while Moon-Beam-8B-A1B reaches 3.5–3.6 tok/s.
  • The repository records 160 experiments and emphasizes hardware measurements, with trade-offs including increased perplexity, slower long-context generation, and models unsuitable for always-on service.

#llm inference #quantization #arm neon #mixture-of-experts

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 477, latest observation ; +469 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 180, latest observation ; +178 over observed period since

Aliases

SourceExternal IDURL
Hacker News50468888https://github.com/hyperionaidev/smolmaxxing/ Read discussion