Oriel — AI & technology worth noticing, every day.

FlashML-org/FreeToken

Repository · https://github.com/flashml-org/freetoken

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

First seen by Oriel on .

AI summary

FreeToken is an edge-native Mixture-of-Experts serving engine claiming to run 290B+ open-weight models locally on consumer hardware at interactive speeds.

  • It unifies GPUs, CPUs, host memory, and interconnects into an elastic inference platform for local MoE inference.
  • The runtime uses bandwidth-adaptive CPU–GPU co-execution, double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and FTW fast weight format.
  • Semantic anchor checkpoints let agentic context edits, including tool calls and thinking blocks, avoid redundant context recomputation.
  • VRAM can be dynamically reallocated between expert caches and KV memory without engine restarts or weight reloading.
  • FreeToken supports MXFP4, NVFP4, FP8, and BF16 models, Anthropic/OpenAI-compatible APIs, and NVIDIA RTX 30, 40, and 50 series GPUs.

#mixture-of-experts #model serving #edge inference #gpu computing

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Stars

Total repository stars at each observation.

  • GitHub Trending: 16,384, latest observation ; +1,304 over observed period since

Stars gained (1d)

Daily star gain reported by GitHub Trending. A change compares two daily values; it is not additional stars between observations.

  • GitHub Trending: 653, latest observation ; +421 over observed period since

Forks

Total repository forks at each observation.

  • GitHub Trending: 1,604, latest observation ; +72 over observed period since

Aliases

SourceExternal IDURL
GitHub Trendingflashml-org/freetokenhttps://github.com/FlashML-org/FreeToken Read discussion