FlashML-org/FreeToken
Repository · https://github.com/flashml-org/freetoken
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
First seen by Oriel on .
AI summary
FreeToken is an edge-native Mixture-of-Experts serving engine claiming to run 290B+ open-weight models locally on consumer hardware at interactive speeds.
- It unifies GPUs, CPUs, host memory, and interconnects into an elastic inference platform for local MoE inference.
- The runtime uses bandwidth-adaptive CPU–GPU co-execution, double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and FTW fast weight format.
- Semantic anchor checkpoints let agentic context edits, including tool calls and thinking blocks, avoid redundant context recomputation.
- VRAM can be dynamically reallocated between expert caches and KV memory without engine restarts or weight reloading.
- FreeToken supports MXFP4, NVFP4, FP8, and BF16 models, Anthropic/OpenAI-compatible APIs, and NVIDIA RTX 30, 40, and 50 series GPUs.
#mixture-of-experts #model serving #edge inference #gpu computing
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Stars
Total repository stars at each observation.
- GitHub Trending: 16,384, latest observation ; +1,304 over observed period since
Stars gained (1d)
Daily star gain reported by GitHub Trending. A change compares two daily values; it is not additional stars between observations.
- GitHub Trending: 653, latest observation ; +421 over observed period since
Forks
Total repository forks at each observation.
- GitHub Trending: 1,604, latest observation ; +72 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| GitHub Trending | flashml-org/freetoken | https://github.com/FlashML-org/FreeToken Read discussion |