Edge0/Edge0-145B-q3k
AI model · https://huggingface.co/Edge0/Edge0-145B-q3k
First seen by Oriel on .
AI summary
This Hugging Face repository presents Edge0-145B Q3_K, a natively 4-bit latent sparse MoE claiming high-throughput local agent inference on a single consumer GPU.
- The model uses 48 hybrid-attention layers, 512 experts, 10 active per token, a 256K context, and latent width ℓ = 1792.
- On an RTX 5090 D 24 GB, reported performance is 121.2 tok/s agent decode, 3554.9 tok/s 16K prefill, and 86.5 terminal-bench 2.1.
- Latent routing reduces routed expert parameters from 120.8 B to 84.6 B and reports 2.0× Qwen decode speed under a 16 GiB VRAM plus 20 GiB RAM budget.
- edge0-infer and Edge0 Chat support local command-line or desktop use, with OpenAI- and Anthropic-compatible endpoints and optional image input.
- Running it requires NVIDIA CUDA 13 or Apple Silicon Metal support, while SSD offload and residency controls trade memory usage against access speed.
#latent moe #q3_k #local inference #multimodal
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Likes
Total likes reported by the source.
- Hugging Face Models: 135, latest observation ; +36 over observed period since
Downloads (30d)
Rolling 30-day downloads. A change in this value is not new downloads during the observed period.
- Hugging Face Models: 92,228, latest observation ; 0 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Models | Edge0/Edge0-145B-q3k | https://huggingface.co/Edge0/Edge0-145B-q3k Read discussion |