Oriel — AI & technology worth noticing, every day.

Edge0/Edge0-145B-q3k

AI model · https://huggingface.co/Edge0/Edge0-145B-q3k

First seen by Oriel on .

AI summary

This Hugging Face repository presents Edge0-145B Q3_K, a natively 4-bit latent sparse MoE claiming high-throughput local agent inference on a single consumer GPU.

  • The model uses 48 hybrid-attention layers, 512 experts, 10 active per token, a 256K context, and latent width ℓ = 1792.
  • On an RTX 5090 D 24 GB, reported performance is 121.2 tok/s agent decode, 3554.9 tok/s 16K prefill, and 86.5 terminal-bench 2.1.
  • Latent routing reduces routed expert parameters from 120.8 B to 84.6 B and reports 2.0× Qwen decode speed under a 16 GiB VRAM plus 20 GiB RAM budget.
  • edge0-infer and Edge0 Chat support local command-line or desktop use, with OpenAI- and Anthropic-compatible endpoints and optional image input.
  • Running it requires NVIDIA CUDA 13 or Apple Silicon Metal support, while SSD offload and residency controls trade memory usage against access speed.

#latent moe #q3_k #local inference #multimodal

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Likes

Total likes reported by the source.

  • Hugging Face Models: 135, latest observation ; +36 over observed period since

Downloads (30d)

Rolling 30-day downloads. A change in this value is not new downloads during the observed period.

  • Hugging Face Models: 92,228, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
Hugging Face ModelsEdge0/Edge0-145B-q3khttps://huggingface.co/Edge0/Edge0-145B-q3k Read discussion