Compress any LLM 3x in one click, near-lossless, in under 50 MB RAM
Repository · https://github.com/neuracryption/spectra
First seen by Oriel on .
AI summary
SPECTRA is an architecture-agnostic 4-bit neural compression and GPU serving runtime claiming fast, calibration-free local inference for models that otherwise exceed consumer and edge GPU memory.
- Compression streams tensors directly from disk, uses under 50 MB host RAM, requires no calibration dataset, and takes 2.0–4.4 minutes for tested 1.5B–4B models.
- It quantizes eligible matrices using Group-32 4-bit microscaling and HDSQ outlier channels while preserving normalization vectors, biases, convolution filters, and RoPE tables in FP16/BF16.
- Fused Triton kernels unpack weights in registers, while TRDC containers, quantized KV caches, and optional asynchronous layer offloading support constrained hardware.
- On an RTX 3050 Ti with 4 GB VRAM, tested models loaded in 3.22–3.90 seconds and reached 9.58–82.53 tokens per second.
- DeepSeek-R1-Distill-1.5B scored 35.58% on ARC-Challenge versus 34.73% for BF16, with 91.0% first-token agreement across 3,200 generated tokens.
#llm compression #gpu inference #quantization #triton
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 21, latest observation ; +15 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 0, latest observation ; 0 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50450729 | https://github.com/NeuraCryption/SPECTRA Read discussion |