Oriel — AI & technology worth noticing, every day.

Compress any LLM 3x in one click, near-lossless, in under 50 MB RAM

Repository · https://github.com/neuracryption/spectra

First seen by Oriel on .

AI summary

SPECTRA is an architecture-agnostic 4-bit neural compression and GPU serving runtime claiming fast, calibration-free local inference for models that otherwise exceed consumer and edge GPU memory.

  • Compression streams tensors directly from disk, uses under 50 MB host RAM, requires no calibration dataset, and takes 2.0–4.4 minutes for tested 1.5B–4B models.
  • It quantizes eligible matrices using Group-32 4-bit microscaling and HDSQ outlier channels while preserving normalization vectors, biases, convolution filters, and RoPE tables in FP16/BF16.
  • Fused Triton kernels unpack weights in registers, while TRDC containers, quantized KV caches, and optional asynchronous layer offloading support constrained hardware.
  • On an RTX 3050 Ti with 4 GB VRAM, tested models loaded in 3.22–3.90 seconds and reached 9.58–82.53 tokens per second.
  • DeepSeek-R1-Distill-1.5B scored 35.58% on ARC-Challenge versus 34.73% for BF16, with 91.0% first-token agreement across 3,200 generated tokens.

#llm compression #gpu inference #quantization #triton

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 21, latest observation ; +15 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 0, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
Hacker News50450729https://github.com/NeuraCryption/SPECTRA Read discussion