Show HN: DeepSeek V3 in ~200 lines of pure Jax
Repository · https://github.com/vaibhaav-tiwari/deepseek-v3-jax
First seen by Oriel on .
AI summary
This repository implements DeepSeek-V3’s core architecture in 190 lines of pure JAX and demonstrates training a small model from scratch on a MacBook Air.
- The implementation includes MLA, decoupled RoPE, DeepSeekMoE with one shared and 16 routed experts, auxiliary-loss-free balancing, sequence-wise loss, and MTP.
- A 36.2M-parameter model trained for 12,000 steps on 21.7M tokens in approximately 2.6 hours at a median 10.7k tokens per second.
- Expert balancing reduced load variation to 5.7–6.9% per expert, compared with 0.09–18.6% without bias, while validation loss remained nearly identical.
- Dense MoE computation, explicit MLA decompression, FP8 training, DualPipe, node-limited routing, YaRN, post-training, and original-model scale are omitted.
- Reproduction targets Apple Silicon Macs with Python 3.12 and pinned JAX dependencies; a smaller, lightly tested configuration supports 8–16 GB Macs.
#jax #deepseekmoe #mixture of experts #language models
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 272, latest observation ; +265 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 27, latest observation ; +27 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50497589 | https://github.com/Vaibhaav-Tiwari/deepseek-v3-jax Read discussion |