Oriel — AI & technology worth noticing, every day.

Show HN: DeepSeek V3 in ~200 lines of pure Jax

Repository · https://github.com/vaibhaav-tiwari/deepseek-v3-jax

First seen by Oriel on .

AI summary

This repository implements DeepSeek-V3’s core architecture in 190 lines of pure JAX and demonstrates training a small model from scratch on a MacBook Air.

  • The implementation includes MLA, decoupled RoPE, DeepSeekMoE with one shared and 16 routed experts, auxiliary-loss-free balancing, sequence-wise loss, and MTP.
  • A 36.2M-parameter model trained for 12,000 steps on 21.7M tokens in approximately 2.6 hours at a median 10.7k tokens per second.
  • Expert balancing reduced load variation to 5.7–6.9% per expert, compared with 0.09–18.6% without bias, while validation loss remained nearly identical.
  • Dense MoE computation, explicit MLA decompression, FP8 training, DualPipe, node-limited routing, YaRN, post-training, and original-model scale are omitted.
  • Reproduction targets Apple Silicon Macs with Python 3.12 and pinned JAX dependencies; a smaller, lightly tested configuration supports 8–16 GB Macs.

#jax #deepseekmoe #mixture of experts #language models

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 272, latest observation ; +265 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 27, latest observation ; +27 over observed period since

Aliases

SourceExternal IDURL
Hacker News50497589https://github.com/Vaibhaav-Tiwari/deepseek-v3-jax Read discussion