Oriel — AI & technology worth noticing, every day.

Gemma 4 E2B in JAX on One TPU v5e: bf16 or 4-Bit, Python or Rust, and How It Stands Against vLLM

Post · https://dev.to/gde/gemma-4-e2b-in-jax-on-one-tpu-v5e-bf16-or-4-bit-python-or-rust-and-how-it-stands-against-vllm-j72

Gemma 4 E2B served three ways on one Cloud TPU v5e chip: a pure-JAX engine in Python, the same graph written in Rust and compiled by the libtpu that JAX loads, and vLLM, all timed by one client. bf16 beats 4-bit in the JAX engine, the Rust engine matches the JAX engine's tokens, and over HTTP it decodes at vLLM's rate with the fastest first token.

First seen by Oriel on .

AI summary

The article benchmarks Gemma 4 E2B on one TPU v5e, finding a Rust-built StableHLO engine matches vLLM’s single-request decode speed, while bf16 JAX outperforms 4-bit JAX.

  • The pure-JAX engine loads Hugging Face safetensors directly and serves an OpenAI-compatible chat API without PyTorch.
  • On the JAX engine, bf16 decodes at 132.6 tokens per second versus about 103 for both 4-bit builds, which save 2.70 GB of HBM.
  • A Rust engine builds the same graph as StableHLO and compiles it through libtpu; it achieves 148.4 tokens per second over HTTP versus vLLM’s 148.7.
  • Rust and JAX match 7 of 8 completions token for token; Rust’s first-token time is 8.9 ms, compared with 13.0 ms for vLLM.
  • Only vLLM supports concurrent requests in this comparison; the measurements cover one request at a time on one TPU chip.

#gemma #jax #stablehlo #tpu

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 11, latest observation ; +5 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 0, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
DEV Community4972397https://dev.to/gde/gemma-4-e2b-in-jax-on-one-tpu-v5e-bf16-or-4-bit-python-or-rust-and-how-it-stands-against-vllm-j72 Read discussion