Oriel — AI & technology worth noticing, every day.

PyTorch on TPU: Hands On With Colab

Post · https://dev.to/gde/pytorch-on-tpu-hands-on-with-colab-5b5g

Five Colab notebooks take Gemma 4 E2B on one v5e chip from 5 tokens per second under Hugging Face generate to 50 with one compiled decode step, and to the full 131,072-token context with a ring cache, against a JAX engine on the same chip.

First seen by Oriel on .

AI summary

The article benchmarks PyTorch on TPU using TorchTPU to run Gemma 4 E2B on a Colab v5e-1, showing compiled decoding and a custom ring cache improve speed and context capacity.

  • A compiled decode step reaches 31.3–33.4 tokens per second, rising to 50.2–50.4 with a 4-bit embedding table; JAX reaches 130.8 tokens per second on the same chip.
  • Transformers unpacks the model's 4-bit weights to bf16 on its first forward pass, while custom table lookup keeps the largest embedding table packed and frees 3.4 GB.
  • The ring cache uses two compiled programs for prompt processing and decoding, supports contexts up to 131,328 tokens, and uses 0.816 GB for its cache.
  • A 130,981-token prompt takes 35.5–39.8 seconds to read, but the record-recall test misses the target; its answer matches the laptop engine's result.
  • The guide includes Colab notebooks and a terminal workflow for running them on a v5e-1 TPU; its benchmarks use Gemma 4 E2B and compare PyTorch with JAX.

#pytorch #tpu #torchtpu #gemma

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 9, latest observation ; +2 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 1, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
DEV Community4908111https://dev.to/gde/pytorch-on-tpu-hands-on-with-colab-5b5g Read discussion