Oriel — AI & technology worth noticing, every day.

Gemma 4 in Rust on TorchTPU: A Native TPU Rig With PyO3 and MCP

Post · https://dev.to/gde/gemma-4-in-rust-on-torchtpu-a-native-tpu-rig-with-pyo3-and-mcp-47j1

Gemma 4 E2B with the model in Rust, served from a TPU v5e-1 VM through PyO3 and a C++ shim over ATen, with no tch and no pinned PyTorch, an OpenAI endpoint, an MCP server and a local chat UI.

First seen by Oriel on .

AI summary

The article documents a Rust implementation of Gemma 4 E2B served from a Cloud TPU v5e-1 through TorchTPU, reporting 12.20 tokens per second and exact 64-token greedy agreement with Transformers on CPU.

  • A 154-line C++ shim exposes selected ATen operations, letting the Rust crate build against the VM's installed PyTorch without tch's PyTorch-version constraint.
  • PyO3 bindings release the GIL during decoding and keep TPU tensors and the ring cache on a dedicated thread; the service offers streaming OpenAI-compatible endpoints.
  • A fresh TPU VM starts serving in 4 minutes 29 seconds, and the measured median decode rate is 12.20 tokens per second across five requests.
  • The rig is 11.8 times slower than the compared vLLM setup; it handles one request at a time, uses greedy decoding, and has no tool-call parsing.
  • A laptop shim provides a local chat page and OpenAI-compatible address, with request cancellation and a demo MCP server.

#rust #torch tpu #gemma #pyo3

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 15, latest observation ; +8 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 3, latest observation ; +2 over observed period since

Aliases

SourceExternal IDURL
DEV Community4964848https://dev.to/gde/gemma-4-in-rust-on-torchtpu-a-native-tpu-rig-with-pyo3-and-mcp-47j1 Read discussion