Gemma 4 in Rust on TorchTPU: A Native TPU Rig With PyO3 and MCP
Post · https://dev.to/gde/gemma-4-in-rust-on-torchtpu-a-native-tpu-rig-with-pyo3-and-mcp-47j1
Gemma 4 E2B with the model in Rust, served from a TPU v5e-1 VM through PyO3 and a C++ shim over ATen, with no tch and no pinned PyTorch, an OpenAI endpoint, an MCP server and a local chat UI.
First seen by Oriel on .
AI summary
The article documents a Rust implementation of Gemma 4 E2B served from a Cloud TPU v5e-1 through TorchTPU, reporting 12.20 tokens per second and exact 64-token greedy agreement with Transformers on CPU.
- A 154-line C++ shim exposes selected ATen operations, letting the Rust crate build against the VM's installed PyTorch without tch's PyTorch-version constraint.
- PyO3 bindings release the GIL during decoding and keep TPU tensors and the ring cache on a dedicated thread; the service offers streaming OpenAI-compatible endpoints.
- A fresh TPU VM starts serving in 4 minutes 29 seconds, and the measured median decode rate is 12.20 tokens per second across five requests.
- The rig is 11.8 times slower than the compared vLLM setup; it handles one request at a time, uses greedy decoding, and has no tool-call parsing.
- A laptop shim provides a local chat page and OpenAI-compatible address, with request cancellation and a demo MCP server.
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 15, latest observation ; +8 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 3, latest observation ; +2 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 4964848 | https://dev.to/gde/gemma-4-in-rust-on-torchtpu-a-native-tpu-rig-with-pyo3-and-mcp-47j1 Read discussion |