Six Serving Engines on Four Accelerators: Gemma 4 E2B on an RTX 3050, a T4, an L4 and a TPU v5e
The same Gemma 4 E2B, prompts, load sweep and GSM8K and BFCL questions through the same serving engines on an 8 GB RTX 3050, a Tesla T4, an NVIDIA L4 and one TPU v5e chip. Answers hold on every chip; vLLM with its drafter leads on all three GPUs, and SGLang-JAX leads on the TPU.
First seen by Oriel on .
AI summary
This comparison finds that GPU engine choice mainly matters for drafting and high concurrency, while SGLang-JAX is fastest on one TPU v5e chip under load.
- Across 15 CUDA configurations, GSM8K scores range from 0.8954 to 0.9030 and BFCL scores from 0.9150 to 0.9250; drafters change no GPU accuracy.
- vLLM with Google's W4A16 drafter reaches 230.5, 193.0, and 275.3 tokens per second on the RTX 3050, T4, and L4.
- At 64 requests, vLLM leads both shapes on the T4 and L4, while SGLang leads long replies on the 3050 by 3%.
- SGLang-JAX reaches 4,249.4 tokens per second for 64 long replies and 2.52x vLLM on the TPU, but uses bf16 weights and an E2B patch.
- vLLM's TPU drafter does not fit in memory, mistral.rs's T4 drafter fails from a dtype mismatch, and SGLang requires patches on every platform.
#gemma #inference serving #vllm #tpu
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 17, latest observation ; +3 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 1, latest observation ; 0 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 5019892 | https://dev.to/gde/six-serving-engines-on-four-accelerators-gemma-4-e2b-on-an-rtx-3050-a-t4-an-l4-and-a-tpu-v5e-1a0e Read discussion |