Oriel — AI & technology worth noticing, every day.

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

Post · https://dev.to/gde/six-serving-engines-one-8-gb-gpu-gemma-4-e2b-on-an-rtx-3050-ad7

llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one 8 GB RTX 3050, measured with the same prompts, the same load sweep and the same GSM8K and BFCL questions. GSM8K is level on every engine; vLLM is the fastest up to 16 requests and within 3% of SGLang beyond.

First seen by Oriel on .

AI summary

This article compares six inference serving engines for Gemma 4 E2B on an 8 GB RTX 3050, finding vLLM best for low-to-medium concurrency and SGLang slightly best under heavy load.

  • All six engines use the same 4-bit values and produce nearly identical results: GSM8K scores range from 0.8992 to 0.9030, while BFCL ranges from 0.9125 to 0.925.
  • For one request, vLLM with Google's W4A16 drafter reaches 230.5 tokens per second, versus 158.9 for SGLang and 147.4 for llama.cpp with their best drafters.
  • At 64 requests with 128 input and 512 output tokens, SGLang reaches 1,711 tokens per second and vLLM 1,658, while llama.cpp reaches 633 and mistral.rs 308.
  • The drafter improves vLLM by 1.90x but slows mistral.rs to 0.72x; SGLang's drafter falls to 0.30–0.49x of its no-drafter throughput at 64 requests.
  • SGLang requires two patches for this checkpoint, llama.cpp gains no speed from using more GPU memory, and mistral.rs effectively supports only four concurrent requests on this card.

#inference serving #gemma 4 #gpu benchmarks #speculative decoding

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 24, latest observation ; +2 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 1, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
DEV Community4978026https://dev.to/gde/six-serving-engines-one-8-gb-gpu-gemma-4-e2b-on-an-rtx-3050-ad7 Read discussion