Oriel — AI & technology worth noticing, every day.

vllm-project/vllm

Repository · https://github.com/vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

First seen by Oriel on .

AI summary

The vLLM repository presents an open-source library for high-throughput, memory-efficient LLM inference and serving, combining PagedAttention with optimized execution, distributed inference, and broad model and hardware support.

  • Performance features include PagedAttention for attention key and value memory management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, quantization, and speculative decoding.
  • vLLM supports 200+ Hugging Face model architectures, spanning decoder-only, Mixture-of-Expert, hybrid attention and state-space, multi-modal, embedding, retrieval, reward, and classification models.
  • Serving interfaces include an OpenAI-compatible API server, Anthropic Messages API, and gRPC, alongside streaming outputs, structured generation, tool calling, and reasoning parsers.
  • Distributed inference supports tensor, pipeline, data, expert, and context parallelism, plus multi-LoRA support for dense and MoE layers.
  • Hardware support includes NVIDIA, AMD, and Intel GPUs and x86/ARM/PowerPC CPUs, with additional plugins for platforms including Google TPUs, Huawei Ascend, and Apple Silicon.

#vllm #llm inference #model serving #pagedattention

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Stars

Total repository stars at each observation.

  • GitHub Trending: 95,093, latest observation ; +884 over observed period since

Stars gained (1d)

Daily star gain reported by GitHub Trending. A change compares two daily values; it is not additional stars between observations.

  • GitHub Trending: 441, latest observation ; +343 over observed period since

Forks

Total repository forks at each observation.

  • GitHub Trending: 23,791, latest observation ; +157 over observed period since

Aliases

SourceExternal IDURL
GitHub Trendingvllm-project/vllmhttps://github.com/vllm-project/vllm Read discussion