vllm-project/vllm
Repository · https://github.com/vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
First seen by Oriel on .
AI summary
The vLLM repository presents an open-source library for high-throughput, memory-efficient LLM inference and serving, combining PagedAttention with optimized execution, distributed inference, and broad model and hardware support.
- Performance features include PagedAttention for attention key and value memory management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, quantization, and speculative decoding.
- vLLM supports 200+ Hugging Face model architectures, spanning decoder-only, Mixture-of-Expert, hybrid attention and state-space, multi-modal, embedding, retrieval, reward, and classification models.
- Serving interfaces include an OpenAI-compatible API server, Anthropic Messages API, and gRPC, alongside streaming outputs, structured generation, tool calling, and reasoning parsers.
- Distributed inference supports tensor, pipeline, data, expert, and context parallelism, plus multi-LoRA support for dense and MoE layers.
- Hardware support includes NVIDIA, AMD, and Intel GPUs and x86/ARM/PowerPC CPUs, with additional plugins for platforms including Google TPUs, Huawei Ascend, and Apple Silicon.
#vllm #llm inference #model serving #pagedattention
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Stars
Total repository stars at each observation.
- GitHub Trending: 95,093, latest observation ; +884 over observed period since
Stars gained (1d)
Daily star gain reported by GitHub Trending. A change compares two daily values; it is not additional stars between observations.
- GitHub Trending: 441, latest observation ; +343 over observed period since
Forks
Total repository forks at each observation.
- GitHub Trending: 23,791, latest observation ; +157 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| GitHub Trending | vllm-project/vllm | https://github.com/vllm-project/vllm Read discussion |