DeepSeek just open-sourced its inference engine
Repository · https://github.com/deepseek-ai/deepinfer
First seen by Oriel on .
AI summary
DeepInfer is an open-source offline inference framework for DeepSeek-V4.1-Flash, derived from the system powering DeepSeek’s large-scale production services.
- It includes optimized kernels, attention and MoE parallelism, asynchronous scheduling, structured outputs, deterministic sampling, and batch-invariant inference.
- DSpark uses draft tokens and confidence estimates to schedule verification; its maximum draft length defaults to 5.
- Bounded replay reduces decoder prefill work by replaying a retained suffix that must cover the SWA window and maximum backtracking length.
- The release supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, but does not include a KV cache system or replica load balancing.
- Building requires Linux, GCC 13 or later, Python 3.12, and either CUDA 13 or CANN 9.2; some inference features are NVIDIA-only.
#deepinfer #llm inference #deepseek #gpu kernels
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 2,346, latest observation ; +2,289 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 891, latest observation ; +884 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50163277 | https://github.com/deepseek-ai/DeepInfer Read discussion |