Oriel — AI & technology worth noticing, every day.

DeepSeek just open-sourced its inference engine

Repository · https://github.com/deepseek-ai/deepinfer

First seen by Oriel on .

AI summary

DeepInfer is an open-source offline inference framework for DeepSeek-V4.1-Flash, derived from the system powering DeepSeek’s large-scale production services.

  • It includes optimized kernels, attention and MoE parallelism, asynchronous scheduling, structured outputs, deterministic sampling, and batch-invariant inference.
  • DSpark uses draft tokens and confidence estimates to schedule verification; its maximum draft length defaults to 5.
  • Bounded replay reduces decoder prefill work by replaying a retained suffix that must cover the SWA window and maximum backtracking length.
  • The release supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, but does not include a KV cache system or replica load balancing.
  • Building requires Linux, GCC 13 or later, Python 3.12, and either CUDA 13 or CANN 9.2; some inference features are NVIDIA-only.

#deepinfer #llm inference #deepseek #gpu kernels

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 2,346, latest observation ; +2,289 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 891, latest observation ; +884 over observed period since

Aliases

SourceExternal IDURL
Hacker News50163277https://github.com/deepseek-ai/DeepInfer Read discussion