DeepInfer

DeepInfer is an open-source, high-performance inference framework developed by DeepSeek for executing large language models at scale. It powers DeepSeek's production inference infrastructure, while the public release provides core offline inference capabilities. The framework currently supports DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It integrates optimized computational kernels and distributed parallelism techniques for attention and mixture-of-experts operations. An asynchronous scheduler overlaps CPU scheduling with accelerator execution to improve throughput. Speculative decoding and confidence-based verification help accelerate token generation. Additional capabilities include deterministic sampling, batch-invariant inference, and structured output generation using JSON schemas, regular expressions, or grammars. Complete production serving requires additional infrastructure components.