DeepSeek Open-Sources 'DeepInfer' Inference Engine Code for Everyone

Chinese AI company DeepSeek announced on Tuesday that it has open-sourced the "DeepInfer" inference engine used to run its models in production, releasing it on GitHub under the Apache 2.0 license.

In its initial version, the engine supports only the "DeepSeek V4.1 Flash" model and runs on Nvidia SM100 data center processors and Huawei Ascend 950 chips, according to The Next Web.

The release features optimized code for splitting workloads across multiple chips, including speculative decoding that drafts multiple tokens in advance and verifies them in a single pass, alongside tools that ensure output consistency and enforce specific formats. The engine was initially built on open-source vLLM and also utilized SGLang and Nvidia's TensorRT-LLM. However, it does not constitute a complete standalone service, as it excludes a caching system, disaggregation of request processing from answer generation, and hardware load balancing, according to the GitHub page.

DeepSeek measured performance figures across four SM100 processors for request processing and 64 for output generation, figures provided by the company itself and not yet independently tested. The report noted that the repository had surpassed 8,000 stars on GitHub by Tuesday afternoon.

What Do These Terms Mean?

Inference: Running a trained model to answer a request or complete a task; it is the stage in which the model operates after training.

Speculative Decoding: A technique in which the model proposes several words or tokens in advance and then verifies them all at once, speeding up text generation.

Apache 2.0 License: An open-source software license that allows using, modifying, and distributing the code, subject to conditions regarding copyright and liability disclaimers.

Weights: The numerical values learned by the model during training that determine its behavior when generating responses.