What hardware can run DeepInfer?

DeepInfer supports Nvidia SM100 Blackwell GPUs and Huawei Ascend 950 neural processing units. Hopper and other older Nvidia architectures are not on the list.

Is all of DeepSeek's inference code included?

No. DeepSeek has not released its key-value cache management and load-balancing code, saying the two are tightly coupled to internal infrastructure. A blog post on Zhihu explains how both work.

How does DeepInfer speed up generation?

It uses speculative decoding through DSpark, where a draft model proposes five tokens at a time and a confidence head predicts which the main model will accept. DeepSeek estimated a 4% to 7% throughput gain at its production load.

Large Language ModelsOpen-Source AIAI Business & IndustrySiliconANGLE AI3m agoOct 21, 2026, 10:01 JST

DeepSeek open-sources DeepInfer inference framework

Send on LINE

3 Key Points

What happened
DeepSeek released DeepInfer on GitHub under the Apache 2.0 license; it supports only DeepSeek-V4.1-Flash and runs on Nvidia SM100 Blackwell GPUs and Huawei Ascend 950 NPUs, while one cluster serves 155 trillion tokens a day at $38 per GPU per hour in off-peak revenue.
Why it matters
For companies weighing whether to serve large models themselves, DeepInfer exposes DeepSeek's approach to keeping output costs low enough to reach a mass user base at a reasonable price, though key-value cache management and load-balancing code remain unreleased.

WHO IT HITSTeams running or evaluating self-hosted model serving, especially those on Nvidia Blackwell or Huawei Ascend hardware, get a production-tested framework they can inspect; operators planning split prefill and decode deployments must size clusters themselves, with DeepSeek's smallest efficient example using 64 GPUs for prefill and 64 for decoding.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

Context & Analysis

DeepInfer arrives with a history. DeepSeek says the same coupling to internal infrastructure kept it from releasing its full engine in April 2025, when it described the software as an early fork of vLLM built around its cluster management tools. It also said maintaining a large open-source project was more than a small research team focused on developing better models had bandwidth for. The new README names vLLM as its original foundation, and DeepInfer now uses components DeepSeek had already released, such as the FlashMLA, DeepGEMM and DeepEP kernel libraries.

The release also fits DeepSeek's recent hardware work. DeepInfer's Ascend support follows a Sept. 30 release in which DeepSeek brought the TileLang kernel programming language to Huawei's chips with the chipmaker's help, and the startup plans to deploy at least 160,000 Ascend 950DT accelerators at a new data center in Inner Mongolia, Bloomberg reported in September.

The timing is notable: the framework arrived a day after the U.S. Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems, including third-party services built on its technology. DeepSeek said inference code for future models will go into the same repository.

FAQ

What hardware can run DeepInfer?: DeepInfer supports Nvidia SM100 Blackwell GPUs and Huawei Ascend 950 neural processing units. Hopper and other older Nvidia architectures are not on the list.
Is all of DeepSeek's inference code included?: No. DeepSeek has not released its key-value cache management and load-balancing code, saying the two are tightly coupled to internal infrastructure. A blog post on Zhihu explains how both work.
How does DeepInfer speed up generation?: It uses speculative decoding through DSpark, where a draft model proposes five tokens at a time and a confidence head predicts which the main model will accept. DeepSeek estimated a 4% to 7% throughput gain at its production load.

SiliconANGLE AIRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.