DeepSeek Open Source Week Day 6: DeepSeek-V3/R1 Inference System with a 545% Cost-Profit Margin 193

The promised 5-day Open Source Week turned out to bring an extra surprise on Saturday with the launch of the DeepSeek-V3/R1 Inference System.

What is the DeepSeek-V3/R1 Inference System?

The DeepSeek-V3/R1 Inference System is a high-performance inference system newly launched by DeepSeek, designed to improve inference efficiency by optimizing throughput and latency. The system adopts large-scale cross-node Expert Parallelism (EP) technology, significantly improving the efficiency of GPU matrix computation, and reduces latency by distributing expert modules across different GPUs.

Key Features

High Throughput and Low Latency Optimization:

• Adopts cross-node Expert Parallelism (EP) technology, significantly expanding batch size and improving GPU matrix computation efficiency.

• Through a Prefill-Decode disaggregated architecture, different degrees of parallelism are employed in the prefill and decode stages.

Intelligent Load Balancing:

• In the prefill and decode stages, load balancing is achieved for core attention computation and dispatch/send workloads respectively.

• In the Mixture of Experts (MoE) model, the expert computation workload on each GPU is balanced.

Computation-Communication Overlap:

• Adopts a dual-batch overlap strategy, splitting the same batch of requests into two microbatches to hide communication costs and improve overall throughput.

• In the decode stage, a five-stage pipeline technique is adopted to achieve seamless overlap between computation and communication.

Large-Scale Cross-Node Expert Parallelism (EP):

• Only 8 out of 256 experts per layer are activated, and the high sparsity characteristic of the model requires an extremely large overall batch size.

• The prefill stage uses routed experts EP32 and MLA/shared experts DP32, while the decode stage uses routed experts EP144 and MLA/shared experts DP144.

Efficient Resource Utilization:

• During daytime peak hours, inference services are deployed on all nodes; during nighttime low-load hours, inference nodes are reduced and resources are allocated to research and training.

Key Technologies

• Cross-Node Expert Parallelism (EP): The DeepSeek-V3/R1 model can activate a large number of expert networks, but only activates 8 out of 256 experts per layer, generating a highly sparse computation structure.

• Load Balancing: The system design ensures that the computation load is evenly distributed across nodes, thereby effectively utilizing all resources.

• Cost and Profit Margin: According to the model's latest evaluation, assuming a GPU rental cost of $2 per hour, the total cost is $87,072 per day, while the theoretical daily total revenue can reach $562,027, with a cost-profit margin as high as 545%

Applications

Online Inference Services:

• All DeepSeek-V3/R1 inference services run on H800 GPUs, with precision consistent with training.

• Provides high-throughput and low-latency online inference services, suitable for all user requests from web, APP, and API.

Commercialized and Non-Commercialized Services:

• Some services have been commercialized, while web and APP access remains free.

• Automatically applies off-peak discounts during non-peak hours, optimizing resource utilization and cost efficiency.

Statistics

• In the past 24 hours, the combined peak node occupancy of V3 and R1 inference services reached 278, with an average occupancy of 226.75 nodes (each node contains 8 H800 GPUs).

• Each H800 node provides an average input throughput of approximately 73,700 tokens/second during the prefill stage, or an output throughput of approximately 14,800 tokens/second during the decode stage.

• If all tokens are billed at DeepSeek-R1's pricing, the daily total revenue would reach $562,027, with a profit margin of 545%.

Review of the First Five Days of Open Source Projects

Day 1: FlashMLA, an efficient decoding kernel for NVIDIA Hopper GPUs that optimizes the performance of Multi-head Latent Attention (MLA), significantly improving the response speed of AI tools in content creation.

Day 2: DeepeEP, the first open-source communication library for Mixture of Experts (MoE) model training and inference, optimizing communication efficiency in large-scale distributed training and reducing latency.

Day 3: DeepGEMM, an FP8 computation library supporting dense and MoE models, specifically designed for NVIDIA Hopper architecture GPUs, significantly improving computational efficiency and hardware utilization.

Day 4: DualPipe, a bidirectional pipeline parallelism algorithm designed to optimize the overlap of computation and communication in V3/R1 model training. EPLB, an Expert Parallelism Load Balancer, specifically designed to solve the load imbalance problem of expert parallel tasks in large-scale AI models.

Day 5: 3FS, a high-performance distributed file system.