DeepSeek Open Source Week Day 5: 3FS, a High-Performance Distributed File System 246

While GPT4.5 has just been released, DeepSeek's Open Source Week enters the fifth day's open source project. The final day's open source project is: 3FS (Fire-Flyer File System).

What is 3FS (Fire-Flyer File System)?

3FS (Fire-Flyer File System) is a high-performance distributed file system self-developed by DeepSeek, specifically designed to address the challenges of AI training and inference workloads.

Features of 3FS

High Performance:

• Leverages modern SSDs and RDMA networking to provide high throughput and low latency data access.

• Combines the throughput of thousands of SSDs with the network bandwidth of hundreds of storage nodes, enabling applications to access storage resources in a geography-independent manner.

Strong Consistency:

• Ensures strong consistency by implementing Chained Replication with Apportioned Queries (CRAQ), making application code simpler and easier to reason about.

File Interface:

• Employs a stateless metadata service backed by a transactional key-value store (such as FoundationDB).

• Familiar file interface with no new storage API to master.

Diverse Workload Support:

• Data Preparation: Organizes the output of data analysis pipelines into hierarchical directory structures, effectively managing large volumes of intermediate output.

• Data Loader: Eliminates the need for prefetching or data shuffling by supporting random access to training samples across compute nodes.

Checkpoints: Supports high-throughput parallel checkpointing for large-scale training.

• Inference KV Cache: Provides a cost-effective alternative to DRAM caching, offering higher throughput and significantly greater capacity.

Disaggregated Storage and Compute:

• The data storage service is separated from compute nodes and dedicated to storing sample data required for model training.

• Each storage service node is equipped with 16 SSDs of 15TB each and 2 high-speed network cards, delivering powerful read performance and robust network bandwidth, with read/write bandwidth of 7.0TB/s.

Self-Developed Communication Tools:

• Such as hfreduce, which optimizes multi-card parallel communication, replaces NVIDIA's NCCL, and reduces PCIe traffic and computational overhead.

Dedicated Data Format FFRecord:

• Reduces the overhead of opening numerous small files during training by merging multiple small files.

• Improves random batch read performance through the use of sample file offsets.

Applications of 3FS

AI Training and Inference:

• 3FS is specifically designed for AI training and inference workloads and can significantly enhance the efficiency of these tasks.

• Through high performance and strong consistency, 3FS supports large-scale model training and high-throughput inference.

Data Preparation and Loading:

• 3FS's data preparation and loader functionality makes handling large volumes of data more efficient, reducing the need for prefetching and data shuffling.

Checkpoints and Inference Cache:

• 3FS supports high-throughput parallel checkpointing, ensuring stability and fault tolerance during the training process.

• The inference KV cache provides a cost-effective alternative, improving inference throughput and capacity.

Usage of 3FS

Installation and Configuration:

• 3FS can be installed and configured through its open source project on GitHub.

Data Format Conversion:

• Use the FFRecord format for data storage and loading, requiring sample data to be converted into FFRecord format.

• An FFRecord conversion tool is provided to facilitate data format conversion for users.

Integration with PyTorch:

• 3FS is adapted to PyTorch's Dataset and DataLoader interfaces, making it very convenient to load data and initiate training.

GitHub Project URL:https://github.com/deepseek-ai/3FS

Review of the First Four Days' Open Source Projects

Day One: FlashMLA, an efficient decoding kernel for NVIDIA Hopper GPUs that optimizes the performance of Multi-head Latent Attention (MLA), significantly boosting the response speed of AI tools in content creation.

Day Two: DeepeEP, the first open source communication library for Mixture of Experts (MoE) model training and inference, which optimizes communication efficiency and reduces latency in large-scale distributed training.

Day Three: DeepGEMM, an FP8 computation library supporting dense and MoE models, specifically designed for NVIDIA Hopper architecture GPUs, significantly enhancing computational efficiency and hardware utilization.

Day Four: DualPipe, a bidirectional pipeline parallelism algorithm designed to optimize the overlap of computation and communication in V3/R1 model training. EPLB, an Expert Parallelism Load Balancer, specifically addresses load imbalance issues in expert parallel tasks within large-scale AI models.