dots.llm1: Xiaohongshu's Hi Lab Releases Its First Open-Source LLM 76

Xiaohongshu (Little Red Book) open-sourced its first large model, dots.llm1, on June 7, 2025. Developed in-house by Xiaohongshu Hi Lab (Humanistic Intelligence Laboratory), dots.llm1 is a 142-billion-parameter Mixture of Experts (MoE) model that activates only 14 billion parameters during inference, maintaining high performance while significantly reducing training and inference costs.

Model Features

• Large-scale natural training data: Trained on 11.2 trillion tokens of non-synthetic, high-quality data, which is quite rare among currently open-source large models. This enables strong performance in Chinese-language benchmarks, with an average score of 91.3 surpassing DeepSeek's open-source V2 and V3, as well as Alibaba's open-source Qwen2.5 32B and 72B.

• Architecture design: It uses a unidirectional decoder Transformer architecture, but replaces the feed-forward network with MoE. During inference, it does not activate all expert networks; instead, it dynamically selects a small subset of expert networks to perform computation based on the properties of input tokens. This sparse activation approach greatly reduces computational requirements while maintaining high performance. The MoE of dots.llm1 consists of 128 routed experts and 2 shared experts, with each expert network being a two-layer feed-forward network using the SwiGLU activation function. For each input token, a routing mechanism selects the 6 most relevant expert networks, plus the 2 shared experts, activating a total of 8 expert networks.

• Attention layer optimization: It uses the classic multi-head attention (MHA) mechanism, with the addition of RMSNorm normalization to prevent excessively large or small input values from affecting model training, thereby improving the stability and performance of the model.

• Load balancing strategy: An auxiliary-loss-free load balancing strategy is introduced on the MoE module.

Model Performance

• Chinese-language tests: Excellent performance on Chinese benchmarks. For example, it scored 92.6 on CLUEWSC semantic understanding, leading the industry; achieved 92.2 on the C-Eval comprehensive knowledge test, surpassing all comparison models including DeepSeek-V3; scored 80.5 on average across Chinese tasks, a clear lead; and with an average score of 91.3, it surpassed DeepSeek's open-source V2 and V3 as well as Alibaba's open-source Qwen2.5 32B and 72B.

• Mathematical reasoning: Scored 84.8 on the MATH500 baseline test, approaching the industry top level; scored 33.1 on the AIME24 competition problems, demonstrating its ability to handle complex mathematical problems; scored 40.6 on CNMO24 (China Mathematical Olympiad), with an overall mathematics score of 60.9.

• English and coding capabilities: On English benchmarks such as MMLU, DROP, and GPQA, it performs comparably to Qwen, with an average score of 64.4; coding capability average score reaches the level of Qwen2.5.

• Instruction and human preference alignment: On instruction and human preference alignment benchmarks such as IFEval, AlpacaEval2, and ArenaHard, the average score is 77.9, indicating the model's ability to accurately understand and execute complex instructions.

Cost Control Advantages

Compared to Qwen2.5-72B, dots.llm1 reduces GPU hours per trillion training tokens from 340K to 130K. This efficiency improvement is attributed to its unique communication-computation overlap scheme and efficient grouped GEMM implementation.

Model Support and Application Scenarios

• Supported features: Supports both Chinese and English, with a context length of 32,768 tokens, and also supports the function call feature. The official MMLU-Pro evaluation score is better than Qwen3-235b-a22b, but comparable to the latest DeepSeek-V3.

• Application scenarios: Combined with the massive high-quality UGC content accumulated on Xiaohongshu's lifestyle sharing platform, as well as nearly 600 million daily searches and 70% of monthly active users using the search function, dots.llm1 has unique application advantages in lifestyle scenario conversations, creative writing, and life experience Q&A. Xiaohongshu is also building its own AI technology ecosystem around this.

Open-source address:https://huggingface.co/rednote-hilab/dots.llm1.base/tree/main