Crawl4LLM: A Web Crawler Built for LLM Pre-Training 194

What is Crawl4LLM?

Crawl4LLM is a web crawler tool developed jointly by Tsinghua University and Carnegie Mellon University, specifically designed for large language model (LLM) pre-training. It can intelligently assess the usefulness of web pages for LLM pre-training, prioritizing valuable pages and crawling fewer useless ones. By crawling only 21% of the web data, it achieves the same performance as crawling all the data, thereby improving data quality and significantly increasing crawling speed.

Project Overview:

• Source: Open-sourced as a collaboration between Tsinghua University and Carnegie Mellon University.

• Goal: To address the low crawling efficiency and poor data quality of traditional crawlers, making LLM pre-training more efficient.

Main Features of Crawl4LLM:

Intelligent Page Selection

• Smart Evaluation: Automatically determines whether a web page is useful for LLM pre-training and prioritizes crawling useful ones.

• High Efficiency: Crawls far fewer web pages than traditional crawlers, yet achieves similar or even better results, with approximately 5x efficiency improvement.

Multiple Crawling Methods

• Intelligent Mode: The primary mode, automatically selecting valuable web pages to crawl.

• Random Crawling Mode: Used when precise selection of web page content is not required.

• Link-Count-Based Crawling Mode: Can be used when large amounts of data need to be crawled.

State Management and Recovery

• Periodic Saving: Periodically saves the crawler's state. If interrupted, it can resume from where it left off without losing data.

Data Visualization Tools

• Intuitive Viewing: Includes a data browsing tool to view crawled data and monitor progress and results at any time.

Compatibility and Integration

• DCLM Framework Integration: Can extract document IDs, retrieve document content, and seamlessly connect with the DCLM (deep learning model) pre-training framework, providing data directly for model training.

Technical Details of Crawl4LLM:

• Modular Architecture: Includes modules such as a crawler scheduler, web page parser, and data storage.

• Error Handling: Has strong error correction and automatic retry capabilities to ensure stable data collection.

• Data Quality Management: Can deduplicate and clean data to ensure high and consistent data quality.

Use Cases of Crawl4LLM:

• LLM Pre-training: Provides high-quality data for large-scale LLM pre-training.

• Dataset Construction: Build your own datasets.

• Search Engine Optimization: Analyze and improve SEO strategies.

• Web Monitoring and Analysis: Monitor the web in real time.

• Web Monitoring and Analysis Applications: Perform sentiment analysis and other data analysis.

How to Use Crawl4LLM:

1. Preparation:

• Obtain the ClueWeb22 dataset.

• Create a Python virtual environment and install the necessary libraries (numpy, tqdm, fasttext, pyyaml, wandb).

• Download the DCLM FastText classifier and place it in the specified directory.

2. Create a Configuration File: Create a YAML configuration file in the configs/ directory, specifying parameters such as the dataset path, seed document list, output directory, number of documents selected per iteration, number of worker threads, crawling state save frequency, maximum number of documents to crawl, document selection method, ranking method, and whether to enable wandb logging.

3. Start the Crawler: Run python crawl.py crawl --config in the command line.

As a highly specialized crawler system, Crawl4LLM is well-suited for pre-training large language models. Its advanced algorithms and technical features can significantly improve both the efficiency and quality of data crawling, thereby accelerating the development of large language models.

GitHub Repository: https://github.com/cxcscmu/crawl4llm

Paper: https://arxiv.org/pdf/2502.13347