MinerU: An All-in-One Open-Source PDF Data Extraction Tool 135

MinerU is a tool that converts PDFs into machine-readable formats (such as markdown and JSON), making it easy to extract content into any format. MinerU was born out of the pre-training process of InternLM. MinerU preserves the structure of the original PDF document, extracting text, images, image descriptions, tables, and other content. It automatically recognizes and converts LaTeX and HTML formats, provides automatic OCR, and supports multiple languages.

MinerU Key Features:

• Removes elements such as headers, footers, footnotes, and page numbers to ensure semantic coherence

• Outputs text in human reading order, suitable for single-column, multi-column, and complex layouts

• Preserves the structure of the original document, including headings, paragraphs, and lists

• Extracts images, image descriptions, tables, table captions, and footnotes

• Automatically recognizes and converts formulas in documents to LaTeX format

• Automatically recognizes and converts tables in documents to HTML format

• Automatically detects scanned PDFs and PDFs with garbled text, and enables OCR functionality

• OCR supports detection and recognition of 84 languages

• Supports multiple output formats, such as Markdown for multimodal and NLP, JSON sorted by reading order, and intermediate formats with rich information

• Supports multiple visualization results, including layout visualization, span visualization, etc., making it easy to confirm output quality and perform inspection efficiently

• Supports CPU and GPU environments

• Compatible with Windows, Linux, and Mac platforms

MinerU Core Features and Advantages:

• Magic-PDF Module: Focused on PDF document processing, it intelligently identifies and removes non-body content such as headers and footers, while accurately preserving structures like headings, paragraphs, and lists. It supports the extraction of images, tables, and formulas, ensuring the converted Markdown format is both accurate and easy to read.

• Magic-Doc Module: Designed for web pages and e-books, it can extract formal content from web pages.

• Multimodal Content Processing: MinerU not only processes text but also effectively extracts and handles multimodal content such as images, tables, and formulas.

• Multilingual Support: MinerU supports 84 languages, including both Traditional and Simplified Chinese.

• Diverse Formats: Supports multiple output formats and visualization results, adapts to CPU and GPU environments, and is compatible with multiple platforms.
  

• Automatic Recognition and Conversion: Recognizes and converts formulas to LaTeX format, tables to LaTeX or HTML format, and can automatically detect and enable OCR functionality.
  

MinerU is mainly applied in fields such as academic research, market analysis, legal document processing, and knowledge management, enabling us to efficiently extract key information from large volumes of documents, thereby accelerating the data preparation process and providing high-quality data support for large model training, knowledge graph construction, and more.

MinerU was developed by the OpenDataLab team of the large model data foundation at Shanghai Artificial Intelligence Laboratory (Shanghai AI Laboratory), and was released at WAIC (World Artificial Intelligence Conference) in 2024, quickly gaining attention on GitHub and becoming a popular Python project.