SmolDocling: A Lightweight Vision-Language Model for Converting Complex Documents into Structured Text 137

What is SmolDocling?

SmolDocling is a lightweight vision-language model jointly developed by IBM Research and Hugging Face, designed for end-to-end multimodal document conversion. It contains only 256M parameters and can quickly process documents on consumer-grade GPUs, with a processing time of just 0.35 seconds per page. The core mission of SmolDocling is to convert complex content from any image into editable, structured data.

SmolDocling Features

• DocTags for Efficient Markup: Introduces DocTags, an efficient and compact document representation fully compatible with DoclingDocuments.

• OCR (Optical Character Recognition): Accurately extracts text from images.

• Layout and Localization: Preserves document structure and bounding boxes of document elements.

• Code Recognition: Detects and formats code blocks, including indentation.

• Formula Recognition: Identifies and processes mathematical expressions.

• Chart Recognition: Extracts and interprets chart data.

• Table Recognition: Supports structured table extraction with column headers and row headers.

• Figure Classification: Distinguishes between figures and figure elements.

• Caption Correspondence: Links captions to related images and figures.

• List Grouping: Correctly organizes and structures list elements.

• Full-page Conversion: Processes entire pages, including all elements on the page (code, equations, tables, charts, etc.).

• OCR with Bounding Boxes: OCR region recognition using bounding boxes.

• General Document Processing: Trained on both scientific and non-scientific documents.

• Seamless Docling Integration: Can be imported into Docling and exported in multiple formats (such as HTML, Markdown, etc.).

• Fast Inference: Average processing time of 0.35 seconds per page on an A100 GPU.

Model Expansion and Optimization

• Support for Multiple Instructions: Supports a variety of instructions, such as converting pages to DocTags, converting charts to tables, converting formulas to LaTeX, and more.

• Multilingual Support: Although primarily supporting English, it may be extended to support more languages.

• Continuous Improvement: Improved chart recognition, support for multi-page inference, chemical recognition, and more.

SmolDocling Use Cases

• Academic Research: Quickly convert academic papers and research reports into structured formats for easy extraction of key information.

• Business Document Processing: Automatically convert business contracts, reports, and forms to facilitate document storage, retrieval, and analysis for enterprises.

• Technical Document Management: Convert technical manuals, code documentation, and more into editable formats, supporting accurate recognition and formatting of code snippets.

• Education Sector: Extract content from textbooks and lecture notes (such as formulas, charts) and convert it into easy-to-understand formats.

• Medical Document Processing: Process medical reports and research papers to extract key information and assist in medical decision-making.

• Mobile and Low-resource Device Support: Can run on mobile devices or resource-constrained environments.

How to Use SmolDocling

1. Model Download: SmolDocling can be downloaded from the Hugging Face model repository.

2. Local Deployment: Due to its small size, it can run on ordinary laptops or mobile devices.

3. API Calls: SmolDocling can be invoked through API interfaces provided by Hugging Face.

4. Fine-tuning the Model: Developers can fine-tune the model to adapt to specific scenarios, such as medical report parsing or financial form recognition.

Hugging Face Model Repository:https://huggingface.co/ds4sd/SmolDocling-256M-preview

DEMO:https://huggingface.co/spaces/ds4sd/SmolDocling-256M-Demo