ViDoRAG: A Retrieval-Augmented Generation (RAG) System for Visual Document Understanding 196

What is ViDoRAG?

ViDoRAG is a visual document understanding Retrieval-Augmented Generation (RAG) system jointly released by Alibaba's Tongyi Lab, the University of Science and Technology of China, and Shanghai Jiao Tong University. It leverages multimodal hybrid retrieval and a multi-agent iterative reasoning mechanism, significantly improving both the accuracy and efficiency of visual document processing. In plain language, it supports retrieval from image-based documents, allowing you to find information among a bunch of image-containing documents, such as "what color is this product" or "what does this article talk about?".

ViDoRAG Framework

The ViDoRAG framework focuses on multi-agent collaboration and dynamic iterative reasoning, with the following key components:

Multimodal Hybrid Retrieval: Combines visual and textual features, and uses a Gaussian Mixture Model (GMM) to dynamically adjust the retrieval strategy for more accurate identification of key information.

Multi-Agent Collaboration:

• Seeker Agent: Quickly filters out relevant images or document snippets to provide global clues.

• Inspector Agent: Performs detailed inspection on the filtered results and generates preliminary answers.

• Answer Agent: Integrates the preliminary answers, checks their accuracy and consistency, and generates the final answer.

Dynamic Retrieval Strategy: Uses GMM to dynamically adjust the number of retrieval results, reducing computation and improving retrieval efficiency.

Features

1. Multimodal Hybrid Retrieval:

• Dual-Channel Visual and Textual Retrieval: ViDoRAG can simultaneously process visual (image) and textual information for retrieval, enabling a more comprehensive understanding of document content.

• Gaussian Mixture Model (GMM) Fusion: Uses GMM to dynamically adjust the weights of visual and textual retrieval results, selecting the optimal retrieval strategy based on query characteristics, reducing noise and computation.

2. Multi-Agent Collaborative Reasoning:

• Seeker Agent: Quickly filters relevant documents or image snippets to narrow down the search scope.

• Inspector Agent: Performs detailed inspection on the filtered snippets, extracts key information, and generates preliminary answers.

• Answer Agent: Integrates the preliminary answers and verifies them to generate the final answer.

• Dynamic Iterative Reasoning: Multiple agents continuously collaborate and iterate, making the answers progressively better.

3. Dynamic Retrieval Strategy:

• Adaptive Retrieval Quantity: ViDoRAG uses GMM to dynamically adjust the number of retrieval results, avoiding the issues caused by fixed-quantity retrieval.

• Efficient Retrieval Workflow: Automatically optimizes the retrieval strategy based on the similarity distribution between queries and the document collection, improving retrieval efficiency.

4. Complex Document Understanding Capability:

• Support for Visually Rich Documents: Can process documents containing text, charts, tables, and complex layouts.

• Multi-Hop Reasoning Capability: Can handle both single-hop and multi-hop reasoning tasks, addressing complex queries that require cross-document understanding.

5. High Accuracy and Performance:

• High Accuracy: Achieves an accuracy of 79.4% on the ViDoSeek benchmark dataset, significantly outperforming other baseline methods.

• Robustness: Multi-agent collaboration and dynamic retrieval strategies enable ViDoRAG to perform stably when handling noisy data and complex scenarios.

Application Scenarios

• Visual Document Processing: Suitable for scenarios involving visually rich documents such as academic papers, reports, and manuals.

• Question Answering Systems: Can serve as a core module in QA systems to answer questions quickly and accurately.

• Smart Office: Helps users quickly extract key information from documents and improve work efficiency.

GitHub Repository: https://github.com/Alibaba-NLP/ViDoRAG