UniversalRAG: A New Framework for Cross-Modal Retrieval 60

What is UniversalRAG?

UniversalRAG is a new type of Retrieval-Augmented Generation (RAG) framework that can retrieve multiple types of content, including text, images, and videos. It can also select information chunks of different sizes as needed, such as passages, documents, video clips, or complete videos. The framework includes a "modality-aware router" that automatically selects the most appropriate content type based on the requirements, avoiding bias toward any specific type. It can also flexibly choose whether training is needed to optimize retrieval performance based on the needs of different tasks. Through testing, UniversalRAG has been shown to outperform traditional systems across multiple test scenarios, with more accurate and effective retrieval results.

Core Features

• Modality-Aware Routing Mechanism: Traditional RAG methods usually force data from all modalities into a unified representation space, which leads to the "modality gap" problem—retrieval tends to favor content in the same modality as the query. UniversalRAG introduces a modality-aware routing mechanism that dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it.

• Multi-Granularity Retrieval: In addition to modality differences, UniversalRAG also considers data granularity (i.e., the size or unit of each entry in the corpus). It subdivides each modality into multiple granularity levels—for example, text is divided into passage-level and document-level, while video is divided into short clips and complete videos. This design allows for fine-grained retrieval based on the complexity and scope of the query.

• Multimodal Support: UniversalRAG supports retrieval across multiple modalities such as text, images, and videos, and can dynamically select the most appropriate modality based on the needs of the query.

Main Objectives

• Multimodal Knowledge Retrieval: Design a modality-aware routing mechanism to avoid the modality gap problem caused by forcing all modalities into a unified representation space. It can dynamically identify the most appropriate modality-specific corpus and perform targeted retrieval.

• Multi-Granularity Knowledge Integration: Organize each modality into multiple granularity levels, such as passage-level, document-level, image-level, clip-level, and video-level. This allows fine-tuned retrieval based on the complexity and scope of the query, adapting more flexibly to different query requirements.

Research Methodology

• Modality-Aware Routing Mechanism: Maintain an independent embedding space for each modality and introduce a routing module. This module can dynamically select the most appropriate modality-specific corpus for retrieval based on the modality requirements of the query, avoiding the modality gap.

• Multi-Granularity Corpus Construction: Consider data granularity by organizing each modality into multiple granularity levels, enabling the system to select knowledge sources of appropriate granularity based on the complexity and scope of the query, providing more precise and comprehensive answers.

• Routing Module Implementation:

• Training-Free Routing: Leverages the knowledge and reasoning capabilities of pre-trained large language models (such as GPT-4o) by providing instructions and contextual examples to classify the query and select the retrieval type. No additional training is required, but it may be limited by the biases and knowledge boundaries of the pre-trained model.

• Trained Routing: Trains the routing module by constructing a training dataset, enabling it to more accurately predict the optimal retrieval type for a given query and better adapt to specific tasks and dataset requirements.

Experimental Setup and Results

Experimental Setup

• Experiments were conducted on multiple benchmark datasets covering different modalities (text, image, video) and retrieval settings. Several Large Vision-Language Models (LVLMs) were used as generators, and multiple baseline methods were set up for comparison.

Experimental Results

• UniversalRAG achieved higher average scores than all baseline methods across all benchmark datasets, with outstanding performance on queries requiring multimodal knowledge.

• The trained routing model outperformed the training-free routing model in terms of routing module performance, but the training-free routing model still outperformed other baseline methods, indicating that zero-shot routing is effective within the UniversalRAG framework.

• Supporting text and video corpora at different granularities significantly improves the performance of UniversalRAG. Compared with models without granularity control, it can select the appropriate granularity level based on the query to generate more accurate answers.

• On cross-domain datasets, GPT-4o as the training-free routing model achieved the highest routing accuracy and average QA scores, while the trained routing model suffered from overfitting due to insufficient query diversity in the training data. Introducing an ensemble routing approach that combines the advantages of trained and training-free routing achieved better performance on cross-domain datasets.

Advantages

• Improved Factual Accuracy: By retrieving information from knowledge sources of multiple modalities and granularities, UniversalRAG can more comprehensively support the model's responses and reduce errors caused by a single modality or insufficient granularity.

• Flexibility: The framework can flexibly handle different types of queries, whether they are simple factual questions or complex analytical questions.

• Scalability: By extending the routing logic, UniversalRAG can seamlessly integrate new modalities without modifying existing modality-specific retrievers.

Project Page: https://universalrag.github.io/

Paper: https://arxiv.org/abs/2504.20734