VRAG-RL: A Vision-Based RAG Framework from Alibaba's Tongyi Team 85

小编 · 2025-06-03

VRAG-RL is a vision-based RAG framework recently released by Alibaba's Tongyi team. It delivers a 30% performance boost on Qwen2.5-VL-3B and a 20% improvement on Qwen2.5-VL-7B.

VRAG-RL: A Vision-Based RAG Framework from Alibaba's Tongyi Team

What is VRAG-RL?

VRAG-RL is a model based on a vision-perception RAG framework recently launched by Alibaba's Tongyi team. This model delivers a 30% performance improvement on Qwen2.5-VL-3B and a 20% improvement on Qwen2.5-VL-7B.

Through reinforcement learning training, VRAG-RL equips the model with multi-turn thinking and reasoning capabilities, progressively enhancing its understanding of vision-language models (VLM) and the accuracy of its answers. It defines a vision-perception action space that enables the model to focus on information-dense regions from coarse to fine, precisely extract key visual information, and improve the VLM's capabilities in retrieval, reasoning, and understanding of visual information.

In addition, VRAG-RL features multimodal retrieval and iterative reasoning capabilities. Through multi-turn interaction, it progressively refines its understanding of visual information and ultimately generates accurate and comprehensive answers. It is suitable for a variety of vision tasks, such as image understanding, chart analysis, complex layout parsing, and other scenarios.

VRAG-RL: A vision-perception RAG framework-based model launched by Alibaba Tongyi team.webp

Limitations of Traditional RAG Methods

Insufficient visual information processing capability: Unable to effectively parse information in images, lacking the ability to understand and reason about visual data.
Fixed pipeline limits dynamic reasoning: Mostly adopt a fixed retrieve-then-generate pipeline, making it difficult to dynamically adjust the reasoning path in complex tasks, which limits the model's ability to mine visual information.
Insufficient retrieval efficiency and reasoning depth: When handling complex vision tasks, it often fails to efficiently locate key information, resulting in generated answers that are not precise enough.

Core Innovations of VRAG-RL

Vision-perception action space: Introduces a variety of vision-perception actions, such as region selection, cropping, and zooming, enabling the model to progressively focus on information-dense regions from coarse to fine granularity and precisely extract key visual information. This coarse-to-fine perception approach not only improves the model's understanding of visual information but also significantly enhances retrieval efficiency.
Reinforcement learning framework: Optimizes the model's reasoning and retrieval capabilities through reinforcement learning (RL). The model interacts with the search engine over multiple turns, autonomously samples single-turn or multi-turn reasoning trajectories, and continuously optimizes based on these samples.
Comprehensive reward mechanism: Designs a comprehensive reward function that includes a retrieval-efficiency reward, a pattern-consistency reward, and a model-based outcome reward. This reward mechanism focuses on the final result while optimizing the retrieval process, enabling the model to acquire relevant information more effectively.
Multi-expert sampling strategy: Combines the reasoning capabilities of large-scale models with the precise annotation capabilities of expert models, enabling the model to learn more effective vision-perception strategies.
Multi-turn interaction training: Based on a multi-turn interaction training strategy, the model progressively optimizes its reasoning process through continuous interaction with the external environment, improving the stability and consistency of reasoning.

Technical Advantages of VRAG-RL

Performance improvement: Experimental results show that VRAG-RL achieves significantly better performance than existing methods across multiple vision-language benchmark datasets. Task types cover a variety of visually rich scenarios, ranging from single-hop to multi-hop reasoning, from pure text understanding to chart recognition and complex layout parsing.
Multi-turn interaction support: Can progressively focus on information-dense regions during the reasoning stage, achieving coarse-to-fine information acquisition.
Efficient training: Introduces the industry-leading GRPO algorithm and simulates real-world application scenarios through a locally deployed search engine, achieving zero cost for search engine calls and making model training more efficient.

Application Scenarios of VRAG-RL

Complex chart analysis: Extract key data from financial reports and scientific charts.
Design draft parsing: Understand and reason about the layout and elements within design drafts.
Document retrieval: Efficiently retrieve information from document knowledge bases that contain images and tables.
Multimodal question answering: Combine text and images for multi-hop reasoning to answer complex questions.