ColQwen-Omni: An Open-Source Multimodal RAG Model 66

小编 · 2025-07-19

ColQwen-Omni is an open-source multimodal RAG model built on ColQwen2 and Qwen2-VL, and the first open-source model to extend the ColBERT retrieval paradigm to the audio modality.

ColQwen-Omni: An Open-Source Multimodal RAG Model

ColQwen-Omni is an open-source multimodal RAG model, built on ColQwen2 and Qwen2-VL, and is the first open-source model to extend the ColBERT retrieval paradigm to the audio modality.

It supports cross-modal retrieval and can handle text, audio, image, and other multimodal data. The training dataset contains 127,460 query-page pairs, with 63% from public academic dataset training sets and 37% from synthetic datasets, and it possesses zero-shot audio retrieval capabilities.

ColQwen-Omni: An open-source multimodal RAG model

Core Highlights

Cross-modal retrieval: Supports text-to-audio, audio-to-image, and more; all modality data is retrieved in the same vector space.
Direct audio/video processing: No transcription needed; processes raw audio and video data directly, preserving emotion, tone, and visual details.
Ultra-fast retrieval: 30 minutes of audio processed in 10 seconds, short videos each segment in under 10 seconds, and a single query takes less than 100 milliseconds.
TMRoPE synchronization: Time-aligned multimodal embeddings enable precise alignment of video and audio frames.
Lightweight and efficient: With 3 billion parameters, it can run on Colab's free GPU.

ColQwen-Omni Technical Details

Training strategy: Trained with colpali-engine==0.3.11, built on Qwen2.5-Omni-3B-Instruct, and adopts a ColBERT-style multi-vector representation strategy.
Training data: Contains 127,460 query-page pairs, covering public academic datasets and synthetic datasets.
Dynamic resolution processing: Supports dynamic image resolution input, without scaling or cropping images, preserving the original aspect ratio.
Zero-shot audio retrieval capability: Although the training data only contains image-text pairs, the model can still retrieve audio content.

ColQwen-Omni technical details

Applicable Scenarios

Audio and video retrieval: Text-to-TED talk, audio-to-YouTube cover, and more.
Document Q&A: PDF/image Q&A, without OCR, capable of parsing layouts and charts.
Education and research: Retrieve lecture videos/notes and perform cross-modal analysis.
Entertainment analysis: Search short videos/podcast content while preserving emotion and ambient sounds.
Privacy-sensitive scenarios: Can run locally on Colab, with no need to upload data.

Key Questions

What are the characteristics of the ColQwen2.5-Omni training dataset?

The model's training dataset contains 127,460 query-page pairs, with 63% from public academic dataset training sets and 37% from synthetic datasets (including PDF pages scraped from the web and pseudo-questions generated by VLM). The dataset is designed to be entirely in English to facilitate research on zero-shot generalization to non-English languages. Among these, 2% of the samples are used as a validation set for tuning hyperparameters. Moreover, no multi-page PDF documents are used in both ViDoRe and the training set, avoiding evaluation contamination. The language model pretraining corpus and multimodal training may contain multilingual data.

How does ColQwen2.5-Omni differ from ColPali in image processing?

ColQwen2.5-Omni accepts dynamic image resolutions without scaling to change the aspect ratio, creating up to a maximum of 1024 image patches at maximum resolution; ColPali alters the image's aspect ratio.

How to use ColQwen2.5-Omni for audio retrieval?

First, install colpali-engine with a version higher than 0.3.11 (you can install it via pip install git+https://github.com/illuin-tech/colpali); then import the relevant libraries and load the model and processor; process the audio dataset and create a data loader to generate audio embedding vectors; finally, define a retrieval function, input the query text, generate query embedding vectors through the model and score them against the audio embedding vectors, obtain the _topk results, and you can then display the corresponding audio.