Script Planner Assistant▸ 
AI Apps 
AI News Ovis: Alibaba International's Latest Open-Source Multimodal Model
小编 ·
Ovis: Alibaba International's Latest Open-Source Multimodal Model
Alibaba International released its latest multimodal large model Ovis on September 19, 2024. The Ovis model demonstrates outstanding capabilities across multiple fields, particularly in mathematical reasoning Q&A, object recognition, text extraction, and complex task decision-making.
According to data from the multimodal evaluation platform OpenCompass, the Ovis1.6-Gemma2-9B version surpasses several well-known open-source models including Qwen2VL-7B, InternVL2-26B, and MiniCPM-V-2.6 on the authoritative comprehensive evaluation benchmark OpenCompass, ranking first among open-source models under 30 billion parameters. Notably, Ovis even outperforms the closed-source large model GPT-4o-mini on certain tasks.
Core Capabilities of Ovis:
Multimodal Task Processing: Ovis can excellently handle multimodal tasks such as visual perception reasoning, mathematical and scientific understanding, and analysis of real-life scenarios. It not only helps understand complex mathematical concepts, but also assists in analyzing financial reports and even guides cooking processes based on images, such as making fish and chips.
Technical Details: The highlight of Ovis's design lies in the way it structurally aligns visual and text embeddings. By introducing a learnable visual embedding table, it converts continuous visual features into probabilistic tokens, which then generate structured visual embeddings. This overcomes the limitations of fusing visual and text information in traditional multimodal models.
Performance and Advantages: Ovis achieves impressive scores in tasks such as mathematical reasoning and visual understanding, comparable to some models with significantly higher parameter counts. It also provides better control over hallucination phenomena and error rates compared to models of the same class, ensuring high text quality and accuracy.
Open-Source and Business-Friendly: Released under the Apache 2.0 open-source license, Ovis is very friendly for commercial applications, encouraging broader use and secondary development.
Architectural Innovation: Through innovative architectural design, including a dynamic subgraph scheme, Ovis supports processing images with extreme aspect ratios and high resolutions. Combined with comprehensive data optimization strategies covering Caption, VQA, OCR, Table, Chart, and other multimodal data directions, it enhances the overall performance of multimodal tasks.
Ovis Application Cases:
Image Recognition and Processing: Ovis can accurately recognize and translate handwritten content, handle complex mathematical formulas, and for example, provide detailed recipe preparation steps by recognizing images.
Autonomous Driving: Integrating data from cameras, radar, and LiDAR.
Medical Diagnosis: Used for image analysis and diagnostic support.
Video Content Understanding: Analyzing and generating video content.
Image Description Generation: Automatically generating image descriptions.
Visual Q&A: Supporting text extraction and Q&A in multiple languages.
Ovis Open-Source Information:
The Ovis series models are released under the Apache 2.0 License.
The data, models, training, and inference code for Ovis 1.0 and 1.5 are open-source.
In the Ovis1.6 series, the weights for Ovis1.6-Gemma2-9B have also been open-sourced.
Developers can obtain the models and code on GitHub.
The open-source code can be used for commercial purposes and improvements.
Ovis Technical Advantages:
Innovative Architecture Design: Overcoming the limitations of MLP connector architectures in MLLMs.
High-Resolution Image Processing: Compatible with high-resolution images.
Comprehensive Data Optimization: Significantly improving performance in multimodal Q&A, instruction following, and other tasks.
Outstanding Model Performance: Ranking first overall on OpenCompass.
Hallucination phenomena and error rates are significantly lower than models of the same class.
Ovis GitHub Address:
https://github.com/AI4DC-AI/Ovis
The Alibaba International team stated that data over the past six months shows merchants' demand for AI continues to grow, with API calls doubling every two months on average. Ovis will undoubtedly help more merchants improve operational efficiency.
