Alibaba Ovis-U1: A Unified Model for Multimodal Understanding, Image Generation, and Editing 64

小编 · 2025-06-30

On June 29, 2025, Alibaba's International AI team released Ovis-U1, the latest version in the Ovis series. Built on the original architecture, it is a multimodal large model with 300 million parameters.

Alibaba Ovis-U1: A Unified Model for Multimodal Understanding, Image Generation, and Editing

On June 29, 2025, Alibaba International's AI team launched the multimodal large model Ovis-U1, the latest version in the Ovis series. It is developed based on the original architecture and has a parameter scale of 3 billion.

This model integrates multimodal understanding, image generation, and editing capabilities. Its core components include a visual tokenizer, a visual embedding table, and a large language model. By optimizing the alignment between visual and text embeddings, it addresses some of the limitations of traditional multimodal models and performs better in complex scenarios. It supports text and image input and excels in tasks such as mathematical reasoning and object recognition. It can accurately identify objects or handwritten text in images, and also generate or modify images based on instructions.

Core Capabilities

Multimodal understanding: Capable of analyzing complex visual scenes and textual content, answering image-related questions, and completing tasks such as visual question answering (VQA) and image captioning.
Text-to-image generation: Generates high-quality images based on textual descriptions, adaptable to various styles and complex scenario requirements.
Image editing: Performs operations such as adding, adjusting, replacing, and removing image elements through textual instructions, and supports style transfer.

Technical Architecture

Visual decoder: Employs a diffusion-based Transformer architecture (MMDiT) to convert text embeddings into high-quality images.
Bidirectional token refiner: Strengthens the interaction between text and visual embeddings, enhancing image synthesis and editing performance.
Visual encoder: Fine-tuned based on pre-trained models (such as Aimv2-large-patch14-448) to adapt to multimodal tasks.
Adapter: Connects the visual encoder with the multimodal large language model (MLLM), aligning the two types of embedding data.
Multimodal large language model: Serves as the core module, processing textual and visual information and supporting multi-task processing.

Alibaba Ovis-U1: Features a trinity of multimodal understanding, image generation, and image editing capabilities

Training Method

Ovis-U1 jointly trains multimodal understanding, image generation, and editing tasks, sharing knowledge to improve generalization capabilities. The training is divided into six stages to progressively optimize the performance of each task.

Data Composition

Multimodal understanding data: Sourced from public datasets such as COYO, Wukong, and Laion, as well as internally developed data.
Image generation data: Detailed descriptions based on the Laion5B and JourneyDB datasets, as well as generated by pre-trained models.
Image editing data: Data covering tasks such as reference image-driven generation and pixel-level control.

Performance Optimization

In image editing, instruction execution is precisely controlled by adjusting the text and image guidance coefficients (CFG). Benchmarks such as OpenCompass and GenEval are used to comprehensively evaluate model capabilities.

Application Scenarios

Content creation: Assists artists and video editors with creative ideation.
Advertising and marketing: Generates advertising images and posters based on product and audience descriptions.
Game development: Generates images of game scenes, characters, and props.
Architectural design: Outputs architectural concept drawings and interior layout plans.
Scientific research: Visualizes complex scientific phenomena and experimental scenarios.

Continuing the open-source tradition of the Ovis series, the model code, weights, and training data have been publicly released under the Apache 2.0 license on Hugging Face and GitHub, allowing developers to quickly reproduce and deploy the model.