Tarsier: ByteDance's Large-Scale Vision-Language Model 188

A family of large vision-language models (LVLM) from ByteDance focused on video understanding tasks—video captioning, question answering, video grounding, hallucination benchmarking, and more.

#Ai工具箱 #Ai开源项目

Visit Tarsier: ByteDance's Large-Scale Vision-Language Model

Tarsier: ByteDance's Large-Scale Vision-Language ModelIntroduction

Tarsier is a series of large-scale vision-language models (LVLM) launched by ByteDance, focused on video understanding tasks, including video description, question answering, grounding, and hallucination testing.

Main Features of Tarsier

Video Description Generation: Tarsier can generate high-quality video descriptions covering various details in the video, including actions, scenes, and plots, helping users better understand the video content.
Question Answering Capability: The Tarsier model supports video question answering, allowing users to ask questions based on video content, and the model will provide relevant answers.
Grounding Functionality: Detects and locates the timing of specific events in videos, supports grounding across multiple video segments, and can identify and mark specific objects or scenes in videos.
Hallucination Testing: Through optimized training strategies, Tarsier2 significantly reduces the likelihood of the model generating false information.
Multilingual Support: Supports video description generation in multiple languages.

Tarsier Application Scenarios:

Content Creation: Tarsier can help content creators generate video descriptions, improving video accessibility and search engine optimization (SEO) performance.
Education Sector: In educational videos, Tarsier can provide detailed content descriptions to help students better understand learning materials.
Social Media: Social platforms can leverage Tarsier to generate automatic descriptions of video content, enhancing user experience.
Video Surveillance: In the security monitoring field, Tarsier can analyze video streams and generate real-time descriptions, helping security personnel quickly identify potential threats.
Robotics: Generates detailed step-by-step instructions for specified tasks.
Autonomous Driving: Helps vehicles recognize road conditions and assists in decision-making.

The latest version of the Tarsier model, Tarsier2, has made significant improvements in several aspects, particularly in terms of data volume and diversity. The pre-training data was expanded from 11 million to 40 million video-text pairs, enhancing the model's learning capability. In addition, Tarsier2 introduced fine-grained temporal alignment during the supervised fine-tuning stage, further improving the accuracy of video descriptions and detail-capturing ability. Through Direct Preference Optimization (DPO) training, Tarsier2 can generate video descriptions that better align with human preferences, reducing the likelihood of generating hallucinations.

In terms of performance evaluation, Tarsier2 performs excellently in the DREAM-1K benchmark, with an F1 score 2.8% higher than GPT-4o and 5.8% higher than Gemini-1.5-Pro. Across 15 public benchmarks, Tarsier2 achieved new state-of-the-art results, covering video question answering, video grounding, hallucination testing, and question answering, demonstrating its versatility as a powerful general-purpose vision-language model.