BuboGPT | ByteDance Large Model 137

ByteDance has introduced a new large model named BuboGPT. BuboGPT is an advanced large language model (LLM) capable of integrating multimodal inputs such as text, images, and audio, with the unique ability to align its responses with visual objects. It demonstrates outstanding conversational abilities in understanding aligned or unaligned arbitrary image-audio data.

Through text descriptions, image localization, and sound localization, BuboGPT can accurately determine the source of sounds, and even when there is no direct relationship between audio and images, it can reasonably describe the possible relationship between the two.

Compared with other multimodal large models, BuboGPT leverages the rich information and explicit correspondences between text and other modalities to deliver fine-grained understanding of visual objects and given modalities.

To achieve multimodal understanding, BuboGPT uses a shared semantic space and constructs a visual grounding pipeline that includes a tagging module, a localization module, and an entity matching module.

Using language as a bridge, BuboGPT is able to connect visual objects with other modalities. The researchers also demonstrated BuboGPT's capabilities in image captioning, sound source identification, and more, and open-sourced the code and datasets, releasing a playable demo.

BuboGPT Core Features:

1. Multimodal Understanding: BuboGPT achieves joint multimodal understanding and conversational capabilities across text, vision, and audio.

2. Visual Grounding: BuboGPT can accurately associate text with specific parts of an image, achieving fine-grained visual grounding.

3. Audio Understanding: BuboGPT can accurately describe each sound component within an audio clip, even when some audio segments are too brief for humans to perceive.

4. Aligned and Unaligned Understanding: BuboGPT can handle matched audio-image pairs to achieve perfect aligned understanding, and produce high-quality responses for arbitrary audio-image pairs.