UniTok: ByteDance's Open-Source Visual Tokenizer for Generation and Understanding Tasks 180

What is UniTok?

UniTok is an open-source visual tokenizer by ByteDance designed for both generation and understanding tasks. It efficiently converts visual information into tokens and is suitable for a variety of autoregressive generation models and multimodal understanding models. Through innovative visual tokenization techniques, it achieves high-quality visual content generation—such as reconstruction of images and videos—using an extremely low number of tokens (e.g., 256 tokens).

Core Features of UniTok

• Efficient Visual Encoding: UniTok can decompose complex visual information (such as images and videos) into discrete tokens, making it easier for models to process.

• Multimodal Support: It applies not only to images but also to video content, supporting both continuous and discrete tokenization.

• Low Resource Requirements: UniTok maintains high reconstruction quality even with a low number of tokens—for example, the reconstruction error at 256 tokens is only 0.39.

• Flexible Compression Ratio: Users can adjust the number of tokens based on their needs to balance generation quality and computational efficiency.

Application Areas of UniTok

• Visual Generation Tasks: Such as image and video reconstruction, and language-model-driven visual synthesis.

• Multimodal Understanding: Combining textual and visual information to enable more efficient training of multimodal models.

• Diffusion Models: Integrated with diffusion models such as DiT and Latte to generate high-quality visual content.

Technical Implementation of UniTok

UniTok is built upon VQVAE (Variational Quantized Autoencoder) and VAE (Variational Autoencoder) technologies, achieving efficient visual encoding through a two-stage training process. Its training process includes:

• Single-resolution Image Training: Image training performed at a fixed resolution.

• Joint Multi-resolution Image-Video Training: Joint training of images and videos across multiple resolutions.

• VAE Fine-tuning: Fine-tuning the VQVAE model with KL loss to obtain the VAE model.

Advantages of UniTok

• Single Model and Weights: UniTok handles image and video tokenization with a single model and set of weights, simplifying model management and deployment.

• High Adaptability: Supports high-resolution inputs and long videos, adapting to a wide range of application scenarios.

• Strong Compatibility: Compatible with language models and diffusion models, enabling diverse visual generation.

By leveraging multi-codebook quantization techniques, UniTok addresses the bottlenecks of traditional tokenizers in generation and understanding tasks, providing a more efficient solution for multimodal large language models (MLLMs).

Project Page: https://foundationvision.github.io/UniTok/

GitHub Repository: https://github.com/FoundationVision/UniTok