MoshiVis: A Vision-Speech Model That Listens, Sees, and Discusses Images in Fluid Voice 191

What is MoshiVis?

MoshiVis is a visual speech model developed on top of Moshi, and retains Moshi's low latency and natural conversational abilities, enabling it to discuss image content in a natural conversational style. It supports multi-turn conversations, can describe the text content in images, and answer questions about images.

It incorporates visual information into the speech stream through a cross-attention mechanism while maintaining low latency and low memory usage. MoshiVis offers multiple backend support (such as PyTorch, Rust, MLX) and has released pretrained models and synthetic visual dialogue datasets, making it suitable for real-time speech interaction scenarios.

Core Features of MoshiVis

• Vision and Speech Integration: MoshiVis adds visual understanding capabilities on top of Moshi, enabling it to incorporate image information into speech conversations. By extending the core Transformer architecture, it introduces a cross-attention mechanism to inject visual information into the speech stream.

• Low Latency and Low Memory Usage: To maintain low latency and reduce memory usage, MoshiVis shares the cross-attention projection weights across layers and regulates the visual input stream through a gating mechanism.

• Natural Conversational Style: While integrating visual capabilities, the model retains Moshi's original natural conversational style.

Technical Architecture of MoshiVis

• Base Model: Built on Moshi, a 7B-parameter speech-text foundation model.

• Visual Encoder: Uses a pretrained frozen 400M visual encoder from the PaliGemma2 family.

• Adapter Parameters: Adds approximately 206M adapter parameters on top of Moshi for processing visual information.

• Cross-Attention Mechanism: Injects visual information into the speech stream through a cross-attention module, while regulating the influence of the visual input through a gating mechanism.

MoshiVis Model Release

• Model Variant: Released a MoshiVis model based on Moshika (female voice), including complete model weights, including the speech codec, text tokenizer, image encoder, and base Moshi model.

• Backend Support: Provides three backend support options, including PyTorch, Rust, and MLX, supporting different quantization formats (such as BF16, Q8_0, etc.).

• License: Model weights (excluding the visual encoder) are released under the CC-BY 4.0 license, and the visual encoder is released under the Gemma license.

How to Use MoshiVis

• WebUI Frontend: Provides a WebUI frontend that supports echo cancellation to enhance model performance. Users can use it through pre-built static versions or by compiling the source code themselves.

• Backend Operation:

• PyTorch: Requires approximately 24GB of GPU memory and does not support quantization.

• Rust: Supports GPU acceleration and requires the Rust toolschain and CUDA (or Metal on macOS).

• MLX: Supports bfloat16 and quantized (q4, q8) formats.

• Run Commands: Detailed run commands are provided, allowing users to choose different backends and quantization formats as needed.

Related Links:

GitHub Project:https://github.com/kyutai-labs/moshivis

HuggingFace Model:https://huggingface.co/collections/kyutai/moshivis-v01-67cef4acae6a5d75d6d6c883

Demo:https://vis.moshi.chat/

Paper:https://arxiv.org/abs/2503.15633