Llama 4: First Multimodal AI Models Using a Mixture-of-Experts Architecture 217

What is Llama 4?

Llama 4 is the latest generation of open-source AI model series released by Meta on April 5, 2025. It includes Llama 4 Scout and Llama 4 Maverick, which are the first multimodal AI models to adopt a Mixture of Experts (MoE) architecture. Scout has 17 billion active parameters, 16 "expert" models, and a total of 109 billion parameters, while Maverick also has 17 billion active parameters but is equipped with 128 "expert" models, with a total parameter count of up to 400 billion. In addition, there is a more powerful version under development, Llama 4 Behemoth, which has 288 billion active parameters, 16 "expert" models, and a total parameter count approaching 2 trillion. These models are trained on large amounts of unlabeled text, image, and video data, and possess visual understanding capabilities.

The Llama 4 Model Series

Llama 4 Scout

• Parameter scale: 17 billion active parameters, 16 experts

• Hardware requirements: Can run on a single NVIDIA H100 GPU

• Context window: Supports an ultra-long context window of up to 10 million tokens

• Performance advantages: Outperforms Gemma 3, Gemini 2.0 Flash-Lite, and Mistral 3.1

Llama 4 Maverick

• Parameter scale: 17 billion active parameters, 128 experts, total parameter count of 400 billion

• Performance advantages: Excellent performance in multimodal tasks, outperforming GPT-4o and Gemini 2.0 Flash

• Reasoning capabilities: Reasoning and coding capabilities comparable to DeepSeek v3, but with less than half the active parameters

Llama 4 Behemoth (Preview)

• Parameter scale: 288 billion active parameters, 16 experts, total parameter count approaching 2 trillion

• Performance advantages: Outperforms GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro across multiple STEM benchmarks

• Role positioning: Serves as a teacher model for distilling other Llama 4 models; currently still in training

Llama 4 Features

• Multimodal capabilities: Llama 4 is a multimodal large model capable of processing and integrating different types of data such as text, video, images, and audio, and converting content between these formats. For example, Llama 4 Scout can analyze image content in complex environments.

• Mixture of Experts architecture: This is one of the core features of Llama 4. This architecture divides the model into multiple "expert" sub-models focused on specific tasks, thereby improving efficiency during training and when answering user queries. For example, Llama 4 Maverick has a total of 400 billion parameters, but only 17 billion parameters are active during actual computation.

• Long context processing capability: Llama 4 Scout's context window can reach up to 10 million tokens, which makes it perform particularly well when handling lengthy documents.

• Voice functionality: Llama 4 introduces native voice processing capabilities, able to directly process voice information without first converting speech to text, inputting it into the model, and then converting text back to speech. This greatly improves the efficiency and fluency of voice interactions.

Applications

• Customer service: Enterprises can use this multimodal AI model to quickly analyze questions raised by users and provide more accurate and efficient responses.

• Education: Intelligent teaching assistants can explain complex concepts by understanding students' questions and supplementing them with images, greatly enhancing learning outcomes.

• Virtual meetings: In virtual meetings, Llama 4 can act as an assistant, helping users with real-time translation, recording meeting highlights, and even simulating participants' speech to improve communication efficiency.

• Smart device control: Through voice commands, users can easily complete tasks such as writing text, sending messages, and controlling smart home devices.

• Information-intensive fields: Llama 4's long context processing capability makes it very useful in information-intensive fields such as medicine, science, engineering, mathematics, and literature.

Technical Characteristics of Llama 4

• Model scale and parameter count: Llama 4 Scout has 17 billion active parameters, 16 "expert" models, and a total parameter count of 109 billion; Llama 4 Maverick also has 17 billion active parameters but is equipped with 128 "expert" models, with a total parameter count of up to 400 billion; Llama 4 Behemoth has 288 billion active parameters and a total of 2 trillion parameters.

• Training data and diversity: All models have been trained on large amounts of unlabeled text, image, and video data to gain broad visual understanding capabilities.

• Architectural optimization: A Mixture of Experts architecture is adopted, achieving higher operational efficiency by decomposing data processing tasks into subtasks and assigning them to smaller "expert" models.

• Inference efficiency: Inference efficiency is improved by reducing model operating costs and latency. For example, Llama 4 Maverick can run on a single NVIDIA H100 DGX host.

Meta estimates the inference cost of Llama 4 Maverick at $0.19 to $0.49 per 1 million tokens (with an input-to-output ratio of 3:1), making it much cheaper than proprietary models like GPT-4o.

Read in detail:https://ai.meta.com/blog/llama-4-multimodal-intelligence/

Model download:https://www.llama.com/llama-downloads/?utm_source=llama-home-hero&utm_medium=llama-referral&utm_campaign=llama-utm&utm_offering=llama-downloads&utm_product=llama