Script Planner Assistant▸ 
AI Apps 
AI News Magma: Microsoft's Multimodal AI Agent Foundation Model for Versatile Scenarios
小编 ·
Magma: Microsoft's Multimodal AI Agent Foundation Model for Versatile Scenarios
What is Magma?
Magma is a multimodal ai agent foundation model released by Microsoft that can handle complex interactions in both virtual and real-world environments, enabling tasks such as image captioning and visual question answering, video captioning and question answering, UI navigation, and robot manipulation.
Features of Magma
Multimodal capabilities: Supports tasks such as image captioning and question answering, video captioning and question answering, UI navigation, and robot manipulation.
Interaction between the digital and physical worlds: Capable of handling tasks in both virtual and real environments.
Versatility: A single model possesses general image and video understanding abilities, while also being able to generate goal-driven visual plans and actions.
Advanced performance: Excels at multimodal tasks, particularly in spatial understanding and reasoning.
Scalable pretraining strategy: Can learn from unlabeled videos and exhibits strong generalization capabilities.
Technical Principles of Magma
Multimodal pretraining: Combines image, video, and action data for large-scale pretraining within a unified framework, learning cross-modal connections.
Set-of-Mark (SoM): Marks actionable objects in images to help the model ground its actions.
Trace-of-Mark (ToM): Marks the motion trajectories of objects in videos to enhance the understanding of temporal dynamics.
Vision-language integration: Uses a convolutional network to encode visual information into a token sequence, combined with a language model to generate actions or language descriptions.
Generalization and fine-tuning: The pretrained model has zero-shot generalization capabilities and can be further improved through fine-tuning.
Cross-task adaptation: Applicable to a wide range of tasks (such as UI navigation, robot manipulation, image and video understanding), demonstrating strong generalization capabilities.
Installation and Usage
Clone the repository:
git clone https://github.com/microsoft/Magma.git cd Magma
Install dependencies:
conda create -n magma python=3.10 -y conda activate magma pip install --upgrade pip pip install -e .
Inference example
from PIL import Image
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
model = AutoModelForCausalLM.from_pretrained("microsoft/Magma-8B", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("microsoft/Magma-8B", trust_remote_code=True)
model.to("cuda")
image = Image.open("example.jpg").convert("RGB")
convs = [
{"role": "system", "content": "You are an agent that can see, talk and act."},
{"role": "user", "content": "\nWhat is in the image?"}
]
prompt = processor.tokenizer.apply_chat_template(convs, tokenize=False, add_generation_prompt=True)
inputs = processor(images=[image], texts=prompt, return_tensors="pt").to("cuda")
with torch.inference_mode():
generate_ids = model.generate(**inputs)
response = processor.decode(generate_ids[0], skip_special_tokens=True).strip()
print(response)Application Areas of Magma
Intelligent surveillance and security: Analyzes video streams in real time, predicts behavior, and enhances security monitoring capabilities.
Autonomous driving: Processes multimodal data to assist autonomous driving systems.
Robot manipulation: Guides robots to perform complex tasks and adapt to different hardware.
UI navigation and interaction: Efficiently completes multi-step operations on web pages or device interfaces.
Education and personalized learning: Analyzes student performance and provides customized teaching plans.
Medical diagnosis: Combines imaging and medical records to assist doctors in formulating diagnostic and treatment plans.
Content creation: Combines image generation and text creation to provide creative inspiration.
Intelligent assistants: Provides task navigation for virtual assistants and guides them through complex operations.
Video analysis and prediction: Describes video content and predicts the next action.
Smart home and automation: Helps home robots learn new tasks and improves automation levels.
Magma GitHub Repository:https://github.com/microsoft/Magma
Magma Project Official Website:https://microsoft.github.io/Magma/
Magma Arxiv Paper:https://www.arxiv.org/pdf/2502.13130
