Embodied-Reasoner: An Open-Source Multimodal Embodied Model by Zhejiang University, Alibaba DAMO, and More 124

小编 · 2025-04-04

Embodied-Reasoner is an open-source multimodal embodied model jointly proposed by the Institute of Software CAS, Zhejiang University, Alibaba DAMO Academy, and others. It extends deep reasoning capabilities to embodied interaction tasks, handling complex embodied tasks such as searching for hidden objects, manipulating and transporting items in the AI2-THOR simulator.

Embodied-Reasoner: An Open-Source Multimodal Embodied Model by Zhejiang University, Alibaba DAMO, and More

What is Embodied-Reasoner?

Embodied-Reasoner is an open-source multimodal embodied model jointly proposed by institutions including the Institute of Software of the Chinese Academy of Sciences, Zhejiang University, and Alibaba DAMO Academy. It extends deep reasoning capabilities to embodied interaction tasks and is capable of handling complex embodied tasks, such as searching for hidden objects in the AI2-THOR simulator, manipulating and transporting items, etc.

Embodied-Reasoner：浙江大学和阿里巴巴达摩院等机构联合提出的一个开源的多模态具身模型.jpg

Core Features of Embodied-Reasoner

Deep Reasoning Capability: Possesses multiple reasoning abilities including analysis, spatial reasoning, reflection, and planning.
Multimodal Interaction Processing: Capable of handling long sequences of image-text interaction context.
Environment Interaction Capability: Can autonomously observe the environment, explore rooms, and find hidden objects.
Open-source Models and Datasets:

Provides open-source models in two sizes: 7B and 2B.
The dataset contains 9.3k interactive observation-reasoning-action trajectories, covering 64K images and 8M thinking tokens.

Technical Highlights of Embodied-Reasoner

Task and Trajectory Engine: Automatically synthesizes coherent "observation-thinking-action" trajectories, covering 107 diverse indoor scenes such as kitchens and living rooms, involving 2,100 interactive objects and 2,600 containers.
Long Chain-of-Thought and Diverse Thinking Patterns: Includes analysis, spatial reasoning, reflection, planning, and verification. These coherent image-text interleaved trajectories enhance its spatial and temporal reasoning capabilities.
Iterative Training Pipeline: A three-stage iterative training process combining imitation learning, self-exploration tuning, and self-correction tuning.
Interactive Evaluation Framework: Includes 809 test cases covering 12 novel scenarios, evaluating the model's performance in instruction understanding, key actions, and final state.

Embodied-Reasoner：浙江大学和阿里巴巴达摩院等机构联合提出的一个开源的多模态具身模型.webp

Performance of Embodied-Reasoner

Success Rate: Measures whether the task is successfully completed.
Search Efficiency: Evaluates task efficiency — the more steps, the lower the efficiency.
Task Completeness: Calculates the proportion of predicted actions that belong to key actions.

Application Cases of Embodied-Reasoner

Simulator Experiments: Embodied-Reasoner exhibits spontaneous thinking behaviors, such as analyzing environmental states, reflecting on missed details, reasoning based on the latest observations, and recalling clues for efficient planning. In contrast, general VLMs lacking thinking capabilities perform poorly in long-horizon interaction tasks, easily forgetting the task or repeating searches.
Real-world Experiments: In real environments, Embodied-Reasoner is able to eliminate distractions and ultimately find the target object and complete the task, while OpenAI's o3-mini model fails to formulate a reasonable plan.

Embodied-Reasoner Usage Guide

Training

1. Install Dependencies:

conda create -n llama-factory python=3.11
conda activate llama-factory
git clone -b embodied-reasoner https://github.com/iGangao/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]"
pip install wandb accelerate deepspeed importlib-metadata

2. Data Preparation: Refer to the data/README.md file to understand the format of the dataset files.

3. Run the Training Script: Run the training script.

Evaluation

1. Install Dependencies:

conda create -n embodied-reasoner python=3.9
conda activate embodied-reasoner
pip install -r requirements.txt

2. Run the Evaluation Script: Run the evaluation script.

Data Engine

TaskGenerate.py: Synthesizes task templates and the corresponding key actions.
o1StyleGenerate.py and o1StyleGenerate_ordered.py: Synthesize trajectories for 10 different subtask types respectively; the latter is used to synthesize more complex sequential object transfer tasks.