AnimeGamer: Infinite Anime Life Simulation Powered by Multimodal LLMs 128

What is AnimeGamer?

AnimeGamer is a multimodal large language model project jointly launched by Tencent PCG ARC Lab and City University of Hong Kong. It generates dynamic animation clips through language instructions, simulating interactions with anime characters and game worlds. Players can update character states in real time, experience multi-turn interactions, and enjoy immersive anime life. AnimeGamer can be applied in scenarios such as games, story creation, and education.

AnimeGamer Features

• Dynamic Animation Generation: Capable of generating dynamic animation clips to showcase character actions and behaviors.

• Character State Updates: The system updates the character's stamina, social, and entertainment values based on game states.

• Multi-turn Interaction: Supports multi-turn game state generation, allowing players to continuously interact with the game world through language instructions.

• Contextual Consistency: The generated game states maintain visual and semantic consistency, ensuring player immersion.

How AnimeGamer Works

• Animation Clip Encoding and Decoding: Uses an animation clip encoder to encode animation clips into motion-aware multimodal representations, then decodes these representations into high-quality video clips through a decoder based on a video diffusion model.

• Game State Prediction: Utilizes MLLM to predict the multimodal representation of each game state, with inputs including historical animation clip representations and current instructions. MLLM outputs include dynamic animation clips and character state updates.

• Adaptive Decoder Training: Through an adaptive training phase, the decoder is optimized to generate high-quality video clips, reducing artifacts in the generated videos.

AnimeGamer Application Scenarios

• Anime Character Interaction Games: Players can play as characters from anime films and interact with the game world through language instructions.

• Story Creation and Visualization: Can be used to create and visualize stories, generate coherent animation clips, and support multi-turn interactions.

• Education and Training: In education and training scenarios, it can generate dynamic simulation environments to help students better understand and engage with learning content.

• Entertainment and Interactive Experiences: Can be used to develop new entertainment and interactive experiences, such as virtual reality (VR) and augmented reality (AR) applications.

AnimeGamer Tutorial

1. Download the checkpoints for AnimeGamer and Mistral-7B, and save them in the ./checkpoints folder.

2. Set up the inference environment by running the following commands:

git clone https://github.com/TencentARC/AnimeGamer.git
cd AnimeGamer
conda create -n animegamer python==3.10 -y
conda activate animegamer
pip install -r requirements.txt

3. Generate motion-aware multimodal representations and update character states:

python inference_Decoder.py

4. Modify the instructions in ./game_demo to customize the gaming experience.

Related Links

Project Homepage:https://howe125.github.io/AnimeGamer.github.io

GitHub:https://github.com/TencentARC/AnimeGamer

Model:https://huggingface.co/TencentARC/AnimeGamer

Paper:https://arxiv.org/abs/2504.01014