Alter3: GPT-4 Powers Motion and Conversation Control for a Humanoid Robot 177

Scientists at the University of Tokyo have linked Openai's GPT-4 large language model (LLM) with a humanoid robot. As explained in a newly published paper, this robot, named Alter3, can understand conversational prompts well, thereby controlling the robot's movements and gestures.

Alter3 Project URL:https://tnoinkwms.github.io/ALTER-LLM/

The University of Tokyo team demonstrated Alter3 adopting poses such as taking a selfie, playing the guitar, or pretending to be a ghost when guided by natural language, without the need for explicit programming of each motion. GPT-4, which also powers ChatGPT, is able to understand what people describe in whatever terms they prefer, respond physically, or generate images associated with DALL-E 3 in roughly the same way.

This breakthrough by the scientists bridges the gap in conversational interaction with physical robots, which typically requires fine-grained motion control using specialized hardware-based code. The researchers translate high-level commands into instructions that Alter3 can execute. The robot can learn motions just as a human intuitively picks them up—from basic shuffling to more coordinated movements. Users can guide Alter3's poses and help it distinguish nuances, such as different dance movements.

This is a humanoid robot capable of generating spontaneous movements using a large language model (LLM), specifically GPT-4. This achievement was accomplished by integrating GPT-4 into our proprietary robot, Alter3, effectively linking the LLM with Alter's body movements. Typically, low-level robot control relies on hardware and falls outside the scope of the LLM corpus, which poses a challenge for direct LLM-based robot control.

However, in the case of a humanoid robot like Alter3, direct control is feasible by mapping linguistic descriptions of human movements onto the robot's body through program code. Notably, this approach enables Alter3 to adopt various poses, such as a "selfie" pose or "pretending to be a ghost," and generate a sequence of movements over time without the need for explicit programming of each body part.

This demonstrates the robot's zero-shot learning capability. Furthermore, verbal feedback can adjust poses without the need for fine-tuning.