What Is OpenAI's GPT-4o and What Can It Do? 231

Openai just released GPT-4o at 10 AM Pacific Time on Monday, May 13. It is a new AI model that combines text, images, video, and voice into an all-in-one model. It can respond to user requests in real time and answer you via voice in real time—you can interrupt it at any time. GPT-4o has visual capabilities, can recognize objects and make rapid responses and answers based on vision, and possesses very strong logical reasoning abilities. It can also detect emotions from the user's voice. It is 2x faster than GPT-4 Turbo, and 50% cheaper!

What is GPT-4o?

GPT-4o ("o" for "omni") is a step towards more natural human-computer interaction—it accepts any combination of text, audio, and image as input, and generates any combination of text, audio, and image as output. It can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in a conversation. Its performance on English text and code matches that of GPT-4 Turbo, with significant improvements on non-English text, while the API is also faster and 50% cheaper. Compared to existing models, GPT-4o is especially strong in vision and audio understanding.

Before GPT-4o, you could use Voice Mode to converse with ChatGPT, with average latency of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4). To achieve this, Voice Mode was a pipeline of three separate models: a simple model that transcribes audio to text, GPT-3.5 or GPT-4 which takes in text and outputs text, and a third simple model that converts that text back to audio. This process meant that the main source of intelligence, GPT-4, lost a lot of information—it could not directly observe tone, multiple speakers, or background noise, and it could not output laughter, singing, or express emotion.

With GPT-4o, we trained a new model end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. Since GPT-4o is our first model combining all these modalities, we are still only scratching the surface of exploring the model's capabilities and its limitations.

GPT-4o Model Evaluation

On traditional benchmarks, GPT-4o achieves GPT-4 Turbo-level performance in text, reasoning, and coding intelligence, while setting new high watermarks in multilingual, audio, and vision capabilities.

GPT-4o Language Tokenization

20 languages were selected as representatives of different language families for compression by the new tokenizer.

Model Safety and Limitations

GPT-4o has built-in safety in its cross-modal design through techniques such as filtering training data and refining model behavior through post-training. We have also created new safety systems to provide guardrails for voice output.

We evaluated GPT-4o according to our Preparedness Framework and in line with our voluntary commitments. Our evaluations on cybersecurity, CBRN, persuasiveness, and model autonomy indicated that GPT-4o does not score above medium risk in any of these categories. This evaluation involved running a set of automated and human evaluations throughout the entire model training process. We tested both pre-safety-mitigation and post-safety-mitigation versions of the model using custom fine-tuning and prompts to better elicit the model's capabilities.

GPT-4o has also undergone extensive collaboration with over 70 external red teamers and external experts in fields such as social psychology, bias and fairness, and misinformation, to identify risks introduced or amplified by the newly added modalities. We leveraged these experiences to develop safety interventions that improve the safety of interacting with GPT-4o. We will continue to mitigate newly discovered risks.

We recognize that GPT-4o's audio modalities introduce a variety of new risks. Today, we are publicly releasing text and image inputs along with text outputs. In the coming weeks and months, we will work on the technical infrastructure, post-training usability, and safety necessary to release the other modalities. For example, at launch, audio output will be limited to a selection of preset voices and will comply with our existing safety policies. We will share more details on the full range of GPT-4o modalities in an upcoming system card.

We hope to gather feedback to help identify tasks where GPT-4 Turbo still outperforms GPT-4o, so we can continue to improve the model.

Model Availability

GPT-4o is our latest effort to push the boundaries of deep learning, this time moving in the direction of practicality. Over the past two years, we have invested heavily in improving efficiency at every layer of the stack. As the first result of this research, we are able to more broadly offer GPT-4-class models. GPT-4o's capabilities will be rolled out iteratively (starting today with expanded red team access).

GPT-4o's text and image capabilities start rolling out in ChatGPT today. We are offering GPT-4o in the free tier and up to 5x higher message limits to Plus users. We will roll out the alpha version of the new Voice Mode GPT-4o in ChatGPT Plus over the coming weeks.

Developers can now also access GPT-4o in the API as a text and vision model. Compared to GPT-4 Turbo, GPT-4o is 2x faster, half the price, and has 5x higher rate limits. We plan to roll out support for the new audio and video capabilities of GPT-4o to a small group of trusted partners in the API over the coming weeks.

More details on GPT-4o: https://openai.com/index/hello-gpt-4o/