SuperGPQA: A Knowledge Reasoning Benchmark from ByteDance Doubao 182

What is SuperGPQA

SuperGPQA is a new comprehensive benchmark jointly launched by the Doubao large model team at ByteDance and the M-A-P open-source community. It is primarily designed to comprehensively evaluate the knowledge level and reasoning capabilities of large language models across 285 graduate-level disciplines.

This benchmark features a powerful human-machine collaborative filtering mechanism, which combines the responses of large language models with feedback from experts to continuously refine questions and eliminate content that is of little value or ambiguously phrased. Its coverage is exceptionally broad, ranging from common disciplines like mathematics, physics, and computer science to relatively niche fields such as light industry, agriculture, and service science.

Features of SuperGPQA:

• Broad discipline coverage: SuperGPQA covers 285 graduate-level disciplines, including mainstream fields such as mathematics, physics, and computer science, as well as long-tail disciplines like light industry, agriculture, and service science. This enables SuperGPQA to comprehensively evaluate the reasoning capabilities of large language models (LLMs) across diverse knowledge domains.

• Human-machine collaborative filtering mechanism: By combining LLM responses with expert feedback, SuperGPQA employs iterative refinement to eliminate trivial or ambiguous questions, ensuring high-quality, in-depth questions.

• Curated high-quality dataset: The dataset is constructed through a three-step process of expert screening, standardized transcription, and multi-layer quality inspection. On average, each question has 9.67 options, and 42.33% of the questions require mathematical computation or formal reasoning.

• Comprehensive model performance evaluation: It enables detailed comparison of performance across different models and supports various evaluation methods, including zero-shot and few-shot.

• Question quantity and difficulty: It contains 26,529 specialized questions, with an average of 9.67 options per question, and 42.33% of the questions require mathematical computation or formal reasoning, effectively testing model performance on high-difficulty tasks.

Applications of SuperGPQA:

• Evaluating AI models: Assessing the reasoning capabilities of large language models across different knowledge domains to identify their strengths and weaknesses.

• Academic research: Providing researchers with a standardized testing framework to help them develop more powerful artificial intelligence models.

• Education sector: Can be used to develop intelligent educational tools such as automated knowledge assessment systems.

• Industry applications: Evaluating the professional knowledge reasoning capabilities of AI models in industries such as healthcare, legal, and finance, enhancing the level of intelligence in these sectors.

Paper:https://arxiv.org/pdf/2502.14739

HuggingFace:https://huggingface.co/datasets/m-a-p/SuperGPQA

GitHub Repository:https://github.com/SuperGPQA/SuperGPQA