Ikhtisar benchmark

Graduate-level science questions in physics, chemistry and biology, designed to be Google-proof.

Thousands of expert-written questions across dozens of fields, the hardest closed-ended academic benchmark in wide use.

Second-generation abstraction and reasoning tasks measuring fluid intelligence on novel visual patterns.

American Invitational Mathematics Examination problems solved without tool assistance.

Research-grade mathematics problems crafted by professional mathematicians.

Competition mathematics dataset spanning algebra to number theory.

Real GitHub issues resolved end-to-end; the industry standard for agentic coding.

Harder, contamination-resistant successor to SWE-Bench with commercial-repo issues.

Continuously refreshed competitive programming problems immune to data contamination.

Scientific coding problems requiring domain knowledge and multi-function solutions.

Multi-step command-line tasks in realistic terminal environments.

Function-level Python completion, the classic code-generation benchmark.

Real computer-use tasks across operating systems: browsers, office apps and file management.

Multi-tool orchestration tasks across real-world APIs.

Tool calling over the Model Context Protocol across complex server graphs.

Dual-control conversational agent tasks in a simulated retail environment.

Long-horizon professional agent tasks with multi-app workflows.

Short fact-seeking questions measuring hallucination rates.

Harder 14-subject version of MMLU with ten-way multiple choice.

MMLU translated across 14 languages.

Massive Multi-discipline Multimodal Understanding with college-level image reasoning.

Harder multimodal reasoning successor of MMMU.

Reasoning over scientific charts and figures from arXiv papers.

GUI grounding on high-resolution professional application screens.

Multi-round coreference resolution over needle-in-haystack long contexts.