Reasoning models overturn expert answer keys in research mathematics

Frontier artificial intelligence models are fundamentally altering the evaluation of machine reasoning by outperforming human baselines and identifying flaws in expert-authored benchmarks.

Recent evaluations of advanced reasoning architectures have exposed a structural shift in how frontier models handle complex mathematics. Earlier benchmarks such as GSM8K and the MATH dataset have reached saturation, with leading systems regularly scoring above 90 per cent. In response, evaluation teams designed harder testing suites to probe genuine research-level competence.

FrontierMath, an evaluation suite developed by Epoch AI in collaboration with dozens of professional mathematicians, was built to resist data contamination and memorisation. The benchmark consists of hundreds of unpublished, research-tier problems spanning number theory, algebraic geometry, topology, and combinatorics. Most problems require hours or days for human experts to resolve. When the benchmark launched, standard frontier models, including GPT-4o and Claude 3.5 Sonnet, solved fewer than two per cent of the tasks.

New reasoning models built on test-time compute scaling have shifted these performance baselines. Recent evaluations demonstrate that systems designed to allocate additional compute to internal chains of thought can resolve advanced mathematical problems previously considered inaccessible to automated systems. The introduction of OpenAI’s o3 model demonstrated this trajectory across multiple technical benchmarks, recording an 87.5 per cent score on ARC-AGI under high-compute configurations and competitive programming ratings surpassing 2,700 Elo on Codeforces.

More striking than raw accuracy figures is the behaviour of frontier reasoning systems during the verification phase. When evaluation teams examined discrepancies between model outputs and reference grading sheets, they found that several model responses were correct despite conflicting with the ground truth. The models had not hallucinated; instead, they had identified subtle oversights, edge cases, and erroneous assumptions embedded in the reference answers provided by human mathematicians.

This inversion marks a critical juncture for model evaluation. For years, benchmark integrity relied on the assumption that human subject-matter experts represented an absolute upper bound of correctness. In research-level domains, however, verifying long-horizon mathematical proofs imposes severe cognitive strain. A model that spends minutes exploring thousands of search branches can identify obscure counterexamples that human authors overlook during manual peer review.

The mechanism behind these leaps relies on reinforcement learning applied to reasoning trajectories. Rather than generating text solely through next-token prediction, reasoning models perform search, self-correction, and verification before committing to a final output. In formal sciences, where intermediate steps can be verified for logical consistency, this search process allows the model to backtrack when an error is detected. As compute budgets expand at inference time, models systematically exhaust false pathways that frequently mislead human problem-solvers.

The ability to correct expert keys creates acute practical bottlenecks for benchmark developers. Generating unreleased, novel problem sets requires significant intellectual labour from elite specialists. If human experts are prone to systemic errors on their own questions, validating benchmark integrity becomes exponentially more expensive. Researchers must now deploy multi-stage verification pipelines, often using peer-review panels and independent automated proof assistants, simply to confirm whether an unexpected model answer is brilliant or flawed.

Similar dynamics are appearing across physics and chemistry evaluations. On the GPQA Diamond benchmark, which tests graduate-level science reasoning, reasoning models routinely match or exceed the accuracy of PhD-level domain specialists who lack access to external search engines. The compression of the gap between automated generation and expert verification is forcing evaluation bodies to reconsider how long-form reasoning is audited.

As frontier systems move further into abstract domains where gold-standard solutions cannot be easily crowdsourced, the boundary between evaluating intelligence and conducting novel scientific research continues to blur. If human specialists can no longer reliably grade frontier models without machine assistance, how will researchers independently verify the next generation of mathematical discoveries?