HorizonMath

Overview

Can AI make progress on important, unsolved mathematical problems? We introduce HorizonMath, a benchmark of 113 predominantly unsolved problems across eight domains in mathematics and the mathematical sciences, paired with an open-source evaluation framework. It targets the generator-verifier gap: problems where discovery requires mathematical insight, but candidate answers can be checked by straightforward computation. Unknown solutions make the benchmark resistant to data contamination. Across nine model configurations from seven frontier model families, most score below 10%. We report six distinct research discoveries: GPT-5.4 Pro finds three, and GPT-5.6 Sol Max reproduces these and finds three additional solutions. HorizonMath is an open challenge and growing community resource, where each verified solution is a candidate contribution to the mathematical literature.

Benchmark Overview

The August 2026 paper evaluates a snapshot of 113 problems: 10 solved calibration problems and 103 problems in the unsolved tiers at benchmark assembly. The public dataset continues to grow beyond this evaluated snapshot.

Key Results

Six distinct discoveries across the unsolved tiers. GPT-5.6 Sol Max leads with 13 accepted solutions out of 113 problems (11.5%), including seven calibration problems and six unsolved-tier problems.

Evaluations use pass@1. For GPT-5.4 Pro problems with a verified first-attempt solution, one additional query explored further improvements. Counts in the last column overlap across models; they represent six distinct problems, not thirteen separate discoveries.

How Verification Works

Models submit self-contained Python functions. The framework executes candidates in a sandbox, checks correctness, and uses an LLM-based compliance checker to screen for forbidden operations.

Closed-form answers must use finite expressions built from permitted constants and functions. Numerical quadrature, root-finding, infinite series, and fitted numerical constants are disallowed. The compliance checker is imperfect, so automated evaluation reduces the need for expert review without replacing it.

Example Problems

These examples from Section 5 of the draft cover all three evaluation modes: closed-form discovery, bound improvement, and new construction.

def proposed_solution():
    from mpmath import mp
    mp.dps = 100
    result = ...  # finite symbolic expression
    return result

def proposed_solution():
    return {
        "n": 7, "k": 5,
        "rows": [
            [0, a01, a02, a03, a04, a05],
            ...  # 7 rows total
        ]
    }

def proposed_solution():
    # Each L is a 10-by-10 integer array with entries 0 through 9.
    return {"squares": [L1, L2, L3]}

Comparison with Existing Benchmarks

Comparison as reported in Table 1 of the August 2026 draft.

BibTeX

@article{wang2026horizonmathmeasuringaiprogress,
      title={HorizonMath: Measuring AI Progress Toward Mathematical Discovery with Automatic Verification},
      author={Erik Y. Wang and Sumeet Ramesh Motwani and James V. Roggeveen and Eliot Hodges and Dulhan Jayalath and Charles London and Kalyan Ramakrishnan and Cheng Zhang and Flaviu Cipcigan and Philip Torr and Alessandro Abate},
      year={2026},
      eprint={2603.15617},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.15617},
}