Are There Any Good Benchmarks Comparing OpenAI API Models?

API
1

I’m looking for benchmark results that compare OpenAI models specifically on mathematical reasoning. Most of the discussions I find are focused on coding or general reasoning, but I’m interested in seeing how the current models perform on benchmarks such as AIME, FrontierMath, or other math-focused evaluations.

Does anyone have links to benchmark comparisons or personal experience using OpenAI models for math-heavy workloads?

2

Welcome to the community, @bar1s … Paging @EricGT for you if he has the time.

There’s a lot of other smart people here too, so hopefully one of them chimes in.

Good to have you with us.

3

Thanks @PaulBellow

A note up front: I am not a mathematician, just someone on this forum with some familiarity with the question.

Without more specifics, such as the area of mathematics, the level of mathematics, and the intended goal — for example, solving problems, proving results, formalizing proofs, or assisting with exploration — this is difficult to answer in a general way.

Even with those details, I would be cautious about treating any single benchmark as measuring “mathematical reasoning” in the broad sense. Benchmarks can be useful, but they usually measure performance on a particular kind of task under a particular setup.

Two mathematicians who have occasionally written about using AI are:

  • Timothy Gowers

A recent experience with ChatGPT 5.5 Pro

  • Terence Tao

For something very recent and closer to the Lean/formalization side, see:

Jacobian challenge

That thread may be especially relevant if the question is not just about solving math problems, but about AI-assisted formalization or proof-related workflows.

4

Regarding my personal experience with math workloads, the answer is putting together the LLM with something like Lean 4 or even just Python’s SymPy via Code Interpreter. I think standard LLMs fail at raw calculation

5

If you are not already a member of the Lean 4 Zulip forum, it is worth joining. It is free, and the Lean community is in a much better position to help with questions about combining LLMs with Lean 4.

What do you mean by “raw calculation” here?

Do you mean arithmetic computation, symbolic manipulation, multi-step mathematical derivations, formal proof construction, or something else? Those are different tasks, and LLMs tend to fail in different ways depending on which one you mean.