When OpenAI dropped its shiny new “o3” AI model in December, it came with a bold claim: it crushed some seriously hard math problems like a genius robot with a calculator and a caffeine addiction.
Specifically, OpenAI said o3 could solve over 25% of the questions in a notoriously tough math benchmark called FrontierMath. That was a big deal, because the next-best AI model on the market could only manage around 2%. Naturally, the internet and AI nerds everywhere freaked out.
But here’s the twist: the version of o3 that most of us actually get to use doesn’t seem to be quite that smart.
😬 The Reality Check: Independent Tests Say… Only 10%
A research group called Epoch AI — the pwoplw who created the FrontierMath benchmark in the first place — decided to double-check OpenAI’s results. They tested the public version of o3, the one regular people and companies have access to.
Their verdict? o3 only scored around 10%, not 25%. Not even close.
So, what gives?
Epoch suspects OpenAI used a supercharged internal version of o3 with more computing power and maybe even a cherry-picked test set when they first ran the numbers. In other words, they may have used a tricked-out Ferrari version of the model to show off, then handed the public a Honda Civic.
🤖 Wait, Are We Using a Different Model?
Yep. Kind of.
Another research group, the ARC Prize Foundation, said that the o3 model available to the public is a “different model” entirely — one that’s tuned for real-world stuff like chatting, not hardcore math problem solving.
OpenAI has even admitted this. One of their own engineers, Wenda Zhou, said last week that the o3 now in production is optimized for speed and real-world usefulness — not necessarily benchmark dominance. In plain English: it’s faster, cheaper, and more user-friendly, but not quite the same brainiac that crushed the math tests in December.
🔍 Why This Matters (And Why It Keeps Happening)
First off, don’t panic — OpenAI isn’t giving us junk. In fact, some of their newer models, like o3-mini-high and o4-mini, actually perform better than o3 on FrontierMath. Plus, an upgraded version of o3 (called o3-pro) is supposedly just around the corner.
But this whole situation is a classic case of “AI benchmark bingo.” In the race to show off the smartest bots on the block, companies sometimes make big claims using internal models or souped-up versions we’ll never see. And when independent testers get their hands on the public versions, things don’t always add up.
And this isn’t just an OpenAI thing — other big players are doing it too. Elon Musk’s xAI was recently accused of fudging benchmark results. Meta got caught hyping scores from a version of their model that was not what they actually released. Even Epoch AI faced heat for not disclosing OpenAI’s involvement in FrontierMath until after o3 was released.
🧠 Bottom Line: Don’t Believe Every Benchmark Hype
AI is moving fast — like, “blink and there’s a new model” fast. But if there’s one takeaway from this whole saga, it’s this:
Don’t take AI benchmark scores at face value.
Especially when they come from the companies selling the models. Always look for third-party tests and independent verification. And remember — the flashiest number on the chart isn’t always the one you actually get in your hands.
OpenAI has released o3, their highly anticipated reasoning model, along with o4-mini, a smaller and cheaper model that succeeds o3-mini.
We evaluated the new models on our suite of math and science benchmarks. Results in thread! pic.twitter.com/5gbtzkEy1B
— Epoch AI (@EpochAIResearch) April 18, 2025
Re-testing released o3 on ARC-AGI-1 will take a day or two. Because today's release is a materially different system, we are re-labeling our past reported results as "preview":
o3-preview (low): 75.7%, $200/task
o3-preview (high): 87.5%, $34.4k/taskAbove uses o1 pro pricing…
— Mike Knoop (@mikeknoop) April 16, 2025
