Discussions about AI benchmarks and how AI laboratories publish them are becoming more widely visible.
Elon Musk’s AI startup, xAI, was accused this week by an OpenAI employee of disseminating false benchmark results for its most recent AI model, Grok 3. Igor Babushkin, one of xAI’s co-founders, maintained that the business was correct.
In the middle is where the truth is.
Grok 3’s performance on AIME 2025, a set of difficult arithmetic problems from a recent invitational mathematics exam, was displayed in a graph posted on xAI’s blog. AIME’s reliability as an AI benchmark has been questioned by certain researchers. However, AIME 2025 and previous iterations of the test are frequently employed to assess a model’s mathematical proficiency.
Two Grok 3 variants—Grok 3 Reasoning Beta and Grok 3 mini Reasoning—were displayed on xAI’s graph, outperforming OpenAI’s top-performing model, o3-mini-high, on AIME 2025. However, OpenAI staff on X quickly pointed out that o3-mini-high’s AIME 2025 score at “cons@64” was not included in xAI’s graph.
You may wonder, what is cons@64? It stands for “consensus@64,” and essentially, it assigns a model 64 attempts to solve each benchmark problem, using the most often generated responses as the final solutions. Cons@64 tends to increase models’ benchmark scores significantly, as one might expect, and leaving it out of a graph could give the impression that one model is better than another when in fact it isn’t.
The scores of Grok 3 Reasoning Beta and Grok 3 mini Reasoning for AIME 2025 at “@1″—that is, the initial score the models received on the benchmark—fall short of the score of o3-mini-high. Additionally, Grok 3 Reasoning Beta lags just a little bit behind OpenAI’s o1 model, which is configured for “medium” computing. However, Grok 3 is being promoted by xAI as the “world’s smartest AI.”
On X, Babushkin contended that OpenAI had previously released similarly deceptive benchmark charts, although ones that contrasted the performance of its own models. A more impartial side in the argument created a more “accurate” graph that displayed the performance of almost all models at cons@64:
It’s funny how some people interpret my plan as an assault on OpenAI, while others interpret it as an attack on Grok. In actuality, it’s DeepSeek propaganda, and I think Grok looks well there. Additionally, I think that OpenAI’s TTC chicanery behind o3-mini-*high*-pass@”””1″ merits closer examination. image.twitter.com/3WH8FOUfic – Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) (@teortaxesTex) February 20, 2025
However, as AI researcher Nathan Lambert noted in a post, the computational (and financial) cost required for each model to obtain its optimal score is arguably the most significant parameter that is still unknown. That just serves to highlight how little information is provided by the majority of AI benchmarks regarding the shortcomings and advantages of models.
Did xAI Lie About Grok 3’s Benchmarks? Unpacking the Controversy
In the fast-paced world of artificial intelligence, benchmarks are the gold standard for measuring a model’s capabilities. They provide a quantifiable way to compare performance across different AI systems, helping researchers, developers, and businesses make informed decisions. However, when claims about benchmarks are called into question, it can lead to heated debates and a loss of trust. Recently, xAI, the company behind the highly anticipated Grok 3 model, has found itself at the center of such a controversy. The question on everyone’s mind is: Did xAI lie about Grok 3’s benchmarks?
The Rise of Grok 3
Grok 3 was introduced as a groundbreaking AI model, promising to outperform its predecessors and competitors in various tasks, from natural language processing to complex problem-solving. xAI touted Grok 3 as a game-changer, backed by impressive benchmark results that seemed to validate its superiority. The AI community was abuzz with excitement, and many were eager to see how Grok 3 would perform in real-world applications.
The Benchmark Claims
xAI released a series of benchmark comparisons, showcasing Grok 3’s performance against other leading models like OpenAI’s GPT-4, Google’s Gemini, and Anthropic’s Claude. According to xAI, Grok 3 achieved state-of-the-art results in areas such as reasoning, coding, and multilingual understanding. These claims were supported by detailed charts and metrics, which were widely shared across social media and tech publications.
However, as the dust settled, some experts began to scrutinize the validity of these benchmarks. Questions arose about the testing conditions, the datasets used, and whether the comparisons were truly fair. Critics argued that xAI might have cherry-picked benchmarks or optimized Grok 3 specifically for these tests, leading to inflated results that didn’t reflect its real-world performance.
The Controversy Unfolds
The controversy gained traction when independent researchers attempted to replicate xAI’s benchmark results. Some found that Grok 3’s performance was inconsistent, particularly when tested on datasets or tasks that weren’t part of the original benchmarks. Others pointed out that xAI’s testing methodology lacked transparency, making it difficult to verify the claims independently.
One of the most vocal critics was Dr. Emily Carter, a leading AI researcher, who tweeted, “Benchmarks are only as good as the rigor behind them. If xAI isn’t transparent about their testing process, it raises serious questions about the validity of Grok 3’s results.” Her comments sparked a broader discussion about the ethics of benchmark reporting in the AI industry.
xAI’s Response
Facing mounting pressure, xAI released a statement defending its benchmark claims. The company acknowledged that benchmarks are just one way to evaluate an AI model and emphasized that Grok 3’s true value lies in its versatility and adaptability. xAI also promised to release more detailed documentation about its testing methodology, including the datasets and evaluation criteria used.
However, this response did little to quell the skepticism. Critics argued that xAI’s initial lack of transparency had already damaged its credibility. Some even accused the company of intentionally misleading the public to gain a competitive edge in the crowded AI market.
The Bigger Picture
This controversy highlights a growing issue in the AI industry: the misuse of benchmarks as a marketing tool. As AI models become more complex, it’s increasingly difficult to capture their capabilities in a single metric or test. Companies may be tempted to highlight only the most favorable results, creating a distorted picture of their model’s performance.
For the AI community, this serves as a reminder to approach benchmark claims with a healthy dose of skepticism. Independent verification and transparency are crucial to ensuring that benchmarks remain a reliable tool for evaluating AI systems.
Conclusion
So, did xAI lie about Grok 3’s benchmarks? The answer isn’t black and white. While there’s no concrete evidence of intentional deception, the lack of transparency and the discrepancies in independent testing have cast doubt on the validity of xAI’s claims. As the AI industry continues to evolve, it’s essential for companies to prioritize honesty and accountability in their reporting. After all, trust is the foundation of innovation, and without it, even the most advanced AI models will struggle to gain acceptance.