Every Benchmark Launched 2023-2024 Has Fallen — The METR / SWE-Bench / CORE-Bench / MLE-Bench / PostTrainBench Sequence

📊 Full opportunity report: Every Benchmark Launched 2023-2024 Has Fallen — The METR / SWE-Bench / CORE-Bench / MLE-Bench / PostTrainBench Sequence on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

All six major benchmarks designed to measure AI research and development capabilities, launched between 2023 and 2024, have either saturated or are nearing saturation within months, not years, according to recent analyses.

Research by Thorsten Meyer and Jack Clark reveals that six key benchmarks—covering software engineering, model training, research reproduction, and fine-tuning—launched during 2023-2024, have all reached or are approaching saturation. For example, the SWE-Bench, which measures software engineering skills, improved from 2% to 93.9% in 30 months, achieving saturation. Similarly, the METR time horizon benchmark, measuring task durations, expanded from 30 seconds to 12 hours over four years, a 1,440-fold increase, and is nearing a plateau.

Other benchmarks, such as CORE-Bench for research reproduction, declared themselves ‘solved’ by authors in late 2025 after reaching 95.5%, while MLE-Bench for ML engineering is tracking toward saturation with a 3.8× improvement over 16 months. The pattern across all six benchmarks is consistent: rapid progress leading to saturation within a short timeframe, indicating that AI capabilities are advancing faster than many models predicted.

Implications of Benchmark Saturation for AI Progress Predictions

The saturation of these benchmarks suggests that AI systems are rapidly closing the gap on tasks previously considered challenging or requiring human-level expertise. This pattern confirms that AI capability growth is accelerating, which could influence policy, investment, and workforce planning. It also questions the reliability of models that assume slower, linear progress, emphasizing the need for updated forecasts and strategic responses to AI’s swift evolution.

Background on AI Benchmark Development and Expectations

Throughout 2023 and 2024, multiple AI benchmarks were launched to measure progress across different facets of AI research, including software engineering, model training, and research reproduction. These benchmarks were explicitly designed to be challenging, with initial performance levels often near human or baseline levels. Prior to these developments, projections of AI progress varied widely, with some experts estimating slow, incremental improvements, while others anticipated rapid breakthroughs. The recent saturation across all six benchmarks indicates a shift toward faster, more comprehensive capability development, aligning with some of the more optimistic forecasts.

“The pattern across these six benchmarks is not noise; it’s a curve indicating rapid saturation and capability growth.”

— Thorsten Meyer

Uncertainties in Long-Term AI Capability Trajectories

While the benchmarks indicate rapid saturation, it remains unclear how these results translate into broader, real-world AI applications beyond the tested tasks. It is also uncertain whether future benchmarks will continue to saturate at this pace or if new, more complex challenges will emerge that slow progress. Additionally, some experts caution that saturation in benchmarks may reflect overfitting or measurement noise rather than genuine, sustained capability improvements.

Next Steps in Monitoring AI Benchmark Trends

Researchers and industry analysts will closely monitor upcoming benchmark releases and real-world AI deployment to verify whether saturation continues across new tasks. Further investigations are expected to explore whether these rapid gains translate into robust, generalizable AI capabilities or if they plateau at specific tasks. Policy discussions and investment strategies are likely to be influenced by these developments, emphasizing the need for ongoing assessment of AI progress and potential regulation.

Key Questions

What do benchmark saturation results mean for AI safety?

Saturation indicates rapid capability growth, which could raise concerns about AI safety, control, and alignment as systems become more powerful and potentially unpredictable. Ongoing research is needed to understand these implications fully.

Are these benchmarks representative of real-world AI performance?

While they are designed to challenge AI systems, benchmarks may not fully reflect the complexity of real-world tasks. Saturation in benchmarks suggests capability improvements but does not guarantee equivalent performance in practical applications.

Will future benchmarks continue to saturate at this pace?

It is uncertain. The current pattern shows rapid saturation, but the emergence of more complex, diverse tasks could slow progress. Researchers are watching for new benchmarks that test broader capabilities.

How might this affect AI development timelines?

If the saturation trend continues, AI development could accelerate, reaching advanced capabilities sooner than previously expected. This may impact policy, workforce planning, and investment strategies.

What are the limitations of current benchmark assessments?

Benchmarks may be susceptible to overfitting, measurement noise, or gaming, which could inflate apparent progress. It remains essential to interpret saturation results within broader performance and safety contexts.

Source: ThorstenMeyerAI.com