🔍 Read the full analysis: FrontierCode: Can Maintainers Work With AI-Written Contributions? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, which grades coding changes against checks designed to reflect whether maintainers would accept them. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts; the results show that passing tests alone does not establish that a patch is ready to merge.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that evaluates whether a repository change meets maintainer-style acceptance criteria rather than merely passing tests. Claude Opus 5.5 ranked first with a 52.2% score, but cleared every blocking check in just 57.7% of attempts, indicating that even the top result does not amount to consistently merge-ready work.
FrontierCode’s results were added to Artificial Analysis’s evaluations on October 12, 2026. The organization ran each model using the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and up to three attempts per task. The benchmark uses a 100-task Main set drawn from a larger collection of 150 tasks.
GPT-6.1 Sol scored 49.1%, placing second, and Claude Fable 5.1 scored 49.0%, close behind. Xiaomi’s open-weights MiMo-V2.6-Pro earned 45.4%; DeepSeek V4.1 Flash scored 44.0%, and Kimi K3 scored 43.6%. Nine of the 100 tasks were not solved by any model on any attempt, according to the source report.
The benchmark distinguishes its overall score from its blocking-check pass rate. Claude Opus 5.5 passed all blocking checks in 57.7% of attempts; GPT-6.1 Sol did so in 54.7%. An attempt that fails any blocking check receives a zero, while one that passes all blocking checks receives a score from 70% to 100%, with quality checks affecting the final result.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Maintainer-Style Review Changes the Measure
Many coding evaluations focus on whether generated code passes tests. FrontierCode adds criteria aimed at problems that can make a patch difficult or risky to review: whether tests catch regressions, whether changes stay within an appropriate scope, and whether the implementation satisfies review standards set for the task. A green test suite is not the same as an acceptable pull request, and this benchmark is designed to measure that gap.
The results matter to engineering teams weighing how much repository work to delegate to AI agents. A leading blocking-check pass rate of 57.7% still leaves substantial review work; the benchmark does not show that model-generated contributions can be merged without human oversight. It is also a comparative test under specific benchmark conditions, not a direct measurement of performance across every company’s codebase or review process.
Artificial Analysis’s cost figures add a practical trade-off. It reports $0.81 per task for GPT-6.1 Sol and $10.15 for Claude Opus 5.5. Opus scored 3.1 percentage points higher, at more than 12 times the listed cost. Those figures can inform model selection, but they do not capture the full cost of human review, integration or correcting failed attempts.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Tests Repository Changes
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026; version 1.1 followed in July with rules against searching online for solutions. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source report says maintainers spent more than 40 hours on each task. Tasks are private, a design intended to reduce the chance that models encountered their solutions in training.
Its checks cover hidden tests, command-based checks for regressions and cleanliness, and reverse tests that check whether an agent’s own tests fail on the original code. It also uses adaptive tests for complex behavior, limits on files and diff size, and an LLM review against maintainers’ criteria. GPT-6 Luna (medium) serves as the review judge in Artificial Analysis’s runs. The benchmark also assigns a zero to attempts flagged by Cognition’s trajectory scanner for looking up a solution.
The rankings do not simply mirror general model evaluations, according to the source report’s comparison with Artificial Analysis’s Intelligence Index. Claude Sonnet 5.5 ranked second among the models in that index comparison but fifth on FrontierCode, while DeepSeek V4.1 Flash moved from a lower general score to seventh on the coding benchmark. That difference suggests general rankings may not predict maintainer-style coding results; it does not establish how models will perform on every software task.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, Celery maintainer, in Cognition’s announcement, as quoted by ThorstenMeyerAI.com
As an affiliate, we earn on qualifying purchases.
Limits of the Current Model Results
The leaderboard reflects one benchmark setup: a selected 100-task set, the Stirrup harness, a 250-turn cap, up to three attempts and a particular LLM review step. It does not establish real-world acceptance rates across different repositories, teams or human review standards. The source material also does not provide enough detail to determine how the reported task costs would change with different agent configurations or usage patterns.
Several models had no FrontierCode result as of October 13, 2026, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published ranking is not a complete comparison of all models discussed in the source report. The report also does not establish whether the measured results will hold as models or benchmark tasks change.
Finally, the 57.7% figure is a pass rate for blocking checks, not a claim that 57.7% of contributions would be accepted by human maintainers. FrontierCode’s score combines benchmark-specific rules and an LLM review; how closely that outcome tracks actual maintainer decisions remains unclear.
As an affiliate, we earn on qualifying purchases.
Missing Models and Further Comparisons
The immediate next point to watch is whether the models without results are added to the leaderboard. The source report specifically identifies Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as missing on October 13; it does not give dates for their evaluation.
Further comparisons could clarify whether the ranking changes with additional models, benchmark updates or different evaluation conditions. For teams applying the results, the practical next step is to test candidate models on their own repositories and retain human review, rather than treating a benchmark score as proof that a change is safe to merge.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether an AI agent’s change to a software repository meets a set of maintainer-style acceptance criteria. Those include correctness and regression checks, test quality, change scope and an LLM review against task-specific criteria.
Which model ranked first?
Claude Opus 5.5, in the max-with-fallback configuration tested, led with a 52.2% FrontierCode score. Its blocking-check pass rate was 57.7%.
Does a 52.2% score mean that model’s changes are accepted 52.2% of the time?
Not necessarily. The score is produced under FrontierCode’s benchmark rules and includes quality grading after blocking checks. It is not a measured rate of human maintainer acceptance across real projects.
Can teams use the results to merge AI-written code without review?
The results do not support that conclusion. Even the leading model failed at least one blocking check in many attempts, and the benchmark does not replace project-specific testing or human review.
Why are some models missing from the results?
The source report says Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of October 13, 2026. It does not explain why they were absent or when they might be evaluated.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
