My News Pipeline’s AI Test Winner Didn’t Make It Into Production

🔍 Read the full analysis: My News Pipeline’s AI Test Winner Didn’t Make It Into Production on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he chose GPT-6 Luna for his news-writing pipeline rather than the top-scoring model, GPT-6 Astra, after a test of 71 configurations found Astra cost about 100 times more per article. In his October 13 report, Meyer said Luna’s quality was sufficient for routine publication, while testing also led him to remove two backup models after they produced fabricated quotations or output his pipeline could not use.

Meyer tested models from September 8 to October 13, 2026, using the production prompt and real news sources. Each configuration wrote in English and German. The initial evaluation included a long source of 6,340 characters and a short source of 619 characters; a later robustness run tested the strongest configurations on 10 additional articles, mostly based on short sources.

Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, assessed source fidelity, fabrication, clarity about uncertainty, news value, structure and polish. Meyer said a result counted only if the article produced valid output after the same repair step used in production. He also cautioned that, with only two articles in the initial test, score gaps below roughly 0.3 points could be noise.

GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test, with source-fidelity scores of 5.00. In the robustness run, it averaged 3.93 across 12 articles, with a reported 95% bootstrap interval of 3.83 to 4.02. GPT-6 Luna averaged 3.67 in the robustness run, with source fidelity of 4.71; Meyer said all of its articles were valid. He reported Astra at about $0.12 per article, compared with about $0.0012 for Luna, and average generation times of roughly 50 seconds and 17 seconds, respectively.

Cost Changed the Production Choice

The result highlights a practical distinction between best-scoring and best suited to production. For a high-volume publisher, an improvement in judged quality may not justify a steep rise in per-article cost, particularly if the less expensive model consistently produces valid copy that clears the publisher’s quality threshold.

Meyer’s report also points to a risk that aggregate rankings can obscure: fallback models may be called on when a primary service fails, and weak output in those cases can still reach the publishing workflow. In this test, the backup models raised concerns not because of a small score difference but because of fabricated attributions and invalid formatting. The account is a single publisher’s evaluation, however, and does not establish how the models perform for other prompts, sources or editorial standards.

A Benchmark Built Around Publishing

Meyer’s stated reason for running the test was that general leaderboards do not directly answer which model can write the articles his sites publish. He evaluated API, subscription, free-tier and open-weight models on a specific newsroom task, rather than treating broad intelligence or coding results as a substitute for writing performance.

The source mix was intended to resemble the pipeline, where Meyer said many items begin with short source material. He therefore treated the robustness run, which used mostly thin sources, as more reliable for ranking than the two-article initial comparison. His test also included a production-validity rule, making formatting and repair behavior part of the assessment rather than scoring prose alone.

The comparison included models that Meyer uses for different purposes. He reported Claude Opus 5.5 at 3.46 in English and 3.39 in German in the initial writing test, despite describing it as his software-development model and noting its coding benchmark reputation. He said the finding reinforced that performance on one task does not automatically transfer to another.

“The cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of This Model Test

The reported scores reflect one publisher’s prompt, sources and judging setup; the test does not show how the models would perform across other newsrooms or editorial workflows. Meyer identified only two articles in the initial test and said small score gaps there could be noise, though he considered the additional 10-article robustness run a more dependable ranking. The 95% intervals he supplied apply to the robustness results, not necessarily to every comparison in the initial test.

The report also relies on AI judges, including Claude Opus 5.5, which was itself among the models evaluated. Meyer did not provide human-editor scores in the supplied account. It is therefore unclear how the rankings compare with newsroom editors’ judgments, or how the systems would perform on a broader range of developing, sensitive or highly technical stories. The test gives no basis for treating the results as a general ranking of model quality.

Luna Remains the Production Model

Meyer said his production pipeline now uses GPT-6 Luna and that the two backup models were removed from the news route on October 7. He reported a recent production cost of about $0.0013 per article and a median generation time of 17 seconds. The supplied report does not name a replacement fallback, so it is unclear how the route handles a Luna failure now.

The next useful evidence would be continued results from routine production, including article validity, source fidelity, editorial corrections and costs over a larger set of stories. Meyer’s account does not announce a scheduled follow-up test or a date for changing models; for now, his reported decision is to retain Luna rather than pay Astra’s substantially higher price.

Key Questions

Why did GPT-6 Astra not go into production?

Meyer reported that Astra scored highest in his test but cost about $0.12 per article, roughly 100 times Luna’s reported $0.0012 test cost. He judged Luna’s lower-cost performance sufficient for his publishing workflow.

Which model is Meyer using for news articles?

Meyer said GPT-6 Luna is the production model. He reported that its articles were all valid in the robustness run and that production costs over the preceding week were about $0.0013 per article.

What went wrong with the backup models?

Meyer said GPT-4.1 nano produced invented quotations and scored 1.00 for source fidelity, while locally run gpt-oss-120b produced no valid articles in four attempts and also generated invented named quotes. He said both were removed from the news route on October 7.

Does the test prove Luna is better than Astra?

No. In Meyer’s evaluation, Astra scored higher, while Luna was much less expensive and selected for production. The test measured one specific publishing task and does not establish a universal ranking or show which model would suit another newsroom.

Source: ThorstenMeyerAI.com