🔍 Read the full analysis: Using AI To Explore Whether A Research Idea Has Been Studied on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers tested six large language models on whether scientific ideas were novel and found that results could change sharply with evaluation choices. In one pairwise test, a small prompt wording change shifted accuracy by more than 50 percentage points; retrieval and extra reasoning did not consistently improve judgments. The team released a benchmark, data, code and an evaluation pipeline, while warning that some benchmark labels are limited.
Ai2 researchers report that six large language models gave sharply different assessments of whether scientific ideas were novel when researchers changed evaluation choices. In one pairwise test, changing a few prompt words shifted accuracy by more than 50 percentage points, raising questions about whether such systems can reliably screen research proposals before teams invest in literature searches or experiments, as detailed in the original analysis. Ai2 released a benchmark, data, code and a pipeline alongside the study.
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The researchers used both absolute ratings of an idea’s novelty and pairwise comparisons between ideas. They report that relatively small changes in the evaluation setup could alter the models’ judgments, and that adding literature retrieval or allowing more reasoning did not consistently improve results.
For one benchmark, the team drew on ICLR 2026 peer reviews, selecting ideas from accepted papers in the top-rated 10% of their research areas when reviewers praised originality and no reviewer disputed it. This high-novelty group contained 154 ideas. The lower-novelty group contained 145 ideas from rejected papers in the bottom 10%, selected when reviewers raised originality concerns and none praised novelty. The authors caution that acceptance and rejection alone are not definitive measures of originality.
A second test paired the 154 human-authored ideas with 154 proposals from a basic language-model generator. The researchers treated generated ideas as less novel, but describe these labels as weak rather than individually verified. In a separate expert check of 30 pairs—selected heavily from cases the strongest model judge got wrong—the expert preferred the human idea in every pair and judged none of the generated ideas novel. An annotator also agreed with the direction of extracted reviewer novelty signals in 47 of 50 checked cases; the authors say the three disagreements did not change review classifications.
Why Unstable Scores Affect Research Screening
Research groups may use language models to filter proposed ideas before commissioning a detailed literature review, asking experts to assess them or beginning experiments. If a verdict changes substantially after a small prompt edit, the score may reflect how the test was configured as much as the idea’s originality. That would make it harder to compare proposals across projects, research groups or studies.
The result also bears on efforts to automate parts of scientific discovery. Systems that generate hypotheses or summarize literature need credible ways to assess whether a proposal adds something new. The study does not establish that all model-based novelty evaluations fail, or that language models cannot assist researchers. It does indicate that performance on one chosen prompt or benchmark is not enough to show dependable judgment. The authors also report that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline, a finding that highlights the need to test systems against clear and repeatable comparisons.
Products Worth Considering
Logitech MX Keys S Wireless Keyboard, Graphite
Jackery Explorer 300 Portable Power Station, 292Wh
LG UltraGear 32GX850A-B 32-Inch 4K UHD Glossy OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Ai2 Tested Novelty Judges
A novelty judge is a model asked to decide whether a research idea adds something new relative to earlier work. It may assess a proposal on its own or compare it with another. The study authors say there is no standard recipe for these evaluations: teams can vary the model, prompt, comparison format, reasoning budget and access to retrieved literature. Those choices can affect both how a judge is used and how its performance is measured.
The authors argue that evaluation sets often rely on human-written papers even when a judge is intended to assess ideas generated by AI. Their benchmark therefore includes a setting with model-generated proposals as well as one using reviewer comments as evidence about human-authored ideas. They say some conclusions about disagreement between evaluation setups do not depend on benchmark labels, although the supplied description does not detail how broadly that finding holds across all tests. The project is part of Ai2’s wider work on tools for scientific discovery, including systems for exploring data, proposing and testing hypotheses, and synthesizing research literature.
““Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Novelty Labels
The benchmark uses reviewer comments and paper selection criteria as signals of relative novelty, not as a definitive ground truth. Reviewers can disagree, and papers may be accepted or rejected for reasons beyond originality. The second benchmark’s model-generated ideas were treated as less novel without individually verifying every label. The expert review covered just 30 pairs, so it cannot establish how the full set would be judged.
The supplied study description does not provide full results for every model, prompt, reasoning budget or evaluation format, and it does not state the release’s calendar date. It also leaves open whether the pattern applies across other disciplines, idea-generation systems and judges with broader literature access. The findings show sensitivity in the tested setups, but do not identify a universally reliable evaluation method or establish that retrieval and additional reasoning never help.
Further Tests Across Research Fields
Ai2 has released the paper, benchmark data, code and an automated pipeline intended to help other researchers build evaluation sets. That gives teams materials to reproduce the tests, examine how prompt and model choices affect results, and assess whether their own novelty judges are consistent.
The next step is independent testing across disciplines, idea sources and evaluation designs, alongside work on methods that could make judgments more stable. The release description does not announce a particular follow-up study or timetable. Until further evaluations are available, the study offers evidence of instability in the tested settings, while leaving open which combination of models, prompts and literature access—if any—can support reliable novelty screening.
Key Questions
What did the Ai2 study find?
It found that six tested language models’ novelty judgments varied with evaluation choices. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points.
Did literature retrieval or extra reasoning fix the problem?
No consistent improvement was reported. The authors say retrieval and additional reasoning did not reliably improve judgments across the tested evaluations; that does not show they never help in other settings.
How were ideas labeled as more or less novel?
One test used peer-review comments and paper ratings to select human-authored ideas with positive or negative originality signals. A second compared human ideas with basic model-generated proposals, which the researchers treated as less novel while acknowledging that those labels were weak and not individually verified.
Can researchers use the benchmark?
Ai2 released the paper, data, code and an automated pipeline for building further evaluation sets. The materials can support replication and comparisons, but the study does not establish a single dependable novelty-scoring method.
Primary source: Hugging Face · via ThorstenMeyerAI.com
