🔍 Read the full analysis: Claude’s Model Edit Passed Tests, Then GPT-6 Astra Found Bugs on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
GPT-6 Astra, running in Codex, reviewed a Claude Opus 5.5-led model-editing experiment and found faults that green unit tests had missed. After fixes, all 12 author tests passed, but four of the reviewer’s six provenance checks still failed; the edited model answered 10 published tasks, while its legal score remained zero.
A review by OpenAI’s GPT-6 Astra, running in Codex, found reproducible faults in the test rig for a model-editing experiment led by Anthropic’s Claude Opus 5.5, including a resume function that could reuse old results and a weight conversion that altered parameters even when no edit was requested. The findings matter because the experiment reported improved task completion on a small benchmark, while the review showed that some checks and claims needed correction before those results could be interpreted reliably.
The experiment used Claude Opus 5.5 to apply a directional change to the weights of the local model Ornith 1.5, without training. In the source’s updated account, the edited model answered all 10 published Lexassay tasks, compared with four for the unedited model; its legal score remained zero. The source says the benchmark runs are public at Lexassay. These results describe task completion in that test set, not demonstrated legal competence.
GPT-6 Astra was assigned to review and validate the implementation rather than run another model-improvement experiment. According to the account, it did not inspect the hidden benchmark tasks and left the main implementation unchanged; Claude’s session made the resulting fixes. In its first review on November 4, Astra reproduced four problems: resuming a run could accept answers from a different configuration, an unknown task ID could yield a successful run with no evaluations, the real command line failed despite green parser unit tests, and re-quantizing weights changed them even at edit strength zero.
The source says the team changed the command syntax, made the rig reject unknown task IDs and stored edited tensors in higher-precision bf16 rather than re-quantizing them. It reports that all 12 author tests passed in the second review round, while four of Astra’s six provenance checks failed. The review also measured a separate weight-format conversion that moved a matrix by 0.51% and changed 915,360 of 1,048,576 values. That figure concerns the conversion measurement, not legal quality or an error rate.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Independent Checks Changed the Results
The review illustrates how a passing test suite can coexist with failures in the workflow researchers actually use. A test rig that silently reuses answers or accepts a nonexistent task can make results appear complete when they are not. The reproduced command-line failure also shows why testing only a parser’s individual components may miss errors in the real entry point.
Weight-format changes matter for a different reason: if a conversion changes parameters when the requested edit strength is zero, observed differences cannot automatically be attributed to the intended edit alone. The review does not establish that conversion caused the benchmark result. It identifies a source of possible confounding that the experimenters addressed, and it narrows what the reported improvement can support.
The division of labor also offers a practical review model: one system runs the experiment, while another checks its implementation without accessing hidden test material. That separation can help surface defects, but it does not make the review infallible. The source’s own report records failed provenance checks and limitations in Astra’s final documentation pass.
Products Worth Considering
DJI Osmo Action 4 Essential Combo Waterproof 4K Action Camera
Logitech MX Ergo S Advanced Wireless Trackball Mouse USB-C – Graphite
Samsung Galaxy Tab A11+ 11-Inch Tablet, 6GB RAM, 128GB Storage, Expandable, Gray
As an affiliate, we earn on qualifying purchases.
How the Model Edit Was Reviewed
The account says two AI sessions began work on the experiment on November 4, 2026. After the team recognized that they were working on the same project, it assigned the experiment to Claude and the review to GPT-6 Astra. Claude handled the plan, edit, test rig, benchmark runs and blind grading by three Claude agents. Astra wrote separate checking programs and reported findings; the Claude session applied fixes.
The experiment drew on a directional-ablation idea described in an MIT CSAIL post on weight-space programming. The source’s first article covers the model edit itself. This report focuses on the review, which tested actual entry points with reproducible examples rather than relying only on unit-test results. The source says Astra did not read hidden prompts, topics, answers or reference solutions.
The published figures were updated on November 9 to refer to the 10 published Lexassay tasks. The source links public runs for the edited model, an edit control and model-card sampling. Its claims and measurements should be read within that scope: a small set of benchmark tasks and a technical audit, not an evaluation of legal advice in practice.
Limits of the Review and Benchmark
The source does not provide a full account of what caused all four provenance checks to fail in the second review round, or whether each issue affected the published benchmark results. It also says Astra did not rerun every test during its final documentation step, was not a fourth grader, and measured weight conversion on a CPU rather than the production GPU path. Those limits make it inappropriate to treat the review as a complete independent replication.
The benchmark result is also bounded: the updated account covers 10 published tasks, and the legal score remained zero. The source does not establish performance on a broader legal benchmark or in real-world legal work. The matrix conversion figures are measurements of a format conversion, not evidence of a particular effect on task quality. The account further says one control was not enough to support the strongest causal wording previously used.
The source is a first-party account from the experiment’s author. It describes public runs and reproducible review checks, but the material supplied here does not include an independent assessment of the results by an outside research group. The review found and helped correct implementation and wording problems; it does not by itself settle the model’s capabilities.
Public Runs and Further Validation
The experiment’s public Lexassay runs allow readers to inspect the reported benchmark results and the linked control and sampling records. The source does not announce a specific date for another review, a broader benchmark, or a new model-editing run. Further validation would need to report the test configuration, controls and provenance checks clearly, and distinguish task completion from legal accuracy.
For now, the reported next step is scrutiny of the public records and the corrected implementation. Whether the weight edit itself accounts for the task-completion difference remains a question for additional controlled testing; the account does not claim that the review established that causal link.
Key Questions
What did GPT-6 Astra find?
It reproduced issues including stale results being reused after a run resumed, unknown task IDs being accepted as empty successful runs, a command-line parsing failure missed by unit tests, and weight changes during conversion even when edit strength was zero.
Did GPT-6 Astra run a competing model-editing experiment?
No. The source says Astra reviewed the implementation and ran checks, while Claude Opus 5.5 led the model edit and benchmark experiment. Astra’s quoted explanation says Codex did not run a competing model-improvement experiment.
What did the experiment report on the published tasks?
The updated source says the edited Ornith 1.5 answered all 10 published Lexassay tasks, compared with four for the unedited model. The legal score remained zero, so the result does not establish legal competence.
Were all review checks passed after the fixes?
No. The source reports that all 12 author tests passed in the second review round, but four of Astra’s six provenance checks failed. It does not fully detail those four failures in the material provided.
What remains unproven?
The account does not establish that the weight edit alone caused the task-completion difference, or that the result generalizes beyond the 10 published tasks. Astra also did not rerun every test in its final documentation step, and its conversion measurement used a CPU rather than the production GPU path.
Source: ThorstenMeyerAI.com
