🔍 Read the full analysis: Home Weight-Space AI Editing: How We Kept Legal Scores At Zero on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment used directional ablation to reduce looping in the local Ornith 1.5 35B-A3B model. On 10 published Lexassay tasks, it completed all 10 after the edit, compared with four on the same setup before editing, but its legal score remained zero and its answers contained 55 wrong or invented citations. The results are limited to one run per setting, and the unchanged model completed nine tasks when run with its recommended sampling settings.
A home experiment reported by ThorstenMeyerAI.com found that editing the weights of a local AI model helped it finish more legal benchmark tasks, but did not improve its legal score: on the 10 published Lexassay tasks, the edited Ornith model answered all 10 and scored zero. The result suggests the edit reduced looping rather than adding legal knowledge, and the experiment’s authors stress that it was a single, limited test, not evidence that weight editing can make a model legally reliable.
The experiment applied directional ablation to Ornith 1.5 35B-A3B, a local mixture-of-experts model. A Claude Opus 5.5 session generated code to run the model, identify a direction associated with looping, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. The process did not train the model, add legal data, or involve an AI reading the weights directly. The source says the experiment ran from Wednesday evening to Thursday morning and was updated Nov. 9 with figures covering the 10 published tasks.
On the same setup, the edited model’s number of completed tasks rose from four of 10 to 10 of 10, while its legal score stayed at zero. The 10 edited answers included 55 wrong or invented citations. In a content check covering 50 criteria, the model met seven, up from three before editing. Those results distinguish the ability to produce a completed response from the ability to give sound legal analysis.
A separate run with the unchanged model and sampling settings recommended by its model card completed nine of 10 tasks without any weight edit. The article also says the unchanged model completed eight of 10 in a published run on a different setup, compared with four of 10 on the setup used for the direct edited-model comparison. The author cautions that those setups are not directly comparable. Runs, answers, grades and citation checks are published on Lexassay, according to the report.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Finishing Tasks Is Not Legal Accuracy
The result illustrates a practical distinction for people evaluating AI systems: completion is not correctness. A model that stops repeating itself may appear more useful, but that change does not establish that its answers are accurate, its sources are real, or its reasoning is legally sound. In this test, the headline improvement in completed tasks came alongside a zero legal score and numerous citation errors.
That matters for anyone considering local AI for legal research or document work. A more fluent or complete response can still contain unsupported claims. The experiment provides no basis for relying on the edited model for legal advice; its own report explicitly says it is not legal advice. More broadly, the result cautions against treating a benchmark improvement in one behavior as proof of an improvement in a model’s underlying expertise.
The report also identifies a simpler comparison worth checking before modifying weights: using the model’s recommended sampling settings raised task completion to nine of 10 without editing. Because there was only one run per setting and the setups differed, that comparison does not establish which approach is generally better. It does show why evaluation conditions and baselines matter when reporting an AI result.
Products Worth Considering
Amazon Kindle Paperwhite (2024, 16 GB) – 7-inch Glare-Free Display, Black
Samsung Galaxy Tab A9+ 11-Inch Android Tablet, 64GB, Graphite
DJI Osmo Action 4 Essential Combo Waterproof 4K Action Camera
As an affiliate, we earn on qualifying purchases.
A Home Test Inspired by MIT
The test followed a reported MIT CSAIL study of what its authors call weight-space programming: an AI model directly examines and changes another network’s weights. The source article says the MIT team’s Qwen example involved a model stuck in loops; after an edit, its reported mathematics score rose from 7.9% to 48.1%. The home experiment was inspired by that work but did not reproduce its method. Instead, it used a known approach in which researchers run a model, measure its internal signals, and remove a selected direction.
Lexassay asks models the same 15 questions covering German, U.S. and EU law, without internet access or tools, and checks citations against primary sources, according to the source. At the time of the report, 10 tasks were published and five were hidden. All figures in this experiment refer to the published subset. Ornith was selected because the author says it showed the looping behavior the edit was intended to address; other tested models either did not loop or could not be run under comparable conditions.
The report disclosed that Claude carried out the edit and Claude agents conducted blind grading. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, figures and claims and identified faults, according to the author. That review was a check on one experiment, not a second independent replication.
“The model finishes is not the answer is right.”
— Thorsten Meyer, describing the result
Limits of the Ten-Task Test
The findings are preliminary. The report describes one run per setting, a small development set, and a loop detector that changed during the experiment. It also notes that three graders came from the same model family. The external review by GPT-6 Astra identified faults, but it does not substitute for repeated testing or independent replication.
The completion results varied with setup: the unchanged model answered four of 10 tasks on the setup used for the edited comparison, eight of 10 in another published run, and nine of 10 with model-card sampling settings. The source explicitly says the two local setups are not directly comparable. It is therefore unclear how much of the difference reflects the edit, sampling choices, other setup differences, or ordinary run-to-run variation.
The report does not establish how the edit would perform on the five hidden tasks, on other legal questions, or with other models. It also does not show that the edited model became more reliable in real-world legal use. The available numbers support a narrow claim about task completion in this test, not a general claim about legal competence.
The immediate next step is to compare repeated runs under the same documented setup, including the recommended sampling settings, and publish enough detail for readers to reproduce the analysis. Testing against Lexassay’s hidden tasks could help show whether the observed behavior carries over beyond the published development set, provided those tasks remain protected from tuning.
Additional checks would need to measure more than whether a model reaches the end of a response. They should track citation validity, legal criteria, and looping under consistent conditions, with reviewers independent of the model family used to perform the edit and grading. Until those results exist, the report’s conclusion remains limited: the edit helped this model finish more published tasks in one comparison, while its legal score stayed at zero.
Key Questions
What changed after the model’s weights were edited?
On the same test setup, Ornith 1.5 35B-A3B completed 10 of 10 published tasks, compared with four of 10 before editing. The legal score remained zero.
Did the edit teach the model law?
No legal data was added and the model was not trained, according to the report. The edit targeted a direction associated with looping; the results do not show that it added legal knowledge.
How reliable were the edited answers?
The report says the edited answers contained 55 wrong or invented citations. The model met seven of 50 checked content criteria, so completing every task did not mean its legal answers were accurate.
Was this the same method used in the MIT study?
No. The home test was inspired by the MIT work but used directional ablation: code measured model behavior and removed a selected direction from some weights. The source says the MIT example involved an AI reasoning directly over another model’s weights.
What are the main limitations of the result?
The experiment had one run per setting, used 10 published tasks, and involved a loop detector that changed during testing. Results also differed across setups, and the report says those setups were not directly comparable.
Source: ThorstenMeyerAI.com
