🔍 Read the full analysis: What Score Did The Law Model Get On Our AI Legal Test? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
GLM-5.3 FlashX Legal scored 8 out of 100 on Lexassay, a 15-task legal test run by Thorsten Meyer AI, and scored zero on its five hidden tasks. The publisher says the test is a spot check, not a definitive benchmark, and discloses that Claude agents graded the answers while a Claude model ranked first.
GLM-5.3 FlashX Legal, a model its maker Z.ai describes as optimized for legal workflows, scored 8 out of 100 on a 15-task test run by Thorsten Meyer AI. It received no points on the five hidden tasks, according to the test results published November 4, 2026; the result is a limited spot check, not a broad measure of performance in real legal work.
The test, called Lexassay, asks models questions involving German, U.S. and EU law. Its tasks include calculating legal deadlines, identifying changes in the law, recalling court decisions and recognizing questions based on a false premise. Models answer without internet access or tools, and each response is graded against a prepared answer with weighted criteria. The publisher says citations are checked against official databases.
For GLM-5.3 FlashX Legal, the citation review identified 21 real citations, 25 misattributed citations and one invented citation. Lexassay deducts points for invented or misattributed citations. The model also returned no answer on two German-law tasks after using its full stated reasoning budget of 64,000 tokens, according to the publisher. Its base model scored 3 out of 100 on the same test.
Other results in the report put Claude Opus 5.5 at 79, GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Mistral Large 4 scored 10. Meyer reports that the top model fully met every criterion on eight of the ten public tasks, while saying that greater reasoning effort did not improve results. These scores reflect the test’s own grading rules and task set.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal AI Scores and Citation Risks
The result highlights a practical risk for people using chatbots for legal questions: an answer can sound confident while giving an inaccurate deadline, misstating a court ruling or citing authority that does not support the claim. In legal work, such errors can affect filings, claims and decisions about whether to seek professional advice.
Lexassay’s example involves two German civil appeals served on the same day. Under the scenario described, the appeal deadline differs between Munich and Berlin because Corpus Christi is a public holiday in Bavaria but not Berlin. The publisher says most strong models answered correctly, while one gave Berlin an extra day and another placed the deadline for written grounds a month too late. The example illustrates why jurisdiction and calendar details matter; it is not advice for an actual case.
The publisher also reports that, across all published runs, it verified 2,644 citations as real and identified 804 as invented or misattributed. Those figures are counts from Lexassay’s runs, not rates for all legal AI use. The source advises readers to check cited law and rulings and not rely on a chatbot for deadlines.
Products Worth Considering
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
Bose QuietComfort Ultra Headphones (2nd Gen), Wireless Bluetooth Noise-Cancelling Over-Ear Headphones, Black
Apple MacBook Air 15-inch with M5 chip, 16GB Memory, 512GB SSD, and Liquid Retina Display
As an affiliate, we earn on qualifying purchases.
How Lexassay Tests Legal Models
Lexassay uses 15 tasks per model: ten public questions and five hidden ones. According to its method description, hidden-task scores are published without the questions and answers, to make it harder for later models to learn them directly from the test. Tasks cover German, U.S. and EU law, and include both legal knowledge and reasoning about mistaken assumptions.
The test grades one answer per model per task, with three to six weighted criteria in each prepared answer. It checks citations and applies a deduction when an answer lacks a legal-advice disclaimer or a suggestion to consult a lawyer. Because there is only one answer for each task and model, the publisher cautions that small score differences may be noise. The results should not be treated as a definitive ranking of how models perform across legal practice.
The report includes a disclosure relevant to interpreting the scores: Thorsten Meyer AI runs Lexassay, and answers were graded blind by several Claude agents made by Anthropic. A Claude model ranked first. Meyer says the test’s method and public answers, gold answers and citation checks are available for review. Blind grading describes how answers were assessed, but it does not remove the publisher’s disclosed conflict of interest.
“It is a spot check, not a benchmark in the strict sense.”
— Thorsten Meyer AI, describing the scope of Lexassay
What the Scores Cannot Establish
The results do not establish how these models would perform with legal documents, web search, specialist tools or review by a lawyer. Lexassay tests answers produced without internet access or tools, while other legal benchmarks use different methods. The source notes that on Vals AI’s legal-agent benchmark, which supplies documents, Mistral Large 4 ranked eighth and ahead of Claude Opus 5.5 and GPT-6 Astra. The differing results show that rankings depend on the test design; they do not by themselves establish which model is more useful overall.
The report does not provide repeated runs for each task, so it cannot show how consistent a model’s score would be across multiple attempts. Its claims about citation errors and task performance are the publisher’s findings under its stated grading system. The disclosed grading relationship with Anthropic is also relevant: Claude agents graded the answers, and a Claude model placed first. The source says grading was blind and the checks are public, but independent replication is not described.
Details about performance in particular legal settings, the effect of tools or human review, and whether results generalize beyond these 15 questions remain unclear. The source’s recommendation is not to treat a chatbot answer as a substitute for legal advice, especially where a deadline or legal claim is at stake.
Further Checks and Legal Review
The publisher says readers can review the test method, public answers, gold answers and citation checks to examine how scores were assigned. The report does not announce a date for a new test or independent replication, so no further release is confirmed. Comparisons with other benchmarks may provide additional evidence, but their results need to be read in light of differences in tasks and tools.
For anyone facing a real legal deadline, contract or letter from an authority, the immediate next step is not to rely on a model’s score or answer. Verify the relevant rule and cited decision in authoritative sources, and consult a lawyer for advice on the specific matter. Lexassay’s results describe a controlled test; they do not determine the correct answer in an individual case.
Key Questions
What score did GLM-5.3 FlashX Legal receive?
It scored 8 out of 100 on Lexassay’s 15-task test and scored zero on the five hidden tasks, according to the publisher.
How did the test check legal citations?
Lexassay says it compared citations with official databases and classified them as real, misattributed, invented or not findable. Its scoring rules deduct points for invented or misattributed citations.
Was Claude involved in grading the test?
Yes. Thorsten Meyer AI says several Claude agents graded answers blind, and a Claude model ranked first. The publisher also discloses that it runs Lexassay, a conflict readers should consider when assessing the results.
Does the score prove which AI is best for legal work?
No. The publisher calls Lexassay a spot check: models answered each task once, without internet access or tools. Other benchmarks use different tasks and conditions, and the source reports different rankings on a document-based legal benchmark.
Should I use a chatbot to calculate a legal deadline?
The report advises against relying on a chatbot for deadlines. Verify the applicable rule and calendar details, and ask a lawyer about a real case, filing or legal obligation.
Source: ThorstenMeyerAI.com
