🔍 Read the full analysis: Why Passing About 1 In 5 Tasks Still Earns Z.ai’s Legal AI A High Rank on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s version of Harvey’s Legal Agent Benchmark, placing seventh among 81 models. The all-or-nothing score means it met every required criterion on roughly one in five tasks, although it passed 92.5% of individual criteria. The result is an independent leaderboard measurement; claims that the model is state of the art among open-source systems and suitable for legal work need qualification.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s version of Harvey’s Legal Agent Benchmark, ranking seventh among 81 models in the archived leaderboard. The score means the model met every required criterion on about one in five tasks—not that it got only one in five individual answers right: Vals also reported a 92.5% criteria pass rate.
The benchmark uses all-pass grading. A task counts as complete only if every criterion in its rubric passes; missing even one required detail makes the task a failure. Harvey’s benchmark instructions describe tasks built around a client matter and a requested work product, evaluated against criteria that can include facts, legal conclusions, citations, deadlines, and formatting. That standard makes the task-level score materially different from a measure of how often individual details are correct.
Vals AI’s archived Version 1 leaderboard lists GLM Legal at 19.17% across a 120-task private holdout, with a reported test cost of $0.15. The leaderboard says two AI judges grade each submission and the task score averages their results. Z.ai’s documentation rounds the result to 19.2% and says it was independently evaluated by Vals. The same archived table gives the model’s base, GLM-5.3-Flash, a 6.67% task pass rate. The difference is 12.50 percentage points, matching Z.ai’s claimed improvement.
Z.ai describes the model as optimized for legal workflows, including research, contract review, and drafting. Its developer documentation says it underwent legal-domain mid-training and post-training, including reinforcement learning in tens of thousands of simulated task environments. The company recommends connecting it to services for laws, cases, and company information. Z.ai’s pricing page lists charges of $0.37 per million input tokens and $1.25 per million output tokens; Vals’s separate reported cost per test is not the same pricing measure.
A Strong Rank With a Strict Caveat
The result shows why a model can rank highly while still failing most complete benchmark tasks. Seventh place is relative to the other evaluated systems; a 19.17% all-pass rate is an absolute measure under a demanding rubric. For legal teams, both numbers matter. The rank offers a comparison with other models on the same test, while the pass rate warns against treating the system as ready to produce unchecked work products.
The gap between 92.5% of criteria passed and 19.17% of tasks completed highlights the consequence of a single omission in legal work. A draft may contain many accurate elements yet miss a deadline, cite the wrong authority, or fail to address one required issue. Under the benchmark’s scoring rule, that task does not pass. The metric does not establish that every failed task is unusable, but it does show that high partial accuracy does not reliably produce fully compliant outputs under this test.
That distinction affects procurement and oversight. Firms evaluating a system will need to check not just benchmark rank, but the quality of citations, the handling of client data, the model’s performance on their own matters, and the human review required before work is used. The leaderboard is evidence about performance on a defined evaluation, not a guarantee of accuracy or a substitute for professional judgment.
Products Worth Considering
DJI Mic 3 Wireless Microphone Kit (2 Transmitters, 1 Receiver, Charging Case)
Apple Watch Series 8 GPS, 41mm, Midnight Aluminum Case with Midnight Sport Band (S/M)
DJI Osmo Mobile 7P Phone Gimbal with Mic Mini Transmitter, Infinity Black
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as an open-source evaluation on May 6, 2026. The public set contains 1,250 tasks across 24 legal practice areas, with more than 75,000 expert-written, binary rubric criteria. Harvey’s method treats a task as complete only when every criterion passes. Its stated rationale is that a work product that omits a required issue may be materially incomplete even if most of its contents are correct.
The public leaderboard uses a private holdout set of 120 tasks, according to Vals, and disables internet access during evaluation. Vals says two language-model judges assess submissions, with the reported score calculated from their task pass rates. Because the holdout is private and the grading depends on automated judges, the result reflects this particular test design and scoring method. It should not be read as a direct prediction of performance across every firm, jurisdiction, or legal task.
Harvey’s earlier baseline, published May 26, reported a top all-pass rate of 7.1% among the frontier models it tested. That was a separate run and is not directly comparable with the Vals result. It does, however, show that end-to-end task completion has been difficult for systems tested under this strict scoring approach.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%.””
— Z.ai, in its model documentation
Limits of the Benchmark Evidence
The leaderboard establishes a score on a particular private test set, but it does not show how GLM Legal performs across real client matters, jurisdictions, or firm workflows. The source material does not provide task-by-task results or the share of failures caused by minor rubric misses versus serious errors. The 92.5% criteria rate also does not reveal which types of legal criteria the model most often misses.
Z.ai describes its result as state-of-the-art among open-source models, but the cited documentation does not specify the comparison set. The source material also reports no published weights for the legal model, so its open-source status cannot be verified from the information provided. In a four-task spot check described by the source, the model cited rulings the author could not find; that small exercise is not a benchmark and does not establish a general error rate. The review also reported that other tested systems made citation errors of their own.
It remains unclear what safeguards, data-handling terms, and citation-verification procedures would apply in a firm’s deployment. Z.ai presents outputs as drafts for legal professionals, but the reviewed company pages did not include a “not legal advice” notice. That observation does not establish the terms of every product or deployment.
What Firms Should Verify Next
The next useful evidence would be fuller, reproducible reporting: the evaluation version, model configuration, task-level error patterns, and comparisons with a clearly defined set of models. Vals’s archived leaderboard is versioned, and a later listing, HLAB-Vals-1.1, was updated October 30, 2026. Results from different versions should be compared only after checking whether the test set, grading rules, and model runs are comparable.
For legal organizations considering the model, the immediate next step is a controlled evaluation on representative work, with qualified reviewers checking every citation and required detail. Procurement teams will also need written answers on client-data processing, retention, access controls, and applicable rules, as well as clarity about how the model connects to legal research and internal company data. Until those details and broader performance evidence are available, the leaderboard supports cautious testing—not a conclusion that the system can complete legal work without review.
Key Questions
What does GLM Legal’s 19.17% score mean?
It means the model passed every required rubric criterion on 19.17% of tasks in Vals AI’s 120-task private holdout evaluation. A task with even one missed criterion counts as a failure.
How can its criteria pass rate be 92.5%?
The 92.5% figure is for individual criteria, while 19.17% measures tasks on an all-or-nothing basis. A model can meet most criteria across many tasks without meeting every criterion on most tasks.
Was the result independently evaluated?
Z.ai says Vals AI independently evaluated the result, and Vals’s archived leaderboard lists GLM Legal at 19.17%, seventh among 81 models. Vals says its evaluation uses a private holdout and two AI judges.
Does seventh place prove the model is ready for legal work?
No. The rank compares models on a specific benchmark; it does not establish reliability across real legal matters. The result supports further testing, with human review and citation checks, rather than unreviewed use.
Is Z.ai’s open-source claim confirmed?
The company claims state-of-the-art performance among open-source models, but the cited material does not define the comparison set or provide published weights for the legal model. Its open-source status is not established by the evidence described here.
Source: ThorstenMeyerAI.com
