In October 2026, the large model industry experienced two contrasting narratives.

If one only looked at the results of traditional benchmark tests, mainstream models seemed to have aced the exams.

According to statistics from the third-party evaluation aggregation platform BenchLM as of October 26, among the 710 benchmarks it tracked, 295 were percentage-based benchmarks with actual coverage. Of these, 105 (approximately 36%) were completely “saturated,” with top scores all above 90 points.

When scores are all clustered near the ceiling, it becomes unclear whether a one- or two-point difference is due to a stronger model or random fluctuations in prompts and sampling.

Over the past month, data annotation and evaluation company Surge AI released three consecutive sets of new assessments targeting real-world workplace tasks, pulling many flagship models below the passing mark:

sudo L7 (Senior Engineer-Level Programming Evaluation): 60 tasks sourced from real production codebases. On the leaderboard, Claude Opus 5.5 and Sonnet 5.5 tied for first place with a pass rate of only 41.7%; OpenAI’s GPT-6 Astra scored 33.9%; Mistral Large 4, which launched in preview on October 6, scored just 11.7%.

GDP.xlsx (Cross-Industry Complex Spreadsheet Evaluation): 70 real Excel tasks from the daily work of professionals in 12 industries, including finance, healthcare, and real estate. Google’s flagship model, Gemini 4 Argon, ranked first with a score of only 38.3%.

DAYJOB (End-to-End Job Role Evaluation): 130 long-flow tasks compiled by healthcare and finance experts. The best score in healthcare was 24.7% by Claude Opus 5.5, while in finance, it was 24% by Claude Sonnet 5.5.

Dropping from scores in the nineties on old leaderboards to scores in the twenties or thirties on new ones might sound like the models suddenly became less intelligent, but in reality, the exams changed—old questions tested “whether a model could follow clear instructions,” while new questions tested “whether a model should act this way when faced with a mess and whether it missed anything.”

When the exam questions are crafted by a new set of test-makers, the rankings, the value of the scores, and the underlying rules of the evaluations all change accordingly.

01 Engineering Failures: The Gap Between “Scoring High” and “Deliverability”

Why is a model that scores near-perfect on leaderboards deemed too risky to deploy directly by enterprise technical leaders?

In the sudo L7 evaluation released in October, there was a highly representative real-world question set against a real fintech codebase integrated with Stripe, Mailchimp, and Plaid. The questioner gave the model a clearly written request:

After receiving the instruction, GPT-6 Astra proceeded as follows: it located the code, modified the logic, and ran checks, delivering a neat implementation of “full logging + forced return 200.” Judging by unit tests alone, this answer seemed flawless.

The problem lay in the request itself.

On one hand, returning HTTP 200 tells Stripe, “I’ve processed this callback,” and the other side will not resend it; if the internal processing actually failed at this point, the user’s deduction record might never match up.

On the other hand, the payload of financial callbacks contains customer identities and sensitive account information. Writing the entire package into ordinary logs without desensitization allows all operations and monitoring systems to see it in plaintext, potentially retaining it for years.

A seasoned human engineer receiving such a request would typically halt it first, explain the risks of lost transactions and data leaks, and then propose an alternative: first persistently save the event securely, then return 200, handle failures asynchronously via an internal queue, and desensitize sensitive fields in the logs.

However, GPT-6 Astra did not think of this layer.

Unlike previous evaluations, sudo L7’s detailed scoring for the first time separated “code that runs” from “code that is safe to deploy”:

In terms of “functional correctness,” the difference among the top four models—GPT-6.1 Sol (82.4%), Claude Opus 5.5 (82.1%), GPT-6 Astra (81.5%), and Claude Sonnet 5.5 (79.6%)—was minimal. The correctness of the code was no longer a differentiator among leading models.

The dividing line appeared in engineering judgment.

In the dimension of “Thought Partnership,” which examines whether a model questions requirements and raises objections like a colleague, the two Claude models scored around 75%, while GPT-6.1 Sol and GPT-6 Astra dropped to 53.0% and 51.6%, respectively. In the “communication” dimension, Sonnet 5.5 scored 72.4%, while Sol scored only 44.1%.

But this does not mean Claude can rest easy.

In another charity donation application, the requirement was to “allow users to split donations proportionally among different institutions.” Claude Opus 5.5 wrote an entirely new front-end interface and data tables, even writing and running 267 unit tests and 4 browser tests on its own.

Embarrassingly, the application already supported donation splitting.

Opus 5.5 had encountered the relevant implementation while retrieving the code but did not stop to ask; instead, it started from scratch, leaving two parallel, overlapping logics in the system. All tests passed, but a mature engineer would never deliver it this way.

Together, these two cases illustrate different failure modes of large models: one was too obedient, failing to question absurd requirements; the other was too capable, failing to confirm existing conditions. If one only looked at whether unit tests passed—the criterion of old leaderboards—neither of these “production hazards” would be detected.

02 The “97% Accuracy Paradox”: Small Errors Magnified by Long Flows

If code testing exposed the lack of engineering judgment, GDP.xlsx and DAYJOB revealed another headache for enterprise buyers: models that score high on single-step questions can have their small errors drastically amplified when placed in multi-table dependencies and long-chain processes.

GDP.xlsx included a rent prediction question for a commercial real estate fund.

The workbook had 15 sheets, about 60 cell comments, and an email from an asset manager requesting the identification of all tenants whose rent might change within the next three years, including those whose discounts were expiring and those with early termination rights.

Meta’s Muse Spark 1.3 identified all six tenants whose discounts were expiring, even catching half-price discount clauses hidden in cell comments and emails. Then it stopped, never opening the sheet named BreakClauses on the right side of the workbook—which listed three more tenants who could terminate early, involving approximately $14 million in predicted rent.

In another product certification scheduling question, GPT-6 Astra saw that both products A and B on the main sheet were marked “approved” and recommended launching both simultaneously, failing to check the foundry restrictions on another sheet. It confidently stated that “the workbook does not specify relevant suppliers,” even though the supplier names were written on other sheets.

These errors share a commonality: the model found part of the answer and assumed it had exhausted the entire problem.

In the DAYJOB evaluation, single-step oversights were further amplified by long workflows. Each DAYJOB task came with a binary rubric developed by industry experts, with a median of 47.5 criteria in healthcare and 57.5 in finance.

Consider a rough mathematical estimate:

If a task is only considered qualified when all criteria are met, and assuming the criteria are independent, the highest task pass rate of 24.7% in healthcare implies that the model achieved 97% accuracy on each criterion—calculating 0.97 to the 48th power (for a near-median 48 criteria) yields approximately 23%.

This is just an idealized illustration; in reality, the criteria are not fully independent.

Nevertheless, it intuitively (intuitively) demonstrates a point: in a business process with nearly fifty interlocking steps, even if each step is “highly likely to be correct,” the probability of the entire process succeeding drops below 30% after compounding.

Whether an enterprise dares to entrust a complete job role to AI depends precisely on this final number.

03 Who Grades the Graders: When Evaluation Itself Becomes Big Business

Since old leaderboards have lost their credibility, are new practical evaluation leaderboards like those from Surge AI the ultimate, objective arbiters?

The answer is no.

To understand large model leaderboards, one must first see who sets the questions and how the “graders” make money.

Over the past year, large model evaluation has evolved into a highly valued business. LMArena (formerly Chatbot Arena), which initially relied on anonymous user voting to rank models, completed a $150 million Series A round in January 2026 at a $1.7 billion valuation.

Surge AI, which released three new leaderboards, is LMArena’s most vocal critic.

As early as December 2025, the Surge team manually reviewed 500 real head-to-head votes on LMArena, disagreeing with 52% of the results and strongly disagreeing with 39%. For example, in one vote, the winning model claimed that “a 9-inch round cake pan is the same size as a 9×13-inch rectangular pan.”

Surge explained that ordinary users often vote based on a few seconds of scanning, favoring answers with better formatting or more confident tones.

Not only criticizing crowdsourced blind testing, Surge has also publicly criticized OpenAI’s own workplace benchmark, GDPval, stating directly in DAYJOB’s explanation that only about 20% of the GDPval scoring criteria they reviewed were directly usable.

These criticisms hit the mark, but the issue is that Surge AI’s main business is providing premium data annotation, reinforcement learning environments, and expert human evaluation services to major model labs.

According to Reuters, Surge AI’s revenue exceeded $1 billion in 2024, with clients including multiple leading labs such as Anthropic. The conclusion that “public automated leaderboards are insufficient, crowdsourced voting is noisy, and long-form evaluations by highly paid industry experts are necessary” aligns perfectly with its business model.

From another perspective, the high cost of expert-crafted questions means new leaderboards cannot have a large question volume.

For example, sudo L7 has only 60 questions, with each question accounting for about 1.7 percentage points. Models in the middle of the leaderboard that differ by two or three percentage points may only differ by one or two questions on the exam, introducing significant randomness.

This is not an isolated case.

As early as November 2025, a paper signed by 42 researchers systematically reviewed 445 large model benchmarks, pointing out that many did not clarify what abilities they were testing and rarely performed statistical tests when comparing models.

Among BenchLM’s current methodologies covering 229 ranked models, rankings are only marked as “supported” when different types of external evidence corroborate each other; all others are marked as “estimates.” Scores from vendor self-tests are separately labeled and excluded from rankings, and “generated data” interpolated from similar models is also excluded.

In the evaluation circle, there is no truly “disinterested third party.” For readers and buyers, what matters more than memorizing a specific ranking is first examining the exam’s credentials: who set the questions, how many there are, what the sample error is, and who is paying for the exam.

04 Chinese Models’ “Two Report Cards”

Shifting the perspective back to China, the sense of divergence between “different exams yielding different worlds” is even stronger.

Judging by real-world usage alone, Chinese models are thriving.

According to statistics released by the global API routing platform OpenRouter on October 20, Chinese models accounted for 79.18 trillion tokens in calls on the platform the previous week, approximately 4.5 times that of U.S. models (17.36 trillion). The top five models in weekly call volume were all Chinese.

In the latest weekly leaderboard released on October 26, DeepSeek V4.1-Flash saw an 801% week-over-week increase in call volume, solidifying its global first-place position.

However, on Surge AI’s three new leaderboards, the picture was completely reversed:

On sudo L7, the best-performing Chinese model was Alibaba’s Qwen 3.8 Max (20.6%, ranking sixth); Zhipu’s GLM 5.3 scored 14.4%, Moonshot’s Kimi K3 scored 13.3%, and Tencent’s Hunyuan (Hy4 Preview) scored 12.8%.

On GDP.xlsx, GLM 5.3 scored 21.4%, and Qwen 3.8 Max scored 20.3%, not far behind GPT-6.1 Sol (22.0%), but Kimi K3 (13.4%) and DeepSeek V4 Pro (6.9%) lagged significantly.

The largest gap appeared in DAYJOB’s healthcare track: Qwen 3.8 Max scored only 2%, Kimi K3 scored 1.2%, and both GLM 5.3 and DeepSeek V4 Pro scored just 0.4%.

Why do Chinese models lead globally in call volume while performing nearly blank on long-flow new leaderboards?

A straightforward explanation is that Chinese models currently excel in high-frequency, low-unit-price, single-step tasks, with global developers voting with their feet due to cost-effectiveness; however, they still lag behind U.S. leading models in engineering tasks requiring dozens of consecutive steps and deep integration into complex systems.

Before making a conclusion, there are at least four test conditions that need to be stripped away:

First, the toolchain shell (Harness) is not equivalent.

On sudo L7, Claude runs in its native Claude Code, GPT runs in Codex, and Gemini runs in Antigravity; while for Chinese models, except for Alibaba's Qwen Code, GLM, Kimi, and Hunyuan are all run within third-party OpenCode shells. Engineers experienced in agent development are well aware that an agent's performance is highly dependent on its compatibility with the outer toolchain. Switching to a non-native shell can directly reduce scores.

Second, the tested versions are outdated.

Surge tested the earlier Kimi K3 and DeepSeek V4 Pro; on Artificial Analysis's latest comprehensive intelligence index, the recently updated Kimi K3.1 Pro in October scored 55 points, just 3 points behind the top-ranked Claude Opus 5.5 (58 points); DeepSeek's latest V4.1 Flash also scored 45 points, significantly higher than the tested V4 Pro (36 points).

Third, there are geographical and institutional barriers in the test questions.

The tasks in DAYJOB's healthcare sector cover clinical, pharmacy, payers, operations, and compliance, compiled by experts from this U.S. evaluation company. Content such as healthcare payment and compliance processes varies greatly across countries, which may not be fair for models primarily trained for the domestic market.

Fourth, there is statistical noise from small sample sizes.

The healthcare sector has a total of 65 questions, and the difference between 0.4% and 2% essentially translates to getting half to one question right, which is nearly negligible statistically.

Taking all these variables into account, a fairer judgment can be made: In long-process tasks designed by U.S. evaluation agencies and run on U.S. toolchains, Chinese models do indeed lag significantly; however, determining how much of this gap is due to the foundational models themselves versus the testing conditions requires domestic vendors and evaluation agencies to complete the exam with Chinese business workflows and native toolchains.

05 In Conclusion

After stripping away all ranking controversies, the most practical value the new exam holds for corporate buyers is actually about cost-effectiveness.

According to the same batch of test data disclosed by Surge in the sudo L7 blog (with slightly different pass rates in subsequent ranking list (leaderboard) updates), Claude Opus 5.5 and Sonnet 5.5 have pass rates of about 45%, with costs of approximately $10 and $14 per task, respectively; GPT-6.1 Sol has a pass rate of 30%, with a cost of just over $1 per task.

If we factor in their respective pass rates and convert them into 'how much it actually costs to complete one qualified task':

Claude Opus 5.5: about $22;

Claude Sonnet 5.5: about $31;

GPT-6.1 Sol: just over $3;

Choosing between $22 and just over $3 doesn't depend on who ranks first on the leaderboard, but rather on 'how much a failure actually costs in your specific business.'

If writing a wrong Webhook code leads to online order losses and compliance incidents, spending nearly $20 more for a 15-percentage-point improvement in judgment is still a bargain; if it's for large-volume, low-cost internal daily scripts where error tolerance is high, the top-ranked model that costs nearly ten times more is purely a waste of computing power.

The market has already begun to reflect this differentiation.

Since 2026, Zhipu has raised its average API price by about 101%, with its API business gross margin turning positive from -0.4% to 24.6%, without significant customer loss; after DeepSeek introduced time-based pricing, prices doubled during peak hours yet demand still outstripped supply.

As the era of benchmark hype fades, every corporate technology leader, before paying to access an API, needs to ask only three questions: Do the exam questions on this leaderboard resemble my real business? Is the toolchain shell used in testing the same as what I run online? Based on the failure costs I can bear, how much will I actually pay for each completed task?