DeepSeek V4 Pro leads on cost in the latest 9 LLMs benchmark

The August 12, 2026 run of our 9 LLMs benchmark proves that raw capability is entirely decoupling from execution cost. DeepSeek V4 Pro achieved a perfect 1.000 score in sandboxed code execution while maintaining the lowest cost per passed task at exactly $0.00029. The total cost to execute the entire benchmarking suite across all 9 models was $1.1078, a negligible sum that highlights the shifting economics of API infrastructure.

This test queries every model at the default reasoning level configured by the provider. We measure the baseline performance a developer receives without manually adjusting API parameters. For code generation, the output is executed in a strictly isolated environment with no network access. The score relies on actual test assertions passing, rather than superficial evaluations of the code's appearance. The pricing calculations are based on the official list rates published on the exact date of the measurement. If a vendor changes their pricing tomorrow, the financial calculations for this specific run remain locked to the August 12 rates.

The reality of measurement noise

Before interpreting any variations from the previous run, we must establish the baseline noise of the benchmark. The August 12 run utilized the exact same prompts, test cases, and configuration parameters as the previous execution. Any recorded differences are the result of non-deterministic model behavior.

The telemetry recorded specific deviations across all categories. Constraint adherence showed a mean variance of 0.014 with a maximum of 0.125. The Italian language metric recorded a mean variance of 0.059, peaking at 0.250. Tool calling deviated by a mean of 0.034 and a maximum of 0.307. Coherence fluctuated by a mean of 0.070, up to 0.160. Code execution stability showed a mean shift of 0.042 and a maximum of 0.125.

Latency metrics are similarly volatile. Time to first token shifted by a mean of 260 ms, with spikes up to 610 ms. Output generation speed, measured in tokens per second, showed a mean variance of 14.8, peaking at 49.7. Finally, the total cost per passed task fluctuated by a mean of $0.00030, with a maximum variance of $0.00070.

A metric comprised of eight test cases will shift by 0.125 if a single execution alters its outcome. Reporting variations within these thresholds as definitive regressions or improvements is statistically invalid. If a delta falls inside these parameters, the model's performance is effectively static, and the measurement is simply noisy. Acknowledging this volatility is the baseline for honest API evaluation.

For instance, DeepSeek V4 Pro increased its code execution score from 0.875 to 1.000, a delta of 0.125. This sits exactly on the maximum noise threshold. Claude Opus 5 saw its Italian score drop from 0.813 to 0.563, a 0.250 reduction that precisely matches the maximum recorded noise for that category. MiniMax M3 experienced a tool calling drop from 0.692 to 0.385, a delta of 0.307, which is exactly the maximum noise ceiling. These shifts reflect API instability rather than core capability degradation.

Further telemetry confirms this pervasive instability across the board. Claude Opus 5 saw its time to first token stretch from 3073 ms to 3532 ms, an increase that remains well within the 610 ms maximum noise threshold. Concurrently, its generation speed dropped from 84.4 to 68.4 tokens per second. While its latency fluctuated, its actual cost per passed task decreased from $0.00327 to $0.00257, a $0.00070 delta that exactly hits our absolute maximum recorded noise for cost variance.

DeepSeek V4 Pro exhibited similar non-deterministic shifts. Its coherence score dropped from 0.800 to 0.720, sitting just above the mean noise of 0.070. Its time to first token improved slightly, shifting from 4386 ms down to 4087 ms, while its generation speed marginally increased from 72.9 to 76.7 tokens per second. Its cost per passed task remained utterly static at $0.00029.

MiniMax M3 also demonstrated wild swings bound entirely by the noise ceilings. Its constraint adherence climbed from 0.625 to 0.750, a 0.125 jump identical to the maximum noise allowance. Similarly, its coherence score surged from 0.360 to 0.520, yielding a 0.160 delta that exactly matches the maximum variance recorded for that specific metric. Its time to first token shifted from 3755 ms to 3562 ms, and its output speed jumped from 95.2 to 130.0 tokens per second.

Other models displayed identical baseline noise. Gemini 3.1 Pro (preview) achieved a coherence improvement from 0.960 to 1.000, though its tokens per second dropped steeply from 233.2 to 183.4, and its latency shifted from 5387 ms to 5442 ms. GPT-5.5 saw its coherence drop from 0.800 to 0.760, with latency shifting from 2240 ms to 2360 ms, and its speed dropping from 273.9 to 250.0 tokens per second. Its cost shifted from $0.00306 to $0.00270. Qwen3.8-Max shifted in coherence from 0.760 to 0.840, with latency marginally improving from 13935 ms to 13611 ms. Grok 4.5 experienced a coherence drop from 0.960 to 0.800, improved its first token time from 4223 ms to 4002 ms, but astonishingly maintained an identical 63.4 tokens per second across both runs. Kimi K3 saw a flawless Italian score jump (0.846 to 1.000) while coherence dipped slightly (1.000 to 0.933) and latency held steady at 9457 ms to 9397 ms. GLM-5.2 experienced drops in both Italian (0.813 to 0.750) and executable code (1.000 to 0.875), while its time to first token worsened from 6350 ms to 6960 ms.

Pricing metrics: List rates versus actual execution cost

The advertised cost per million tokens rarely reflects the real financial load in production. The cost per passed task metric normalizes the expense by measuring exactly how much money is required to yield a strictly correct result.

DeepSeek V4 Pro lists at $0.435 per million input tokens and $0.87 per million output tokens. It ranks first as the cheapest model by list price and retains first place for the actual cost per passed task. During the execution, it generated 17,990 output tokens. Notably, 16,196 of those were reasoning tokens, proving that extensive internal computation does not break the budget if the base rate is low enough.

Gemini 3.1 Pro (preview) ranks sixth in list price ($2 input / $12 output per million) but jumps to second place in real execution cost ($0.00038 per task). It accomplished this by utilizing a highly constrained output of only 1,552 tokens, with reasoning tokens remaining undeclared by the provider.

MiniMax M3 is the second cheapest by list price ($0.3 / $1.2) but drops to third in actual execution cost ($0.00049). It output 15,473 tokens, allocating 7,243 to reasoning. Grok 4.5 ($2 / $6) sits fifth in list price and fourth in actual cost. The telemetry for Grok 4.5 recorded a severe anomaly: it reported 1,345 total output tokens, yet claimed 12,355 reasoning tokens.

Claude Opus 5 ($5 / $25) ranks eighth in list price and fifth in real cost, generating 5,014 output tokens without declaring reasoning data. GPT-5.5 is the most expensive model on paper at $5 / $30, placing ninth in list price. It ranks sixth in actual cost ($0.00270 per task), generating 5,133 tokens with 3,172 allocated to reasoning.

The lower tier is dominated by high-volume reasoning models with higher base rates. GLM-5.2 ($1.4 / $4.4) ranks third in list price but falls to seventh in real cost ($0.00285 per task), consuming 30,845 output tokens (29,332 reasoning). Qwen3.8-Max ($2 / $6) places fourth in list price and eighth in real cost ($0.00334 per task), generating 34,363 tokens (32,327 reasoning). Kimi K3 ($3 / $15) ranks seventh in list price and finishes last in real cost ($0.00410 per task), outputting 11,844 tokens (9,604 reasoning).

What each metric measures

Cost per passed task — What it cost to get one good result.

Real spend for the battery — input and output tokens at that day's prices — divided by the number of tasks passed. A model that fails half of them pays for its retries here.

This is the number that decides, and it is not the price per million tokens: a model that thinks for a thousand tokens before answering pays for all of them.

Italian — A grammar check over 16 sentences, not a judgement of style.

16 cases with exactly one right answer, checked by a regular expression: the subjunctive after a concessive, plurals that change gender, past participle agreement after a clitic, the passato remoto, the formal register, elision, accents, and the absence of English loanwords. No judge model: a judge would import its own preferences, and using one contestant to grade the others is worse than not measuring.

It measures control of the language, not the beauty of the prose. A narrow scope, stated: what can be verified is verified, and the rest is not pretended.

The code runs — Not "the code looks right": it runs and returns the right answers.

8 JavaScript functions written to a specification. The generated code really runs, in an isolated Node 22 container with the network switched off, against assertions the model never saw. Edge cases are written into the prompt, so a failure is incompetence rather than a guessing game.

Reading code and judging it measures the judge. Running it measures the code.

Tool calling — Whether the model can be put inside an agent.

One request, 3 declared tools, four independent checks: it called a tool rather than answering in prose; it picked the right one; the arguments are valid JSON; the values are correct, types included. The score is the fraction passed, because they fail independently.

One of the three tools is deliberately close to the right answer. And arguments a caller cannot parse are unusable however right they read.

Consistency — How often it gives the same answer to the same question.

The same question 25 times, at whatever temperature the provider ships. The score is the share of answers matching the most common one.

It measures consistency, not correctness: a model that is consistently wrong scores 1.000. That is deliberate — they are independent properties, and an unpredictable model cannot be tested, cached, or promised to a client.

Constraint adherence — Whether it does the boring thing it was told to do.

8 cases: stay under a word limit, avoid a forbidden word, return JSON and nothing else.

It separates a model that can sit behind an API from one that can only be read by a person.

Time to first token — How long somebody watching the screen waits before anything appears.

Averaged across the battery, timed to the first visible token: a model that thinks for twenty seconds and then answers has kept you waiting twenty seconds, even though it was generating throughout.

This is perceived responsiveness.

Tokens per second — How fast it produces, once started.

Every generated token — thinking included — over the generation window, which opens at the first token of any kind. Not reported when the response was not genuinely streamed: some providers buffer it and flush it in one go, and the naive division read 8,000 tok/s.

It matters if you show the text as it arrives.

Open vs. Closed Ecosystems and Structural Costs

Looking beyond individual performance, the benchmarking suite highlights the financial divide between open-weight and closed-weight architectures, as well as their geopolitical origins. The four open-weight models evaluated—DeepSeek V4 Pro, MiniMax M3, Kimi K3, and GLM-5.2—achieved an average cost per passed task of $0.00193. By contrast, the five closed-weight models—Claude Opus 5, Gemini 3.1 Pro (preview), GPT-5.5, Qwen3.8-Max, and Grok 4.5—averaged a significantly higher $0.00226 per successful execution. The ecosystem measured in this run is closely divided by jurisdiction, featuring five models originating from Chinese providers and four from the United States.

Speed and Undisclosed System Context

When evaluating raw speed, GPT-5.5 leads the pack by delivering the fastest time to first token at 2360 ms, followed by Claude Opus 5 (3532 ms) and MiniMax M3 (3562 ms). At the opposite extreme, Qwen3.8-Max suffers from severe latency, requiring a massive 13611 ms just to begin generating text, while Kimi K3 is also heavily delayed at 9397 ms. Throughput paints a similar picture: GPT-5.5 pushes an impressive 250.0 tokens per second, dominating the field, whereas Kimi K3 drags at just 42.7 tokens per second.

Crucially, performance latency is heavily influenced by how providers handle API inputs. Before a single user token is processed, providers routinely inject undisclosed system prompts that consume the context window. We measured this overhead by sending a prompt containing only the single word "Ciao". The results reveal massive disparities in hidden context bloat. Grok 4.5 injects a staggering 495 hidden tokens. MiniMax M3 follows with 178 undocumented tokens, Kimi K3 adds 87, Qwen3.8-Max injects 50, and GLM-5.2 forces 14. GPT-5.5 injects 8 tokens, DeepSeek V4 Pro adds 6, and Gemini 3.1 Pro (preview) adds a minimal 2 tokens. Only Claude Opus 5 operates completely transparently, injecting 0 hidden tokens for this baseline input.

Language Capabilities: The Italian Stress Test

The evaluation of non-English capabilities, specifically Italian grammar and syntax, exposed stark differences in linguistic refinement. Gemini 3.1 Pro (preview) achieved a perfect 1.000 score, producing no grammatical errors. Kimi K3, GPT-5.5, Qwen3.8-Max, and Grok 4.5 also demonstrated flawless execution with zero logged syntax errors.

On the other end of the spectrum, Claude Opus 5 placed dead last with a score of 0.563, struggling comprehensively across fundamental linguistic rules. It failed test cases evaluating past participle agreement (participio-accordo), the past historic tense (passato-remoto), elision (elisione), indirect pronouns (pronome-indiretto), truncation (troncamento), articulated prepositions (preposizione-articolata), and the present perfect auxiliary (passato-prossimo-ausiliare). GLM-5.2 scored a 0.750, faltering on formal register (registro-formale), elision (elisione), the pluralization of nouns ending in '-cia' (plurale-cia), and the present perfect auxiliary. Both DeepSeek V4 Pro and MiniMax M3 recorded isolated failures exclusively tied to the pluralization of '-cia' nouns.

Syntax failures, coherence, and code anomalies

Claude Opus 5 maintains its position in orchestration, ranking first in both tool calling (1.000) and constraint adherence (1.000), while MiniMax M3 ranks last in both categories (0.385 and 0.750, respectively). However, Opus 5 was not without issues; the telemetry flagged a distinct anomaly during its execution, throwing an AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal: error in its code runs evaluation. DeepSeek V4 Pro, by contrast, secured first place in the strictly isolated code execution tests, achieving a flawless 1.000 score. GLM-5.2 finished last in this category at 0.875.

Coherence—measured by prompting the model with the exact same input across five iterations to evaluate determinism—yielded the most varied data of the benchmark. Claude Opus 5 and Gemini 3.1 Pro (preview) achieved perfect 1.000 coherence scores, outputting the string rust identically across all 25 recorded repetitions.

DeepSeek V4 Pro, despite its high capabilities elsewhere, exhibited notable instability. Across 25 repetitions for the same prompt, it returned four completely different formats: c, rust, c++, and empty responses. Its precise output sequence read: c, rust, c, c, c++, c, c, c, c, rust, c, c++, c, c, c, [empty], c, c, c, rust, c, c, c, c, c++.

MiniMax M3 was similarly erratic in its coherence loops, producing strings of rust, c, c++, go, and multiple blank outputs. GLM-5.2, which ranked last in overall coherence at 0.440, frequently failed to output any string at all, returning empty values 12 times out of 25 iterations. GPT-5.5 fluctuated cleanly between c, c++, and rust, while Qwen3.8-Max and Grok 4.5 simply alternated between rust and c. Kimi K3 also mixed rust and c without blank errors. Finally, alongside its varied outputs, MiniMax M3 triggered an additional anomaly log regarding its undocumented system prompt, which actively interferes with deterministic execution.

Measured on 12 August 2026 against 9 models, each at the reasoning setting its provider ships by default. Prices are the ones in force that day. The whole run cost $1.1078 — every figure here comes from calls I paid for.

---

📖 Related articles

• Agentic Coding: Una Trappola per lo Sviluppo Software?
• File agents.md: utili per gli agenti di coding?
• Lean-ctx: Ottimizzatore Ibrido Riduce Consumo Token LLM del 89-99%