LLM benchmark
Every Monday I measure the frontier models on the same eight metrics and keep the results. This is the most recent run: sort it however you like, and take the data with you.
Read this week’s analysis →| Model | ||||||||
|---|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | 0.875 | 0.938 | 1.000 | 0.760 | 1.000 | 2115 ms | 226.8 | $0.00040 |
| Gemini 3.1 Pro (preview) | 0.875 | 0.938 | 0.846 | 0.960 | 1.000 | 5012 ms | 201.9 | $0.00040 |
| MiniMax M3 | 0.625 | 0.875 | 0.692 | 0.400 | 1.000 | 1903 ms | 221.7 | $0.00044 |
| GPT-6 Sol | 1.000 | 1.000 | not measured | 0.960 | 0.875 | 1748 ms | 207.3 | $0.00073 |
| GLM-5.3 | 1.000 | 0.813 | 1.000 | 0.960 | 1.000 | 5012 ms | 149.4 | $0.00158 |
| Qwen3.8-Max | 1.000 | 1.000 | 1.000 | 0.880 | 1.000 | 9418 ms | 42.5 | $0.00198 |
| Claude Opus 5.5 | 1.000 | 1.000 | 1.000 | 1.000 | 0.875 | 2551 ms | 208.4 | $0.00258 |
| Kimi K3 | 1.000 | 0.867 | 1.000 | 1.000 | 1.000 | 8308 ms | 51.5 | $0.00374 |
| Grok 4.7 | 1.000 | 1.000 | 1.000 | 0.960 | 1.000 | 6428 ms | 83.7 | $0.00403 |
The same records this table is drawn from, prices and sources included. No sign-up, no asterisk.
What the noise is worth
Two runs of the same models, same prompts, same settings, do not give the same numbers: models answer at whatever temperature their provider ships. These are the gaps measured between the last two runs. A difference smaller than this is not a change in the model.
- Constraints: average gap 0.031, largest 0.125
- Italian: average gap 0.065, largest 0.133
- Tools: average gap 0.096, largest 0.307
- Consistency: average gap 0.067, largest 0.120
- First token: average gap 1843 ms, largest 4193 ms
- Tokens/s: average gap 31.5, largest 91.7
- Cost/task: average gap $0.00040, largest $0.00140
Series over time arrive once there are enough runs to tell them from noise. A line through two points taken on the same day would be the first dishonest thing here.
Undocumented system prompt
Input tokens charged for the single word "Ciao". It is prompt the provider adds and never documents, and you pay it on every call.
- Grok 4.7: 1243
- MiniMax M3: 178
- Kimi K3: 87
- Qwen3.8-Max: 63
- DeepSeek V4.1 Flash: 32
- GLM-5.3: 14
- Claude Opus 5.5: 12
- GPT-6 Sol: 8
- Gemini 3.1 Pro (preview): 2
What each metric measures
Cost per passed task
What it cost to get one good result.
Real spend for the battery — input and output tokens at that day's prices — divided by the number of tasks passed. A model that fails half of them pays for its retries here.
This is the number that decides, and it is not the price per million tokens: a model that thinks for a thousand tokens before answering pays for all of them.
Italian
A grammar check over 16 sentences, not a judgement of style.
16 cases with exactly one right answer, checked by a regular expression: the subjunctive after a concessive, plurals that change gender, past participle agreement after a clitic, the passato remoto, the formal register, elision, accents, and the absence of English loanwords. No judge model: a judge would import its own preferences, and using one contestant to grade the others is worse than not measuring.
It measures control of the language, not the beauty of the prose. A narrow scope, stated: what can be verified is verified, and the rest is not pretended.
The code runs
Not "the code looks right": it runs and returns the right answers.
8 JavaScript functions written to a specification. The generated code really runs, in an isolated Node 22 container with the network switched off, against assertions the model never saw. Edge cases are written into the prompt, so a failure is incompetence rather than a guessing game.
Reading code and judging it measures the judge. Running it measures the code.
Tool calling
Whether the model can be put inside an agent.
One request, 3 declared tools, four independent checks: it called a tool rather than answering in prose; it picked the right one; the arguments are valid JSON; the values are correct, types included. The score is the fraction passed, because they fail independently.
One of the three tools is deliberately close to the right answer. And arguments a caller cannot parse are unusable however right they read.
Consistency
How often it gives the same answer to the same question.
The same question 25 times, at whatever temperature the provider ships. The score is the share of answers matching the most common one.
It measures consistency, **not correctness**: a model that is consistently wrong scores 1.000. That is deliberate — they are independent properties, and an unpredictable model cannot be tested, cached, or promised to a client.
Constraint adherence
Whether it does the boring thing it was told to do.
8 cases: stay under a word limit, avoid a forbidden word, return JSON and nothing else.
It separates a model that can sit behind an API from one that can only be read by a person.
Time to first token
How long somebody watching the screen waits before anything appears.
Averaged across the battery, timed to the first visible token: a model that thinks for twenty seconds and then answers has kept you waiting twenty seconds, even though it was generating throughout.
This is perceived responsiveness.
Tokens per second
How fast it produces, once started.
Every generated token — thinking included — over the generation window, which opens at the first token of any kind. Not reported when the response was not genuinely streamed: some providers buffer it and flush it in one go, and the naive division read 8,000 tok/s.
It matters if you show the text as it arrives.
How this is measured
Each model answers at the reasoning setting its provider ships by default — what you get if you touch nothing. Generated code really runs, in an isolated container with no network. Prices are the ones in force on the day and stay frozen with the result.