What Actually Runs on Your Machine

Two reading levels: the running text is the thread — for everyone, no prior knowledge required. The specially marked deep dive boxes are elaborations for specialists and can be skipped without losing the thread.

Price was not it — that was Part 1. What remains is the second reason people give for “local”, and it sounds better: performance. You want the best, you read that the open models are by now on a par, and you picture that best model running from here on under your own desk — no cloud, no corporation, no dependency. A lovely thought. It rests on a mix-up — the one between “open” and “local” — and that mix-up is the whole story of this part.

Because both statements are true, only not the same one: there is an open model almost at the top. And it does not run on your machine. Between those two sentences lies the gap almost nobody works out — and that we work out now.

Three leagues you must not blur together

“Open weights” — freely downloadable models — is not a size class. It is a licensing feature that cuts straight across three utterly different hardware worlds. Throw them into one pot and you produce exactly the fallacy this piece is about.

League A — the cluster. Models with hundreds of billions to well over a trillion parameters. They are open in the sense of downloadable, but you can only run them on a stack of data-centre graphics cards — eight of the most expensive kind and up. This is where the best open model sits.
League B — the one big card. Models around 120 billion parameters that fit on a single professional card or a machine with a great deal of memory. A four- to five-figure purchase, but no rack.
League C — your desk. Everything that fits on a consumer graphics card with 24 to 32 gigabytes of memory — the “5090 class”. This is what most people mean when they say “local”.

And now the numbers, measured by the analysis firm Artificial Analysis in its Intelligence Index, a composite of nine tests (as of July 2026):

Model	League	Index	where it runs
Claude Fable 5 (closed)	Top	60	provider data centre
Kimi K3 (open)	A	57	2.8 trillion parameters — data centre
GLM-5.2 (open)	A	51	server cluster
Qwen3.6-27B (open)	C	37	fits on an RTX 5090
gpt-oss-120b (open)	B	24	one professional card

The gap from the world’s best model (60) to the best open model (57) is three points — exactly the number that feeds the “open source has closed in” narrative. Only it belongs to a model with 2.8 trillion parameters — the total count of a mixture-of-experts build, all of which must sit in memory — that fits in no case standing under a desk. The gap to what actually fits on your card (37) is not three but around twenty-three points.

That is the whole trick of the debate, in one image: they show you the gap between the Premier League and the Champions League and sell it to you as the gap of your second-division side. Both numbers are correct. They just belong to different leagues.

Deep dive · for specialists One detail makes the matter even clearer. In August 2025 the model gpt-oss-120b counted as “the most intelligent model that runs on a single H100”. A year later it sits at 24 in the same index — below the 27-billion model of 2026, which fits on a far smaller card. The progress in the local class did not come, then, from the hardware getting bigger, but from architecture and training getting better. That is the good news — and at the same time the trap: whoever buys a card today for “the local top model” buys a slab of fixed capacity into a market that re-cuts its best models weekly. Six labs pushed a model over the 50 mark within a few weeks; the June before, it was two. Your graphics card depreciates more slowly than the progress it is supposed to capture.

The famous “2.7 %” — and what it really says

One more number before we go on measuring, because it meets you everywhere and is almost always read wrongly. The Stanford AI Index, a much-cited annual survey, writes that the leading US model leads “by only 2.7 %” over the best Chinese one. In headlines this becomes: “China is only 2.7 per cent worse.” Both, however, measure top against top — two data-centre models — and not what concerns us here: the top against your card. For that reason alone it is the wrong number for our question.

But it is questionable in its own right too. The 2.7 per cent is a relative gap on a scale called Arena-Elo — the rating system from chess. Elo measures who beats whom, not how good someone is in absolute points; the scale has no zero point. Forming a percentage on it is as sound as the sentence “20 degrees Celsius is twice as warm as 10 degrees”: arithmetically doable, in substance nothing. Whoever reads the Elo difference as ability-percentages has misunderstood the scale. (For the record: between the best US and the best Chinese model there are currently still 21 Elo points — that too is no percentage, but a rank figure.)

The number that fits our question is not in the Elo table but in the index table above: 60 against 37 in the class that concerns you. It counts not wins against losses but solved tasks — and this gap has a particular shape.

Where the gap comes from: not dumber, but shorter of breath

You might think a model with index 37 is simply “dumber” than one with 60. It is not that simple — and the distinction matters, because it settles what local models are good for and what they are not.

Put a single, clearly bounded question to a good local model — a maths problem, a knowledge question, a summary — and it is often astonishingly close. In serious measurements on exactly the target hardware, the local Qwen3-30B solves competition maths problems at a level that lies between two well-known cloud models. On standard knowledge tests, what separates it from the top model is a noticeable gap, but not a categorical one.

The gap opens elsewhere: with tasks that run over many steps. A modern AI agent that breaks a problem down on its own, calls tools, checks intermediate results and holds out over thirty, forty steps — that is the discipline in which the local class drops out of sight. In the leaderboard of a leading agent benchmark (Terminal-Bench), not a single open model appears among the top ten entries. That does not mean the local models fail there — they are simply not present, because the top places demand a stamina that only begins high up in League A.

The formula worth remembering: the local model is not dumber, it is shorter of breath. For a question it is close. For a task over forty steps it is no longer in the race.

What actually fits on the card

So far we have talked about quality. Now comes the physical hurdle, and it is more unforgiving than the marketing lets on. A model has to fit into the graphics card’s memory — and not just its weights, but also its “short-term memory”, the so-called KV cache, which holds the running context. And this buffer grows with the length of the text the model is meant to keep in view.

Here is the trap, cleanly worked out. A solid 32-billion model from 2025 already takes up around twenty gigabytes with its weights alone. Give it a long context of 128,000 tokens — roughly a thick book — and another thirty-two gigabytes come on top for the KV cache alone. Together over fifty gigabytes, on a card with thirty-two. It does not fit. Both advertising claims — “runs on a 5090” and “128,000-token context” — can each be true. At the same time they are not.

The good news here too is that the solution came not from more hardware but from better architecture. The 2026 models are rebuilt so that their context memory shrinks drastically: the same context window that ate thirty-two gigabytes on the old 32-billion model costs only around eight on the new 27 — a quarter; the leaner build of the same vintage even comes in under three, a tenth. That is exactly why Qwen3.6-27B fits on a 5090 with a long context and its predecessor does not — not because of the card, but because of what was built into it.

Deep dive · for specialists A second, stubborn error concerns long contexts even where they do technically fit. A model that advertises “128k context” does not necessarily process those 128,000 tokens well. The benchmark NoLiMa tested this systematically: of thirteen models that claim 128k, eleven already fall below half of their short-context performance at 32,000 tokens. The reason is mechanical: without a literal match between question and passage, the attention mechanism no longer reliably finds the right information. “Fits in memory” therefore does not mean “gets read”. Anyone planning a local model with a giant context to search whole mountains of files should know this limit before buying the card.

And even when a model fits: the number with which private users want to hang their card off several colleagues — “five users at once, no problem” — does not hold up to measurement. The software common at home gains almost nothing from serving several users simultaneously; the data-centre software gets a multiple out of the same card. Whoever hangs a 5090 off five colleagues with the comfortable home tool is in effect running a single-user machine with a queue.

The trick that makes it fit anyway — and what it costs

To make a big model fit on a small card, you compute it down: quantisation. Instead of storing each weight at full precision, you store it more coarsely — sixteen bits become four. The model shrinks by roughly a factor of four, and remarkably little happens qualitatively in this step. At four bits the loss is so small it vanishes into the measurement noise. That is why “Q4” is the standard on which local models run.

The problem starts one level down, when the memory still is not enough and you go to three bits. A systematic evaluation on a widespread 8-billion model shows: on the jump to three bits, the hit rate on maths problems collapses by over nine points — from around 78 to 68 per cent. And here is the insidious part: the standard metric usually used to measure quantisation — a statistical measure called perplexity — rises only a little in the process, far less than the maths collapse would suggest. Whoever looks only at it does not see the damage. And it matters what you compute down: as long as only the weights go to four bits, the model stays stable — that is precisely why Q4 counts as safe. But squeeze the running intermediate calculations down to four bits as well, and the hit rate on a hard maths competition falls to under four per cent. That is no contradiction of safe Q4, but its flip side: four bits on the weights holds, four bits everywhere does not.

That leads to the most uncomfortable property of a computed-down model, and it matters more than any single number: a degraded model does not hiss like a bad MP3. With bad audio compression you hear the hiss, you know something is missing. An over-quantised model still sounds assured — it phrases fluently, comes across as certain, and is wrong more often. The damage is invisible as long as you do not tease it out with the right tasks. And easy questions do not tease it out: exactly the tests a layperson poses as a check are the ones on which even the damaged model still shines.

Deep dive · for specialists The practical rule is therefore not “as big as possible” but “the biggest model that still fits in Q4”. More parameters beat more precision — but only down to about three bits. Whoever presses a 70-billion model to two bits to cram it into a laptop gets not a big model with small errors but a damaged one. And another trap hides in the fine print: when a data sheet lists “128k context”, that is often not the native length the model was trained on but an arithmetic stretch — the quality on the last tens of thousands of tokens is then yet another question in its own right. “Runs on the card” and “delivers at full length” are two promises, not one.

The swot who has the answer key by heart

A second trick makes small models seem surprisingly clever: distillation. A small model is trained not on raw text but to imitate a big one — it learns from the “teacher” by copying its answers and lines of reasoning. That is how, in 2025, tiny models were made to suddenly do maths. It works, and it has a price hardly anyone names.

A study of this brought out the mechanism nicely. When a model thinks for itself and is uncertain in the process, it verbalises that uncertainty — with filler words like “wait”, “hmm”, “maybe the other way after all”. Those are not quirks, that is the model checking its own hypothesis. A distilled model loses exactly that. Because the teacher whose reasoning it copies already knew the solution — its written-out train of thought is therefore smooth, purposeful and free of real uncertainty. The student mimics this self-assurance without possessing the knowledge that would justify it.

A distilled model is thus like a student who has learned the answer keys by heart. On the exam the teacher would have set, they shine. On a question that resembles no answer key, it shows that they never understood why. That is the same shortness of breath as above, only viewed from the other side: close in the familiar, lost in the unfamiliar.

The second item Part 1 announced

At the end of Part 1 there was a promise: to the electricity bill of your own card belongs a second item that is on no price list — the person who runs all of this. Here it is, worked out.

A local setup is not a device you buy and set down. It is an operation. The model wants updating — and an update in 2026 is not “load a new file” but potentially new runtime software, new quantisation, a new capacity calculation, because between two vintages the whole architecture has changed. The software that serves the models moves to the same beat: a reference server tool that Hugging Face itself maintained until recently now sits in maintenance mode — overtaken by its own community and quietly mothballed. Whoever bet on it in 2024 is migrating in 2026.

That costs working time, and working time has a price. A specialist who looks after such systems earns in Germany on average about 75,000 euros a year; with employer costs it is a good 90,000. Assume such a system eats only ten per cent of that post — half a day a week — and that half day costs, over the year, around 2.4 times an expensive graphics card. It thereby exceeds, in the very first year, the complete 5,000-euro machine from Part 1 — and over three years the half hour a week costs a multiple of the entire hardware.

With that, the number Part 1 promised stands in black and white: the graphics card is not the cost driver. The person who looks after it is. And that person appears in none of the hardware comparisons used to advertise “local”.

What this means for us

Performance is thereby finished as an argument for “local” — not because open models are bad, but because the open top model and the locally runnable model are two different things. And where the local class really does suffice — with single, clearly bounded questions — the same model runs at least as well in the cloud and usually cheaper (that was Part 1). Performance therefore never argues for running it yourself; at best it does not argue against. Three thoughts to take away:

“Open” is not “local”. The small 3-point gap to the top belongs to a model in the data centre. What fits on your card lags by around 23 points. Whoever advertises with the one gap and means the other model is comparing leagues.
The gap is stamina, not stupidity. For a single question the local class is fine. For long, multi-step tasks it drops away — and shrunken models hide their damage behind assured phrasing. Measure with your hard tasks, not with easy ones.
Operation beats acquisition. The card is the cheap part. The expensive part is the person who keeps updates, quantisation and outages under control — and they are on no hardware bill.

So if neither price nor performance justifies running it yourself — then why at all? Perhaps for the freedom: because an open model belongs to you, you may do with it what you want, and no one tells you otherwise. Whether that is true hangs on a word most people read past — “open”. Because open does not mean free, and what the licences really say is what we take on in Part 3.

If this piece gave you something to think about, feel free to share it — and at the next offer that promises “runs locally, top performance included”, ask for two numbers: which league does the model you can get onto your card play in — and with which tasks was it tested, easy ones or yours? This series has three more parts — licence, provenance, and what remains at the end. If you do not want to miss any of them, subscribe to the newsletter: a short note the moment a new piece appears, no promotional newsletter, unsubscribe at any time with one click.

Sources (selection):

Artificial Analysis — Intelligence Index v4.1 (nine tests), as of July 2026: Claude Fable 5 ≈ 60, Kimi K3 ≈ 57 (open, 2.8 trillion parameters), GLM-5.2 ≈ 51, Qwen3.6-27B ≈ 37, gpt-oss-120b ≈ 24: https://artificialanalysis.ai/models/open-source
Artificial Analysis — “gpt-oss-120b: the most intelligent model on a single H100” (August 2025) and market dynamics (“six labs over index 50, in June it was two”): https://artificialanalysis.ai/articles/analysis-openai-gpt-oss-models
Snorkel AI — Terminal-Bench 2.1 leaderboard (no open model among the top ranks): https://snorkel.ai/leaderboard/terminal-bench-2-1/
Stanford AI Index — “the top U.S. model leads by 2.7%”: relative gap on the Arena-Elo scale (Anthropic ≈ 1,503, Alibaba ≈ 1,449, as of March 2026; current arena gap ≈ 21 Elo points), not an ability percentage point.
arXiv 2502.05167 — NoLiMa: Long-Context Evaluation Beyond Literal Matching (ICML 2025): 11 of 13 “128k” models already fall below 50 % of their short-context performance at 32k: https://arxiv.org/pdf/2502.05167
Own KV-cache calculation from the models’ config.json (HuggingFace): Qwen3-32B (2025) 32 GiB KV cache at 128k; Qwen3.6-27B (2026, hybrid attention) ~8 GiB (factor 4); the leaner MoE variant Qwen3.6-35B-A3B 2.5 GiB (factor ~13).
arXiv 2601.14277 — Which Quantization Should I Use? (Llama-3.1-8B, GGUF): GSM8K collapses by 9.32 points at Q3_K_S (77.63 → 68.31) while perplexity barely rises: https://arxiv.org/html/2601.14277v1
arXiv 2504.04823 — Quantization Hurts Reasoning? (COLM 2025): reasoning models are quantisation-sensitive; under W4A4KV4 AIME-120 falls to 3.9 %: https://arxiv.org/html/2504.04823v1
arXiv 2603.24472 (Microsoft Research/KAIST/SNU, via bdtechtalks 13/04/2026) — distillation suppresses the “epistemic verbalisation” (the checking vocabulary “wait/hmm”), because the teacher already knows the solution: https://bdtechtalks.com/2026/04/13/llm-self-distillation-tradeoffs/
Red Hat Developer, 15/06/2026 — vLLM ≈ 44× llama.cpp at 64 parallel users; Hugging Face’s TGI “now in maintenance mode”: https://developers.redhat.com/articles/2026/06/15/llamacpp-vs-vllm-choosing-right-local-llm-inference-engine
jobvector / Glassdoor (2026) — MLOps salary Germany avg. ~75,000 €; with employer costs ~90,000 €/year. Own calculation: 10 % of a post ≈ 2.4× RTX 5090 (median 3,800 €, March 2026, ComputerBase) per year.