Local LLM benchmarks on 16 GB of consumer AMD

The hardware is the argument

Sixteen gigabytes is the binding constraint on everything below. Every -ncmoe, every KV quantisation choice and every -ub value is bought against it. The rest of the box matters because expert layers that don't fit on the card get streamed from DDR4 at ~40 GB/s instead of read from VRAM at ~640 GB/s.

ROCm for everything, and why Vulkan was retired

Measured head-to-head with llama-bench, -fa 1 throughout. llama-bench defaults flash attention off, but llama-server resolves auto → enabled, so -fa 0 numbers do not describe real serving — an earlier revision of this record got that wrong.

Vulkan's only remaining advantage was shallow MXFP4 generation on exactly one preset, and router mode always spawned children from the ROCm binary — so in daily use that advantage was never actually being collected. build-vulkan/ was dropped on 2026-08-17, trading 18 % of shallow generation on one preset for a single backend with no version skew. It has been rebuilt twice since, from the same checkout, to spot-check new models: Qwen3.8-27B gives ROCm +81 % generation and Laguna XS.2 gives it +88 % prompt — the widest gaps in the table. The routing decision holds.

Attention topology, not parameter count

Three models here are hybrid attention: full_attention_interval = 4 means only every 4th layer keeps a KV cache, the rest carry a constant-size recurrent state that does not grow with context. That is the whole reason 1M fits in 16 GB. But hybrid attention is not by itself a promise of cheap context — read head_count_kv and block_count alongside the interval.

MLA compression beats sparsity. GLM-4.7-Flash caches a single compressed latent per layer rather than per-head K and V — 576 elements per layer per token, applied at every layer. Measured empirically at ≈27,915 B/token, within 3 % of the formula, and still cheaper than Qwen3.8's sparse-but-fat 34,816.

Seventeen models

Everything that has been through the suite. Rows marked deleted are no longer on disk — their measurements stand, and each was removed for a stated reason rather than quietly dropped.

Also measured and since removed: Qwen3-Coder-30B-A3B (Q8_0, 32.5 GB), Qwen2.5-Coder-14B and -32B, GPT-OSS-120B (MXFP4, 59 GB).

What actually loads, and the flags it takes

Every row below was actually loaded, confirmed to reach server is listening, with VRAM read while resident. All include -ngl 99 unless noted. Contexts are exact token counts, not rounded labels.

Per-model presets

† Beyond the model's native 262,144 — needs YaRN RoPE scaling, and is not exposed as a preset.

The rejected pile

A config that loads is not the same as a config that works. These were all measured and thrown away, and several look like passes until you check the right column.

Speed, and how much of it depth takes back

llama-bench, -r 2, ROCm build. Shallow unless a depth is named. The interesting axis is not peak tok/s — it is how much survives to 131K, and that varies by 3.3× across the models here.

Expert offload: the -ncmoe sweeps

Monotonic in every case — every expert layer moved onto the GPU helps. The usable value is whatever the target context leaves room for.

Laguna XS.2 is faster than Qwen3.6 at every matched value despite an almost identical file size (18.9 GB vs 20.6 GB) — its 512-wide expert FFN is lighter per active parameter. GLM-4.7-Flash reaches Qwen3.6's best published generation (43.9 at -ncmoe 16) at -ncmoe 12, needing four fewer GPU-resident layers, consistent with a file 4.6 GB lighter.

Ubatch: the most under-used flag, with one big asterisk

Depth sweeps

Does the synthetic curve transfer to real serving?

Twice now, yes. One turn of qwen3.8-32k through Open WebUI with web search (prompt_n 15065, cache_n 11198, so roughly 11K–26K of depth) measured 798.4 tok/s prompt and 27.8 tok/s generation — both landing between the d0 and d32K rows exactly where the sweep predicts. Separately, switching the 64K preset from q4_0 to q8_0 KV was predicted at 29.39 tok/s and measured 29.15 at slightly greater depth: within 0.8 %. llama-bench is not flattering itself here.

Recall is fine. Prefill is the bill.

The common warning is that long-context recall degrades well before the nominal window — "lost in the middle". On these models, inside their native ranges, it does not happen. Two probes: needle-test.py plants distinct verbatim facts at 5/25/50/75/95 % depth; semantic-recall-test.py is the hard one — it plants real helper functions among hundreds of similar utilities, then poses tasks describing what each helper does without naming it, with a deliberate near-miss distractor for each, placed closer to the question than its target so proximity cannot be the cue.

The real cost is prefill, not forgetting

Prefill throughput nearly halves with depth — 552 tok/s at 38K down to 322 at 241K — so cost grows superlinearly. 119K → 241K is 2.02× the tokens but 2.8× the time.

Thinking models will eat your entire token budget

Qwen3.8-27B defaults reasoning_effort to xhigh. At max_tokens: 1200 on one hard prompt, every setting returned zero content — and raising the budget does not converge, it scales with whatever you give it.

Nothing in reasoning_effort puts a ceiling on it. The fix is llama.cpp's own flag, not a template kwarg — so every preset here sets --reasoning-budget 1024. And note the second-order trap: that budget bounds the thinking but does not stop it being billed against the client's max_tokens, and the answer is written after the reasoning. A cap that is comfortable for a non-thinking model silently truncates a thinking one mid-sentence, with truncated = 0 in the server log and no error anywhere. Rule: client max_tokens > reasoning-budget + the answer you actually want.

Executed, not read

code-quality-test.py scores a model by running its output. Seven tasks, each a from-scratch reimplementation of a stdlib behaviour — fnmatch, shlex.split, urljoin, textwrap.wrap, csv.reader, parse_qsl, date arithmetic — with the model's function compared against the real one over edge cases plus a seeded random batch. 50 checks total.

Two things this table says, and one it turned out not to

• The 80B coding specialist does not lead. Qwen3-Coder-Next scores below both Qwen3.8-27B v3 quants that ran cleanly, and it is also the slowest generator of the five at 24.69 tok/s against 29–30 for the dense 27Bs. No axis measured here shows it ahead.
• Gemma 4-E2B reads as simply too small. 17/50 at full BF16 precision, with two of seven tasks emitting code that did not even parse. Quantisation is ruled out as the explanation by construction.
• Laguna's 54 % is the model, not the quant — and the way that was established is the interesting part. Re-run at Q8_0 it scores 27/50 again, but the totals match while the distribution does not: twenty of the fifty checks changed hands, ten each way. The SyntaxError at Q4_K_M was a quantisation artifact (add_months goes 0/6 → 6/6), while csv_rows went 7/7 → 0/7 at the higher quant. No single task's score should be quoted on its own.

Where the models actually differ: one chart, three outcomes

Asked conversationally for an SVG/HTML bar chart with axis, labels and title:

The confounds were eliminated one at a time: quantisation (it is UD-Q4_K_XL, not a Q3), reasoning cap (identical at 1024 and 4096), temperature (still wrong at 0.25), and luck (four regenerations, all four bars negative). The failure is specific to SVG coordinate maths, not to charting — both generalists produced correct scaffolding, labels, axes and colours and fell over only on mapping a value to a y origin and a positive height. If you want a chart out of a non-coding model, ask for CSS bars; it routes around the part they get wrong. Honest caveat: the failure reproduced reliably inside a long chat carrying web-search history but did not reproduce in isolated single-turn API calls.

The other failure class, and the prompt that fixes it

Asked for the same chart using Recharts with no CDN URLs supplied, every model produced a blank page — but for different reasons, and only one reason is the model's fault. The React and Recharts code itself was correct in every case; they failed on dependency URLs. That is a recall failure, and it is the opposite of the SVG case: no amount of thinking recovers a package version you never saw, whereas pinning the URLs in the prompt fixes it completely.

Adding a verification phase — fetch each script URL and confirm HTTP 200 before writing it; do not guess filenames or assume a .min build exists; read the library's own UMD docs for peer dependencies — produced the first unambiguous success. Gemma 4 loaded React, ReactDOM, Babel, @mui/material and both @emotion packages — MUI v5's styling engine, the equivalent of the prop-types trap nobody found on Recharts. All six URLs were independently re-checked and all six genuinely return 200. Make the model verify; do not make it guess.

Note also that near-lossless precision on a smaller model lost decisively to 4-bit on bigger ones: Qwen3-Coder-30B at Q8_0 emitted a single script tag, called ReactDOM.render (React 17) on globals that were never loaded, and claimed data was sorted when it was not — failing on knowledge questions the Q4 models got right, at 15.9 tok/s at 32K against Coder-Next's ~24. Q4_K_M is not what limits coding here.

KL-divergence against BF16

The coding suite resolves 4 checks in 50 across a 68–76 % band. That is too coarse to rank quant tiers, and it produced an ordering that inverted under measurement. KLD is the right instrument for this specific question: how far has a quantised model's output distribution moved from the unquantised one, per token, with error bars — no task design, no scoring rubric, nothing to saturate.

Method. Reference is unsloth/Qwen3.8-27B-GGUF BF16, 54.66 GB across two shards, sha256-verified against the HF LFS oids. Corpus is wikitext-2 wiki.test.raw, the llama.cpp convention, so these are comparable to published figures. -c 512 --chunks 200 = 102,400 tokens prefilled, 51,000 scored. The 25.33 GB base logits file was generated CPU-only in 16 m 49 s — BF16 does not fit in 16 GB, and running it on CPU also kept the router usable throughout.

The instrument was validated three ways

• Self-KLD is zero. A model scored against its own base gives KLD 4×10⁻⁵ and same-top-1 100.000 % — float noise, as it must be.
• Backend is not a confound. The base is computed on CPU while quants are scored on GPU, so some signal could have been CPU-vs-ROCm arithmetic. Scoring BF16 itself through the GPU path against the CPU base gives KLD 0.00000, worst single token 0.000068, same-top-1 100.000 % — the worst case is 30× smaller than Q6_K's mean.
• Mismatched runs cannot be compared by accident. The scorer reads -c and --chunks back out of the base file header and refuses to run on a mismatch.

IQ4_XS is a strict upgrade over Q3_K_XL at 32K

Better on both axes at once — 32 % lower divergence and 1.9 % faster, despite a file 1.06 GB larger. The reason the bigger file is faster is already in the record: Dynamic 3.0 spends fourteen distinct ggml quant types against v2's five, and that was measured costing 3.6 % of generation. IQ4_XS has fewer dequant paths, so UD-Q3_K_XL-v3 pays a dequant tax for a quant that is also less faithful than the one that does not pay it. The cost is headroom, not speed.

What Q6_K costs to actually serve

Q6_K's weights are 20,965 MiB against 16,304 MiB of VRAM, so unlike everything else here it cannot run fully offloaded — -ngl becomes the variable and the remainder sits in DDR4. At -ngl 42, 32K, f16 KV:

The trade is 7× generation and 2.5× prefill for 13× lower divergence. Three runs returned 4.25 / 4.24 / 4.24 — a measurement, not a sample. At 4.24 tok/s a 700-token answer takes 165 s, which puts Q6_K in batch-and-come-back territory rather than interactive or agentic use. That is the honest shape of "highest quality practically" on this card: the quality is available, but not at a speed that survives an agent loop.

TurboQuant KV: tested and rejected, and only KLD shows why

TurboQuant's Walsh–Hadamard-rotated KV types compress the cache far past q4_0 and really do change what fits — UD-IQ4_XS runs at 131,072 with turbo2 where q4_0 cannot allocate its compute buffers, and it passes 5/5 needle and 5/5 semantic at ~121K. It is still the wrong trade:

Freshness is measurable, and it did not separate the labs

cdn-freshness-test.py asks for a self-contained browser page, regexes every src/href, sends a HEAD to each URL and counts what resolves. No judgement calls. Three prompts (Recharts, MUI, Chart.js + D3), three runs each, temperature 0.0.

Native tool calls work — and llama.cpp broke them once per model

Prompt-based tool schemes inject descriptions into the prompt and parse text back, so they succeed even when the API path is broken. To test the real path you need Native function calling and a tool whose output cannot be guessed:

def get_probe_token(self, seed: int) -> str:
    """
    Return the secret probe token for a given seed. There is no way to
    derive this value without calling this function.
    :param seed: Integer seed for the token.
    """
    return f"PROBE-{random.Random(seed).randint(100000, 999999)}"

Expected: seed 42 → PROBE-770487, seed 7 → PROBE-439563, seed 1234 → PROBE-915965. A model answering without calling cannot produce these.

Three things that will waste your day

Pick test values that cannot be memorised. An earlier run of that test used sha256("hello world") and the 5000th prime, and "passed" without executing anything because all three are in training data. The tell was formatting: the sum came back as 4,543,965,37, comma-grouped wrong — what recalled digits look like rather than a formatted integer.

Flags that matter, and one that is a trap

KV cache type — match them, never mix

Mixed precision costs ~23 % prompt throughput for no memory benefit. q8_0/q8_0 is the right default: same speed as f16, half the memory.

q4_0 KV is not free, and the cost scales with depth

The table above invites assuming q4_0 is as cheap again. It is not. Qwen3.8-27B, -ub 1024, tg64, matched depth, nothing else varied:

Zero at depth 0 — there is no cache to read, so the quant cannot matter — and 23 % by 32K. The penalty is per-token dequantisation proportional to cache size, so it grows exactly where you reached for q4_0 in the first place. Treat q4_0 KV as the price of admission, never an optimisation. Where both fit, q8_0 wins on speed and on precision.

One negative result worth recording: q4_0 is widely assumed to degrade output, and a truncated response made that the obvious suspect. Tested directly at ~13,250-token depth, four samples per quant, asking for five titled sections plus a literal end marker — no early stopping and no structural loss from q4_0; if anything it ran long. The real cause of the truncation was the client's max_tokens. That says nothing about factual accuracy, and nothing about 100K+ depth where quantisation error has far more attention steps to compound through.

Threads and offload

The MTP head llama.cpp throws away by default

Qwen3.8-27B's block_count is 65, not 64. blk.64 is a Multi-Token-Prediction head, and llama.cpp discards it by default — model has unused tensor blk.64.* -- ignoring. That message is the default-off path, not a missing feature: --spec-type draft-mtp turns it on, and it is present in both the v2 and v3 files, so this is testable on something you already have.

That beats Muse Glimmer's DFlash drafter (1.64×). The cost is ~2,212 MiB, fixed — not the 198.8 MiB of blk.64 weights, because common_speculative_init_result builds a second full context. Measured identical at 8K and 16K and unchanged by batch size, so it does not scale. On a 16 GB card holding 12.5 GB of weights that caps MTP at 16K. If you cannot live inside 16K, MTP is not for you; the 2.2 GB is not negotiable.

Everything here is reproducible

Model folders under benchmarks/ are generated, not hand-assembled — a result you cannot regenerate is a claim, not a measurement. Every output file carries a header naming the model, context, flags, llama.cpp build and timestamp, so it is interpretable on its own.

Rebuilding the ROCm backend

cd ~/llama.cpp
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
  cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1201 -DCMAKE_BUILD_TYPE=Release
HIP_PATH=/opt/rocm cmake --build build -j16

HIP_PATH is needed at both configure and build time. Every guide shows it only at configure, which gets you a clean cmake run and then fails every .cu file with a misleading fatal error: 'hip/hip_fp16.h' file not found. Two more traps, each of which costs a full build cycle: HIPCXX must point at clang, not clang++ (auto-detection picks clang++, compiles in CUDA language mode, and dies with unsupported CUDA gpu architecture: gfx1201); and a stale CMakeCache.txt survives git pull. Back up the working binary first — a failed build leaves you with no server at all.

Serving

nohup ~/llama.cpp/build/bin/llama-server -m models/<file>.gguf -ngl 99 [-ncmoe N] \
  [-ub 1024 -b 2048] [-fa on -ctk q8_0 -ctv q8_0] -c <context> -np 1 \
  --host 0.0.0.0 --port 8090 > /tmp/llama-server.log 2>&1 < /dev/null &
disown

Switching models — router mode is the default worth using

~/llama.cpp/switch-model.sh router              # serve ALL presets, switch from the client
~/llama.cpp/switch-model.sh <model> <context>   # pin one
~/llama.cpp/switch-model.sh stop                # SIGTERM, then SIGKILL -- frees the GPU
~/llama.cpp/switch-model.sh status              # what's running now
~/llama.cpp/switch-model.sh list                # verified combinations

router starts one process over every preset in models-preset.ini. Picking a model from the client's dropdown loads it on demand and unloads the previous one, and each preset carries its full verified config rather than you retyping flags. It costs exactly one thing: the router spawns children from /proc/self/exe, so every model runs on the binary the router was launched with — there is no per-model backend in this mode. Run with --models-max 1; a second resident model will not allocate on this card.

Getting the GPU back without stopping the server

# launches pass --sleep-idle-seconds 900
# after 15 idle minutes llama-server calls destroy() and releases the model
# measured: 12,464 MiB -> 595 MiB, process still listening on 8090

The next request calls load_model() and reloads it, so nothing breaks — it just pays a cold start. Sleep state is not reported by any endpoint: /health, /props and /v1/models all deliberately bypass sleep, so none of them can tell you it is asleep, and polling them never triggers a reload.

Running the probe suite

./run-suite.sh qwen3.8-27B-UD-IQ3_XXS-v3 Qwen3.8-27B-UD-IQ3_XXS-v3.gguf 196608 \
    "-ub 512 -b 2048 -fa on -ctk q4_0 -ctv q4_0"

benchmarks/needle-test.py           --depth 119000 --no-think
benchmarks/semantic-recall-test.py  --depth 20000  --no-think
benchmarks/kld-test.sh score        # refuses to run on a -c/--chunks mismatch
benchmarks/fit.sh                   # VRAM sweep, with the GTT spill guard

Reading GGUF metadata before downloading anything large

~/llama.cpp/build/bin/llama-server -m models/<file>.gguf -c 512 --no-warmup 2>&1 \
  | grep -iE "block_count|full_attention_interval|head_count_kv|key_length|context_length"

Attention topology decides long-context viability, so check it first — parameter count matters far less. A related warning: Unsloth re-quantised Qwen3.8-27B-GGUF in place on 2026-08-19/20 under the same repo and the same filenames. A file pulled on 08-16 is Dynamic 2.0 and nothing in its name says so. The only reliable check is the byte size or the LFS oid — and the filename does not identify the build: the same UD-Q3_K_XL name reports Q4_K - Small at 3.93 BPW in v2 and Q3_K - Large at 3.80 BPW in v3.

Can they actually do a job?

Throughput, VRAM and recall say nothing about whether a model can research live data, build something and find its own bugs. The task: research the top 10 open-weight coding LLMs from a published benchmark, then build a React + MUI dashboard that renders them — with a verification phase requiring every CDN URL to be fetched and the browser console read before declaring success.

aider wins decisively on raw compatibility — it parses plain-text diffs and plain-text shell suggestions instead of native tool calls, so nothing is disqualified by a protocol quirk, including the two presets blocked under both other harnesses. But getting a real task through it took five structural fixes a human driving it interactively would never hit, because a human naturally supplies the "now run it" nudge and notices when the model says it cannot run shell commands.

Claude Code: 7 of 11 presets, and both failures are server-side

llama.cpp serves the Anthropic Messages API natively, so no translation proxy is involved. What does block it: Claude Code sends a system-role message after the user message, and the dense-Qwen family's Jinja template raises System message must be at the beginning on that shape. That rules out qwen3.8-27B — the model that scores best on code-quality in this whole record. Nothing to fix client-side; it is the template. GPT-OSS fails differently and earlier, rejecting the 30 built-in schemas at grammar compilation.

Time-to-first-token is 96–99 % of a fresh-session turn. Generation is a rounding error, so the throughput tables above predict almost nothing about how an agent feels. In a continued session it is far better because the prefix caches — Laguna goes 25.5 s → 3.1 s on turn two, Qwen3.6 58.6 s → 31.9 s, and a byte-identical repeat measured 65.3 s → 0.1 s.

That produces a genuinely awkward trade: the quality leader is disqualified by its chat template and the speed leader is the weakest model. Laguna scored 27/50 on coding and last on library knowledge, and it shows — asked to reply with an exact token it declined and explained what it was designed for instead. Correct tool calls, unreliable instruction-following. But 3.1 s per follow-up against 31.9 s is a tenfold difference in felt latency.

When the feedback loop gets gamed

The "feedback beats capability" claim has a boundary, and it is worth stating plainly. Nemotron-3-Nano — the model that wins on throughput, KV cost, native context and recall, and which showed the best research instinct of any model on the task by checking for a pre-aggregated endpoint first — fabricated a full dataset of pre-2024 models with real citation URLs attached to invented scores after its real data source failed to resolve, and separately falsified a "clean" verification report on two occasions, claiming files existed and checks passed when they had not. It was deleted from disk for that, not for anything in its benchmark numbers.

A feedback loop only works if the model's report of what it observed is honest. Verifying that the report is real — not just that a check exists — is the harder problem this record leaves open. It is the same shape as the chart that rendered perfectly and was titled "2024 Comparison" from stale search results: nothing malfunctions and the output is still wrong.

What to actually run

Six things that generalise past this box

• Attention topology beats parameter count for long context. Read full_attention_interval, head_count_kv and block_count before downloading anything large. A 27B dense hybrid can cost 2.7× the KV of an 80B, and a conventional full-attention coder needs ~52 GB of cache at 1M.
• A probe that saturates is measuring nothing. Needle recall at 32K, the first version of the coding suite, and both retrieval tests against TurboQuant all returned perfect scores while real differences existed. Build the instrument until it resolves, then say what it cannot see.
• Match the instrument's resolution to the question. A 50-check coding score ranked quant tiers backwards; KLD ordered them monotonically at 20–80 σ. Knowledge probes catch quant damage that coding probes miss entirely.
• "It loads" is not "it fits." Buffers migrate silently to host memory and a naive peak-VRAM reading reports the migration as a comfortable fit. Sample GTT, and treat anything under ~300 MiB free as fragile.
• Check the harness before blaming the model. Tool calling was broken by llama.cpp's per-model derived grammar, not by any model. Retrying cannot help — the model regenerates the same valid-but-rejected output.
• Give the model a way to check its work, then check that it checked. A verification phase turns a silent 404 into something the model has to notice. It does not, on its own, stop a model from reporting a check it never ran.

What these numbers do not say

• Recall probes are saturated at 32K. Every quant scores 5/5. They confirm nothing broke; they cannot rank models. Only runs near the ceiling discriminate — and even those were blind to a 7.9× KLD gap.
• Coding scores are deterministic but narrow. temperature 0.0 is greedy and re-runs reproduce check-for-check, so these are measurements rather than samples. But the spread across the top three is 4 checks in 50, and all three fail on the same near-floor tasks.
• KLD cannot compare different models, only a quant against its own unquantised reference. It is measured on wikitext, which is prose; a code corpus could shift the ordering. And low divergence from BF16 means faithful to the original model, not good.
• Long-context results are single-hop and n=5 per depth, with clean planted signal unlike real code. That rules out systematic collapse, not subtle degradation. Nothing covers the beyond-native 512K/1M configs, where YaRN and q4_0 KV both plausibly hurt.
• The output-quality section is one task, one client, three models. It is not a coding benchmark. Read it as: route code at the coder, and do not assume a model that wins on throughput and recall wins on code.
• Desktop VRAM drifts — the baseline moved between 263 and 442 MiB, and across 306 MiB within a single session. Older Free columns measure the machine's state as much as the model's. Newer rows report model-attributable VRAM against a stated baseline.
• code-quality-test.py executes model-written code. It runs in a temp directory under a wall-clock timeout, which is containment, not a sandbox.
• Beyond-native context rows stack a second RoPE extension on models that are already stretched. GPT-OSS-20B's 131,072 is itself a 32× YaRN stretch from a 4,096 base. Those are the least trustworthy rows anywhere in this record.