MY LOCAL LLM WAS 20× SLOWER THAN IT SHOULD HAVE BEEN. HERE'S EVERY FIX THAT ACTUALLY MATTERED:

Cover image

I spent a 6 months optimizing local inference on a single consumer GPU. Tried every flag, every quant format, every Reddit trick. Most of them did nothing. A handful of them doubled or tripled my speed. One setting I'd overlooked for months was costing me 3× performance the entire time.

This is not a reference manual. This is the ranked list: what to fix first, what to fix second, what to skip entirely. Built from real benchmarks on real hardware. If your local model feels slow, one of these is probably why.

I post the alpha before it becomes common knowledge. If you're into local AI, self-hosting, and the kind of stuff that doesn't show up on your timeline until it's too late. Follow me: @MPxbt

The only number that matters

Before we touch anything: the metric you care about is TG (token generation speed). That's how fast the model outputs text after it's read your prompt. It's what you feel when you're sitting there waiting.

Everything in this article is measured by its impact on TG. Some optimizations affect prompt processing (PP) or time-to-first-token (TTFT) instead. I'll call those out. But if I say "this gave me +30%," I mean +30% on TG.

Here's the human perception scale:

< 5 tok/s - Painful. You'll close the tab.
5-10 tok/s - Usable. About human reading speed.
10-20 tok/s - Comfortable. Interactive chat feels normal.
20-40 tok/s - Fast. Coding agents feel snappy.
40+ tok/s - Instant. You forget it's local.

My baseline on an RTX 4070 (12GB) was sitting around 12 tok/s on a 30B MoE model. After every optimization below, the same model on the same hardware hits 40-100 tok/s depending on the technique. Same GPU. Same model. Just configuration

Tier 1: The fixes that feel like cheating

These are the optimizations where you change one setting and the number jumps so hard you think something is broken.

Fix #1: Your RAM is probably running at half speed

This is the one that cost me months.

Most motherboards ship with RAM defaulting to JEDEC base speed, a fraction of what your sticks are rated for. If you bought DDR5-6000 RAM and never touched the BIOS, it's probably running at DDR5-4800. You paid for a highway and you're driving in the school zone.

For MoE models (Qwen3, DeepSeek, gpt-oss, basically the architectures that dominate open source right now), expert weights stream from system RAM every single token. RAM bandwidth is the bottleneck. Slower RAM = proportionally slower generation.

The fix: go into BIOS, enable XMP (Intel) or EXPO (AMD). One toggle. Reboot.

My result: TG went from roughly 1/3 of expected to full speed. On a 120B MoE model, that was the difference between "barely usable" and "comfortable."

How to check without rebooting:

bashsudo dmidecode -t memory | grep -E "Speed|Configured"

If "Configured Memory Speed" is lower than your RAM's rated speed, you're leaving performance on the table. Fix this before you touch a single llama.cpp flag.

Fix #2: Speculative decoding (MTP)

This is the single biggest speed multiplier available right now, if your model supports it.

Models trained with Multi-Token Prediction heads (Gemma 4, Qwen 3.6, and more coming) ship with a small companion draft model. The draft model guesses several tokens ahead; the main model verifies them in one pass. When the guesses are good, you get multiple tokens for the compute cost of one.

Real numbers on RTX 4070 12GB:

Gemma 4 26B baseline: 38.5 tok/s
Gemma 4 26B with MTP: 100.6 tok/s (2.6× speedup)
Gemma 4 12B with MTP: 120.8 tok/s (2.0× speedup)

That's not a typo. Same model, same GPU, same quant. Just speculative decoding turned on.

The setup in llama.cpp:

llama-server \
  -m model.gguf \
  --spec-draft-model mtp-companion.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2

The catch nobody warns you about: MTP has two separate KV caches: one for the target model, one for the draft. Quantizing the target cache (which normally saves VRAM and works fine) can tank draft acceptance to nearly zero on some models. Gemma 4 needed f16 target KV to keep acceptance above 70%. Always benchmark acceptance rate, not just raw TG.

Fix #3: You're running on the wrong cores

If you have a 12th-gen+ Intel CPU, it has two types of cores: Performance cores (P-cores) and Efficiency cores (E-cores). The E-cores are 20-30% slower at matrix math. By default, the OS schedules work across all of them.

For MoE models where the CPU handles expert computation every single token, E-cores are actively dragging you down.

The fix:

# Pin to P-cores only (i5-12600K example: cores 0-11 are P-cores)
taskset -c 0-11 llama-server ...

My result: +25% TG. Instantly. Zero tradeoff.

Tier 2: The 15-20% gains that add up

Fix #4: Switch to Linux (or tune Windows properly)

Linux gives ~15-20% better inference throughput than Windows. Leaner CUDA driver overhead, better scheduling under sustained load, direct memory control.

If dual-booting isn't an option, at minimum do this on Windows:

Power plan → "Ultimate Performance"
NVIDIA Control Panel → Power management → "Prefer maximum performance"
WSL2 is acceptable but has some VRAM overhead from virtualization

On Linux, there's a sneaky trap: power-profiles-daemon (ships with KDE and GNOME) can silently throttle your CPU at the hardware level. Everything looks fine in sysfs. Governor says "performance," frequency looks right. But TG runs 20-30% below expected. And it varies between reboots.

The fix:

# Replace the broken daemon with tuned
sudo apt install tuned               # Ubuntu/Debian
sudo systemctl enable --now tuned
sudo tuned-adm profile throughput-performance

Fix #5: Let llama.cpp figure out layer placement

For MoE models that don't fully fit in VRAM, the question is: which layers go on GPU, which go on CPU? Getting this wrong leaves either VRAM unused or causes out-of-memory crashes mid-session.

Don't manual-tune this. Use --fit on:

llama-server \
  -m model.gguf \
  --fit on \
  --fit-ctx 65536 \
  --fit-target 512

This probes your free VRAM at startup and computes optimal placement automatically. The --fit-target 512 leaves 512MB headroom. Important because CUDA memory grows as context fills, and a tight fit that works on a short test will OOM on a real conversation.

For a stable production config, run llama-fit-params once to get the exact flags, then hardcode them.

Fix #6: Quantize your KV cache

The KV cache stores attention state for all context tokens. At 64k context with f16 precision, it eats ~4GB of VRAM. On a 12GB card, that's a third of your total budget.

Switching to q8_0 cuts that in half with effectively zero quality loss:

-ctk q8_0 -ctv q8_0

The freed VRAM lets --fit keep 1-2 extra layers on GPU, which translates directly to higher TG. On Qwen3-Coder-Next this gave me +2 tok/s. Not from the cache itself being faster, but from the extra GPU layers the freed memory enabled.

Fix #7: Go headless

If your inference box is a dedicated server (not your daily driver), kill the desktop environment:

sudo systemctl isolate multi-user.target

This frees 200-400MB of RAM plus whatever VRAM the compositor was using. Use zellij in a TTY for terminal splitting.

To get your desktop back: sudo systemctl isolate graphical.target

Fix #8: Route display through iGPU

If your motherboard has HDMI/DisplayPort output (Intel iGPU), plug your monitor there instead of the GPU. The discrete GPU stops reserving 500-1000MB for desktop composition. That's VRAM you can now use for model weights.

Tier 3: The details that matter at scale

Fix #9: Build llama.cpp from source

Distro packages are outdated. Ollama uses llama.cpp internally but hides every tuning knob. MoE inference kernels improve measurably with each release.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
  -DGGML_NATIVE=ON -DGGML_LTO=ON \
  -DGGML_CUDA_GRAPHS=ON -DGGML_CUDA_FA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=89      # RTX 40-series; check yours
cmake --build . --config Release --parallel

Rebuild regularly. A single upstream commit can add tok/s on your model.

Fix #10: Use --no-mmap and --mlock

Without --no-mmap, model weights load lazily via memory-mapped I/O. Every cold page triggers a page fault. For MoE models with non-sequential expert access patterns, this adds latency jitter to every token.

--no-mmap loads the entire model into RAM upfront. Slower startup, stable generation.

--mlock prevents the OS from swapping those pages out under memory pressure. Essential on systems with high vm.swappiness (many distros default to 60+).

llama-server -m model.gguf --no-mmap --mlock ...

Fix #11: Flash attention. Just turn it on

--flash-attn on

Reduces attention memory traffic, avoids materializing the full attention matrix. Required for stable long-context inference. No meaningful downside on CUDA. Always enable.

Fix #12: Set --parallel 1 for single-user setups

Each parallel slot maintains its own KV cache. --parallel 4 means 4× the KV VRAM. If you're the only user, that's wasted memory that could hold model layers.

On gpt-oss-120b, dropping from --parallel 4 to --parallel 1 freed ~540MB VRAM, enough for an extra GPU layer and +1 tok/s.

The quantization decision tree

This is where people burn hours reading Reddit threads. Here's the shortcut:

Use the highest quant that fits. That's the rule. Everything else is details.

Can your setup handle Q5_K_XL / UD-Q5_K_XL? → Use it

Too big? → Drop to Q4_K_M or UD-Q4_K_XL

Model offers QAT Q4? → Try QAT first. It's trained for 4-bit

Still too big? → IQ3 variants, but test on your workload

QAT (Quantization-Aware Training) is the game changer at low bit-widths. Standard post-training quantization just rounds weights and hopes for the best. QAT trains the model with rounding noise baked in, so the weights learn to compensate. A QAT Q4 model can behave closer to Q8 than to a regular Q4.

Unsloth Dynamic quants keep sensitive tensors at higher precision and push easy tensors lower. Often beat uniform quants at the same average size.

The trap: a quant that wins on perplexity benchmarks can still fail on your actual workload. Wiki-based perplexity, coding benchmarks, and "does it stay coherent at 64k context" are three different tests. A quant only "wins" if it survives the job you're actually running.

What to skip

I've tested these so you don't have to waste the afternoon:

--poll tuning: flat across all values on hybrid CPU+GPU inference. GPU kernel execution dominates. Leave at default.

--numa on single-socket systems: does nothing. There's only one memory node. Use taskset for affinity.

GGML_CUDA_FORCE_CUBLAS: forces cuBLAS over llama.cpp's native kernels. Tested on consumer cards: slower. cuBLAS is optimized for datacenter batch sizes, not single-user decode. Skip.

Obsessing over --ubatch-size: the default 512 is fine for most setups. A sweep might find a few percent on your specific prompt shape, but run it last, not first. Start with 512, and only sweep if you've exhausted everything above.

n-gram speculative decoding without benchmarking: promising for repetitive code sessions, but acceptance rates vary wildly. Don't enable it blindly. Measure first, keep it only if end-to-end loop time improves.

The diagnostic cheat sheet

Run this before you start optimizing. Half the time, the problem is one of these:

# Is your RAM at full speed?
sudo dmidecode -t memory | grep -E "Speed|Configured"

# Is the CPU governor set to performance?
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# Is EPP also set to performance? (governor alone isn't enough)
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference

# Are P-cores near max boost?
grep "cpu MHz" /proc/cpuinfo | sort -rn | head -6

# How much VRAM is free?
nvidia-smi | grep MiB

# Is anything swapping?
cat /proc/vmstat | grep -E "pswpin|pswpout"

# What's eating CPU?
ps aux --sort=-%cpu | head -10

# What power profile is active?
sudo tuned-adm active

If any of these are wrong, fix them before touching llama.cpp flags. I've seen people spend days tuning layer placement when their RAM was running at JEDEC base speed the whole time.

The complete launch command

Here's what a fully optimized single-user setup looks like on a 12GB card with a MoE model:

taskset -c 0-11 llama-server \
  -m model.gguf \
  --fit on --fit-ctx 65536 --fit-target 512 \
  -ctk q8_0 -ctv q8_0 \
  --flash-attn on \
  --batch-size 1024 --ubatch-size 512 \
  --threads 10 --threads-batch 12 \
  --parallel 1 \
  --no-mmap --mlock \
  --prio 2 \
  --no-warmup

Add speculative decoding if the model supports MTP:

--spec-draft-model mtp-companion.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2

That's it. Every optimization from this article, one command.

The honest summary

Here's the full list, ranked by impact, with real numbers from my RTX 4070 12GB:

#   Fix                              Impact on TG
────────────────────────────────────────────────────────────────────────
1   Enable XMP/EXPO in BIOS          up to 2-3× (if it was off)
2   MTP speculative decoding         2.0-2.6×
3   QAT quants instead of PTQ        major quality at same VRAM
4   Switch to Linux                  ~15-20%
5   Fix power-profiles-daemon        eliminates 20-30% random drops
6   Pin to P-cores (taskset)         +20-30% on Intel hybrid
7   --fit on (auto layer placement)  major TG vs bad manual placement
8   KV cache q8_0                    frees VRAM → more GPU layers
9   Build llama.cpp from source      cumulative per release
10  --no-mmap + --mlock              eliminates TG jitter
11  --flash-attn on                  required for long context
12  --parallel 1                     frees KV VRAM
13  Headless mode                    frees 200-400MB RAM + VRAM
14  iGPU for display                 frees 500-1000MB VRAM

Fix #1 and #2 together can take you from 12 tok/s to 100 tok/s on the same hardware. Everything else is incremental. But the increments add up. And on a box you're running 24/7, every tok/s compounds into hours saved.

Start at the top. Work down. Stop when you're happy.

Like and share with your friends if you found this helpful. I dig into local AI setups, self-hosting, and the stuff that actually moves the needle before it hits the mainstream. The alpha is here.