You picked a model. Now the download page is showing you four variants with suffixes you weren’t expecting, and nobody told you that Q4_K_M and Q5_K_S are not just a quality dial you turn up or down freely. The quantization level determines how much VRAM the model actually occupies at runtime, how fast it generates tokens on your specific card, and where the quality loss shows up in practice. Getting this wrong doesn’t always mean a crash. Sometimes it just means a slower model, or a model that quietly underperforms on the tasks you actually run it on.
What Quantization Is Actually Doing to Your Model
A full-precision model stores each weight as a 16-bit floating point number. Quantization converts those weights to lower-precision formats: Q4 uses 4 bits per weight, Q5 uses 5, Q8 uses 8. The result is a smaller file that loads into less VRAM. That much is in every guide.
What most guides skip is where the speed benefit actually comes from. Local LLM inference is memory-bandwidth-bound, not compute-bound. Your GPU isn’t bottlenecked by its ability to run calculations. It’s bottlenecked by how fast it can move weight data from VRAM into the processing cores. A Q4 model moves half the data per token compared to Q8, which translates directly to faster generation on cards where memory bandwidth is the ceiling. On most consumer GPUs, it is the ceiling. The RTX 3060 12GB is a good example: its memory bandwidth is the constraint that determines tokens per second long before its compute cores are saturated.
This is why quantization isn’t just a size optimization. It’s a throughput decision. And it’s why the relationship between Q level and speed isn’t linear in the way people expect.
What Changes Between Q4, Q5, and Q8
Q4: the default that earns its position
Q4_K_M is where most consumer hardware users land, and not by accident. A 7B model at Q4_K_M runs around 4.4GB of VRAM. A 14B model at Q4_K_M runs around 8.5GB. Those numbers leave headroom for the KV cache, which is the additional VRAM consumed by the conversation context as it grows. On a card with 8GB of VRAM, that headroom is the difference between a model that handles a long session and one that silently degrades once the context fills.
Quality loss at Q4_K_M is real but narrow. Reasoning tasks, factual recall, and general chat show minimal degradation compared to Q8. Where Q4 starts to show is in sustained precision tasks: complex multi-step math, intricate code logic, and long documents where small errors compound. For most daily use cases, you won’t notice. For tasks where you need to notice, you’ll need to test your specific workload against Q8, not trust a percentage-loss figure from a benchmark.
Q5: the tier that needs justification
Q5_K_M runs a 7B model at around 5GB of VRAM and a 14B model at around 10GB. The quality improvement over Q4_K_M is measurable but modest for general tasks. The cases where Q5 earns the extra VRAM are specific: coding tasks where Q4 is producing occasional syntax errors, reasoning chains where you’re seeing Q4 drift on step 3 or 4 of a multi-step problem, and models where the architecture is known to be sensitive to quantization (some fine-tuned models fall into this category).
Q5 is not a universal upgrade. On an 8GB card running a 14B model, the move from Q4_K_M to Q5_K_M doesn’t just cost you quality headroom, it costs you context headroom. The KV cache has less room to grow, which means the model starts offloading to RAM sooner. A model that fits comfortably in VRAM at Q4 and runs at 8-12 tokens per second can drop to 2-3 tokens per second at Q5 if the KV cache starts spilling. That is not a quality upgrade. That is a performance regression that happens to have slightly better perplexity on a benchmark you’re not running.
Q8: when precision is the point
Q8_0 on a 7B model runs around 7.7GB of VRAM. On a 14B model, you’re looking at roughly 14-15GB, which already exceeds most consumer single-GPU setups for that model size. Q8 is where the quality difference over Q4 becomes consistently measurable across task types, including the ones where Q4 mostly held up. It’s also where token generation slows noticeably, around 25-30% slower than Q4 on the same hardware due to the increased memory bandwidth demand.
The cases where Q8 makes clear sense: you’re evaluating a fine-tuned model and need a quality baseline close to FP16 behavior. You have a 16GB or 24GB card with enough headroom that the model load plus KV cache doesn’t crowd the available VRAM. You’re running a task where you’ve confirmed Q4 is degrading in a way that matters for your specific output. Outside those cases, Q8 is overhead on hardware that didn’t ask for it.
The K_M vs K_S Distinction Matters More Than Most People Realize
The Q number gets all the attention, but the suffix after it often has more practical impact than moving between Q4 and Q5. K_M uses more bits for the attention layers that contribute most to output quality, while applying lower precision to weights where the loss matters less. K_S applies more uniform quantization across the whole model. The size difference between Q4_K_M and Q4_K_S is small, typically under 200MB for a 7B model. The quality difference is consistently in K_M’s favor, particularly for longer outputs and tasks requiring sustained coherence.
If you’re choosing between Q4_K_S and Q5_K_S because you want better quality, stop. Q4_K_M almost always outperforms Q5_K_S at a smaller VRAM cost. The suffix controls where the precision budget goes. K_M spends it where output quality is most sensitive. That’s why Q4_K_M is Ollama’s default recommendation and why it shows up in nearly every practical guide as the starting point, not because Q4 is inherently good, but because K_M makes Q4 punch above what the bit count suggests.
Which Level to Use for Your Task
Coding: Q4_K_M is viable for straightforward generation, refactoring, and debugging. If you’re seeing syntax errors or logic drift on Q4, move to Q5_K_M before assuming the model itself is the problem. Q8 is worth the cost if you’re running a model specifically fine-tuned for code and want to use it at its intended precision.
Writing and chat: Q4_K_M holds up well. Creative variation, tone consistency, and general coherence are not meaningfully degraded at Q4 for most users. Q5 adds little here unless you’re working on long-form content where coherence across several thousand tokens is the specific requirement.
Reasoning and math: This is where Q4 shows its ceiling soonest. Multi-step math and structured logical inference are precision-sensitive in ways that general chat isn’t. Q5_K_M is worth the VRAM cost here if your card has the headroom. Q8 is the right call if you’re running evaluation work or building something that depends on consistent logical accuracy.
For a deeper look at how inference speed and memory bandwidth interact across these levels on specific consumer GPUs, the breakdown in the LLM inference explained post covers the numbers behind what you’re experiencing when generation slows unexpectedly.
When the Decision Gets Made for You
At 6GB of VRAM, Q4 is not a preference. It’s the ceiling for anything larger than a 7B model, and even there you’re watching the KV cache closely. A GTX 1660 6GB running Mistral 7B at Q4_K_M works. Trying to run Q5 on the same card at the same model size produces VRAM pressure that slows generation below usable speeds.
At 8GB of VRAM, Q4_K_M gives you access to 14B models with enough headroom for meaningful context lengths. Q5 on a 14B model at 8GB is a VRAM gamble that depends on how long your sessions run. At 12GB, Q5_K_M for 14B models becomes genuinely viable. The RTX 3060 12GB running Phi-4 at Q4_K_M sits at 8 to 12 tokens per second with room for the KV cache. The same card running Q5_K_M on the same model runs closer to the VRAM ceiling, which affects sustained performance on longer sessions.
At 16GB and above, Q8 for 7B models becomes practical. For 14B models at Q8, you need at minimum 16GB just for the base load, and KV cache pressure still applies.
The other factor most people skip: context length. A longer context window consumes more VRAM for the KV cache regardless of quantization level. A Q4_K_M 14B model that fits fine at 2K context can start struggling at 8K context on the same card. If you’re running long sessions, the headroom Q4 provides over Q5 is often more valuable than the quality bump Q5 offers. The details on how inference speed behaves under different VRAM conditions across consumer hardware are covered in the GPU wattage and inference performance breakdown.
This is the same math that quantization calculators automate. They read your VRAM, read the model size at each quant level, and add a KV cache estimate to tell you what fits. Understanding the components behind that output means you can sanity-check the recommendation when your actual session behavior doesn’t match what the calculator predicted.
Understanding which model to run on your card in the first place, before quantization enters the picture, is a separate decision covered in the local AI model selection guide. For readers who want to go deeper on format differences between GGUF and GPTQ, which affects how quantization is implemented and what tools can run each format, that’s covered as a companion post to this one.
