The Alphabet Problem

You have found the model you want to run. You head to the download page and are immediately confronted by a list of files: model-FP32.gguf, model-Q8_0.gguf, model-Q4_K_M.gguf, model-IQ3_XS.gguf. Each one is a different size. None of the names mean anything obvious. Most guides tell you to "just pick Q4_K_M" without explaining what it actually is — and if your hardware is different from the author's, that advice might be leaving real performance on the table, or costing you quality you didn't need to sacrifice.

Quantisation is not a single thing. It is a family of techniques for reducing the memory footprint of a neural network model by representing its internal numbers with less precision. Understanding what each format actually does — and why — turns a confusing alphabetical grid into a set of deliberate, hardware-matched choices. This guide will get you there.

What a Model Actually Stores

To understand quantisation, you first need to understand what a language model is storing in those gigabytes of data. At its core, a trained neural network is a very large collection of numbers called weights and biases. These numbers encode everything the model has learned — grammar, facts, reasoning patterns, the lot. For a modern large language model, you are talking about billions of such values packed into a single file.

How precisely those numbers are stored is what the format names describe. In mathematics, a real number has infinite precision. In computing, you always have to choose how many bits to use to represent each value. More bits means more precision and a more faithful representation of what the model learned during training. Fewer bits means a smaller file and faster computation, but with some information inevitably lost in the rounding. Quantisation is the engineering discipline of making that tradeoff deliberately and intelligently rather than carelessly.

FP32: The Full-Precision Baseline

FP32 stands for 32-bit floating point, and it is the standard format used during model training. Each weight is stored as a 32-bit number capable of representing values across a wide range with high precision. When researchers train a model, they almost always do so in FP32 (or in mixed-precision modes that use FP32 for the most sensitive operations), because any inaccuracies introduced during training compound across billions of weight updates and can significantly degrade the final result.

For inference — actually running the model to generate text — FP32 is usually overkill. The full precision that training requires simply is not necessary when you are reading weights and multiplying them together to produce output tokens. This is the crack that quantisation slips through: you trained at full precision, but you can often serve at lower precision without a meaningful quality drop, if you do it carefully.

FP32 files are the largest you will encounter. A seven-billion-parameter model in FP32 needs roughly 28 gigabytes of memory — more than most consumer GPU cards carry as VRAM, and more than most people want to dedicate to a single application.

BF16 and FP16: The Practical Training and Serving Formats

The first step down from FP32 is 16-bit floating point, which comes in two flavours with meaningfully different properties. FP16 (or half-precision) halves the bit count to 16, preserving a similar precision range but with less room for very large or very small values. BF16, short for Brain Float 16, is a format developed by Google for neural network workloads. It uses the same 16 bits but allocates them differently — more of the bit budget goes toward representing the exponent (the range) and less toward the mantissa (the precision). This makes BF16 better suited for deep learning, where weight values can vary enormously in magnitude but fine-grained decimal precision matters less than covering that range without overflow.

Both FP16 and BF16 cut memory use roughly in half compared to FP32. A seven-billion-parameter model in BF16 fits into approximately 14 gigabytes — within reach of high-end consumer GPUs. These formats are common for cloud inference deployments and for researchers who want high-quality local inference without aggressive quantisation. If you have the VRAM, a BF16 model will give you output that is essentially indistinguishable from FP32.

INT8: Crossing Into Integer Territory

Below FP16, you leave the floating-point world and enter integer quantisation. INT8 stores each weight as an 8-bit integer rather than a floating-point number. This requires a calibration step: the model's floating-point weights are mapped onto the 0–255 integer range (or −128 to 127 for signed INT8) using a scaling factor that is computed either statically from a calibration dataset or dynamically during inference. That scaling factor is the key to making INT8 work — it tells the runtime how to interpret each integer as an approximate version of the original floating-point value.

INT8 roughly halves the memory footprint again compared to FP16, and integer arithmetic on modern CPUs and GPUs is faster than floating-point operations, which can meaningfully accelerate throughput. The quality loss at INT8 is often negligible for most tasks, making it one of the most common choices for production inference deployments where throughput and cost matter. Libraries like bitsandbytes (popular in the HuggingFace ecosystem) made INT8 loading accessible to a wide audience of practitioners.

GGUF and the K-Quant Formats: Quantisation Gets Smarter

GGUF is not a precision level — it is a file format. Developed as the successor to the older GGML format by the llama.cpp project, GGUF is a container that can hold a quantised model along with all the metadata needed to load and run it: tokeniser vocabulary, architecture hyperparameters, and quantisation settings. When you see a .gguf file, you are looking at a self-contained model package designed to be run by llama.cpp-based runtimes, which includes Ollama, LM Studio, Jan, and many others.

Inside a GGUF file, the quantisation level is described by a naming convention. The most common patterns you will encounter are the K-quant formats, introduced to replace a simpler earlier approach. The number in the name indicates the average number of bits per weight: Q4 means roughly four bits per weight, Q5 means roughly five bits, Q8 means roughly eight bits, and so on. The suffix after the underscore describes the method used. _0 is the simplest legacy approach, while _K_S, _K_M, and _K_L refer to "K-quant Small", "Medium", and "Large" — variants of a smarter quantisation algorithm that groups weights into blocks and applies a more sophisticated scaling scheme within each block. The result is meaningfully better quality at the same bit width compared to the earlier simple quantisation methods.

Q4_K_M is often cited as the sweet spot for local inference on consumer hardware, and there is real reasoning behind that reputation. Four bits per weight gives an aggressive memory saving — a seven-billion-parameter model lands around four to five gigabytes — while the K_M algorithm is tuned to preserve quality in the layers that matter most for model output. For machines with eight to sixteen gigabytes of VRAM or unified memory, it hits a practical optimum that makes much larger models accessible that would otherwise be out of reach entirely.

Q5_K_M and Q6_K push the bit count slightly higher and recover more of the quality difference relative to FP16. These are worth considering when you have a little more memory headroom and want to reduce the quality gap, particularly for demanding tasks like long-form reasoning or code generation where subtle precision differences can accumulate.

IQ Formats: Importance-Aware Quantisation

The IQ prefix — standing for "importance quantisation" — represents a more recent development in the llama.cpp ecosystem. Rather than applying the same bit depth uniformly across all weights, IQ formats use information about which weights are most critical to model quality to allocate bits non-uniformly. Weights that have been identified as high-importance during a calibration process receive more precision; weights that can be rounded more aggressively without significantly affecting output receive fewer bits. The net result is better quality per gigabyte than a naive fixed-bit approach of the same average depth.

IQ3_XS, for example, targets a very low average bit depth — around three bits per weight — but does so with the benefit of importance weighting that makes the quality output considerably better than a straightforward Q3 format would deliver. These formats are most valuable when you are trying to run a model that would otherwise be too large for your hardware, and you need to maximise quality within a tight memory budget. The tradeoff is that IQ quantisation is slower to both generate and sometimes to decode, depending on the runtime's optimisation level for these newer methods.

How to Pick the Right Format for Your Hardware

The decision framework is simpler than the alphabet makes it appear. Start with your available memory — VRAM if you are running on a GPU, or system RAM if you are running on a CPU or Apple Silicon with unified memory. Your goal is to fit the model comfortably with room to spare for the context window and runtime overhead.

  • 24 GB VRAM or more: You can run most seven to fourteen billion parameter models in Q8_0 or even BF16. Prioritise quality over compression. For thirty-billion-parameter models, Q4_K_M or Q5_K_M becomes practical.
  • 16 GB VRAM or unified memory: Q5_K_M or Q6_K for seven-billion-parameter models. Q4_K_M for fourteen-billion-parameter models. A strong practical range where quality is well-preserved.
  • 8–12 GB VRAM: Q4_K_M is your natural home for seven-billion-parameter models. For anything larger, Q3_K_M or IQ formats let you push into territory that would otherwise be off-limits, at a measurable but often acceptable quality cost.
  • Less than 8 GB, or CPU-only inference: IQ3_XS and similar low-bit formats make it possible to run useful models at all on very constrained hardware. Expect slower responses and some quality reduction, but for many tasks the output is still genuinely useful.

One practical note: always choose a larger model at a lower quantisation level over a smaller model at a higher one, all else being equal. A fourteen-billion-parameter model in Q4_K_M will generally outperform a seven-billion-parameter model in Q8_0, even if their file sizes are similar. Quantisation reduces quality incrementally; parameter count shapes it fundamentally.

The Bigger Picture

Quantisation is one of the reasons the local AI ecosystem has become genuinely viable in just a few years. The gap between what a well-quantised model can do and what a full-precision model can do has narrowed significantly, driven by better algorithms, better calibration data, and better tooling. Formats like Q4_K_M and IQ3_XS have made it possible for people with ordinary consumer hardware to run models that would have required server-grade infrastructure not long ago.

The naming conventions will keep evolving — new quantisation schemes are being developed regularly, and the GGUF format itself continues to gain new features — but the underlying logic stays constant. Every format name is an answer to the same question: how much precision can we afford to trade away, and how cleverly can we do it? Once you understand the tradeoffs, picking the right file becomes less like navigating a cryptic alphabet and more like choosing the right tool for the job.