# Making Large Language Models Lighter: A Practical Guide to Quantization and Pruning
## The Deployment Wake-Up Call
Picture this: a team spends three weeks fine-tuning a model, nails the evaluation metrics, and then tries to actually serve it. The checkpoint alone clocks in at 140GB. That single number kills almost every GPU a typical engineering team has in a rack. It forces a complete rewrite of the deployment plan and turns what should have been a launch week into a desperate scramble for hardware nobody budgeted for.
This scenario is far more common than it should be, and it’s almost always preventable. The model didn’t need to ship at full precision with every parameter intact. It needed to ship as the leanest version of itself that still gets the job done. The two techniques that make this possible — quantization and pruning — aren’t exotic or experimental. They’re mature, widely used, and dramatically underutilized by teams operating under the mistaken assumption that “making it smaller” inevitably means “making it worse.”
This walkthrough explains what each technique actually does, why skipping them costs real money and real latency, and then dives hands-on into five specific methods people are running in production right now — each with adaptable code you can start using today.
—
## What Quantization and Pruning Actually Do
These two approaches get lumped together constantly, and it’s worth separating them clearly before going further, because they solve different problems in fundamentally different ways.
**Quantization** reduces the precision of the numbers a model is built from. A weight stored as a 16-bit floating-point number — something like 0.0023847 — gets rounded and re-represented using fewer bits, typically an 8-bit integer or a 4-bit integer. The total number of parameters doesn’t change. Every weight that existed before still exists. It simply takes up less space and computes faster. The analogy is straightforward: a high-resolution photograph saved at a lower bit depth still shows every object in the frame, just with less precision in the shading.
**Pruning** removes weights or entire structures outright. A connection between two neurons, an attention head, sometimes an entire layer gets deleted because the model turns out not to need it. The parameter count itself goes down. Think of it as editing a long document by actually cutting sentences that weren’t adding anything, rather than just writing everything in a smaller font.
Both techniques shrink a model. They shrink it along different axes, and as you’ll see later, they stack cleanly on top of each other rather than competing for the same job.
—
## Why This Matters Right Now
The scale problem underneath all of this is easy to understate until you see the actual numbers. A model with 70 billion parameters stored in half-precision floating point needs roughly 140GB of video memory just to load. In practice, that means four high-end GPUs before a single request gets served — hardware that costs anywhere from $80,000 to $100,000 sitting idle before the model does anything useful.
Quantization changes that math directly. Compress the same 70-billion-parameter model to 4-bit and the memory footprint drops to roughly 35 to 40GB — small enough to fit on a single high-end workstation card instead of a small cluster. This is far from a marginal optimization. This is the difference between a model that requires a dedicated data center and one that runs on hardware a single engineer can have sitting under their desk.
This isn’t a niche concern restricted to hobbyists running models locally, either. It’s actively shaping how the largest labs ship models today. Recent releases have embraced quantization-aware training pipelines that shrink models to roughly 2 bits, enabling deployment on mobile devices. The practical gains are concrete: fewer GPUs to purchase or rent, lower latency per request since less data has to traverse memory, and the ability to put real capability on hardware that was never going to hold a full-size model in the first place.
—
## What Happens If You Skip This — Or Do It Badly
The flip side is worth covering honestly, because both directions of failure show up constantly in practice.
Skip compression entirely, and the failure is usually straightforward and expensive: a model too large to deploy on the hardware you actually have, an inference bill that makes the product commercially unviable, or latency high enough to break any use case that demands a fast response — a live chat interface, a voice assistant, an autocomplete tool. None of this is hypothetical. It’s the default outcome for any team that trains a large model and assumes serving will be someone else’s problem to solve later.
The opposite failure is quieter and more dangerous, because it doesn’t announce itself the way an out-of-memory error does. Quantize too aggressively, without a proper calibration dataset, or ignore the small number of outlier weights that carry a disproportionate amount of the model’s actual capability, and accuracy degrades in ways that don’t always show up in a quick smoke test. The only way to know which you’re dealing with is to benchmark the compressed version on tasks that resemble what it will actually be used for, not just check that it still produces grammatically correct sentences.
Prune carelessly, and the same pattern emerges. The simplest possible pruning approach fails dramatically on large language models even at relatively modest sparsity levels. Large language models turn out to be substantially harder to prune safely than the smaller networks that basic pruning was originally designed for.
The methods covered below exist specifically to sit in the middle of those two failure modes: real, meaningful compression, done carefully enough that it doesn’t quietly wreck the model you spent weeks building.
—
## The Five Methods at a Glance
Before going deep on each one, here’s the map. Three are quantization methods, two are pruning methods, and they differ meaningfully in setup requirements and what they’re optimized for.
| Method | Category | Typical Size Reduction | Retraining Needed | Best Fit |
|—|—|—|—|—|
| bitsandbytes (NF4) | Quantization | ~4x | No (supports optional fine-tuning via QLoRA) | Fast setup; the only option here that also enables fine-tuning |
| GPTQ | Quantization | ~4x | No, calibration only | Mature GPU serving; wide pre-quantized model availability |
| AWQ | Quantization | ~4x | No, calibration only | Production GPU serving; best quality-to-speed ratio on modern kernels |
| SparseGPT | Pruning | ~2x (at 50% sparsity) | No, one-shot with weight update | Large models; structured sparsity for real hardware speedups |
| Wanda | Pruning | ~2x (at 50% sparsity) | No, single forward pass | Very large models where pruning speed itself matters |
—
## Method 1: bitsandbytes (NF4 4-Bit Quantization)
This is the method most teams should reach for first, and it’s a little undersold in many guides precisely because it’s simple enough to use in a single function call. It’s built around a data type called NF4 — NormalFloat4 — designed specifically around the fact that neural network weights tend to follow a roughly normal distribution rather than being spread evenly across the number line, so the available 4-bit values are placed where the actual weights cluster instead of being spaced out uniformly.
It’s also the one method on this list that supports QLoRA, meaning you can load a model in 4-bit and still fine-tune it by training small low-rank adapter weights on top, without ever touching the frozen 4-bit base weights directly. If fine-tuning is anywhere in your plan, this is the natural starting point.
### Working Code
“`python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
model_id = “meta-llama/Llama-3.1-8B-Instruct”
# Configure 4-bit NF4 quantization with double quantization enabled
bnb_config = BitsAndBytesConfig(
load_in_4bit=True, # load weights in 4-bit instead of 16-bit
bnb_4bit_quant_type=”nf4″, # NormalFloat4: a data type tuned for
# the normal-ish distribution of NN weights
bnb_4bit_compute_dtype=torch.bfloat16, # matmuls are upcast to bfloat16 at
# compute time; weights stay stored at 4-bit
bnb_4bit_use_double_quant=True, # quantizes the quantization constants
# themselves, saving roughly another
# 0.4 bits per parameter on top
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=bnb_config,
device_map=”auto”, # spreads layers across available
# GPU(s), offloading to CPU if needed
)
inputs = tokenizer(“Explain quantization in one sentence.”, return_tensors=”pt”).to(model.device)
output = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(output[0], skip_special_tokens=True))
“`
### What Matters in This Code
The `load_in_4bit=True` flag is the switch that triggers the whole process, converting every linear layer’s weights to 4-bit on load rather than requiring a separate offline quantization pass first. That’s exactly why this is the fastest method to get running.
The `bnb_4bit_quant_type=”nf4″` setting picks the distribution-aware format over plain 4-bit integers, which is what keeps quality close to the original model instead of just rounding blindly.
The `bnb_4bit_compute_dtype=torch.bfloat16` setting matters because the weights sit in memory at 4-bit but get temporarily upcast to bfloat16 during the actual matrix multiplication. GPUs don’t have native 4-bit compute kernels for this yet, so this line controls that intermediate precision.
The `bnb_4bit_use_double_quant=True` flag is a small but genuinely free win: it quantizes the scaling constants used to quantize the weights in the first place, squeezing out a bit more memory with no meaningful accuracy cost.
—
## Method 2: GPTQ (Calibrated Post-Training Quantization)
GPTQ was one of the first 4-bit methods that actually held up well on large models, introduced in the original paper from Frantar and colleagues in 2022. The mechanism is what separates it from naive rounding: it quantizes a model layer by layer, and within each layer, it uses second-order information — an approximation of the Hessian matrix — to figure out how rounding one weight affects the ideal values of the weights around it, then adjusts the remaining unquantized weights in that layer to compensate for the error just introduced. It’s error correction built directly into the quantization process, rather than quantizing every weight independently and hoping the errors don’t compound.
That mechanism needs a calibration dataset, typically a few hundred samples of representative text, to estimate those Hessian statistics accurately.
### Working Code
“`python
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
import torch
model_id = “meta-llama/Llama-3.1-8B-Instruct”
tokenizer = AutoTokenizer.from_pretrained(model_id)
# GPTQConfig drives both calibration and quantization in a single pass
gptq_config = GPTQConfig(
bits=4, # target bit-width per weight
dataset=”c4″, # calibration text used to estimate the
# Hessian-based error compensation
tokenizer=tokenizer,
group_size=128, # weights are quantized in groups of 128,
# balancing accuracy against compression ratio
desc_act=False, # skips activation-order permutation for
# faster inference, at a small accuracy cost
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=gptq_config,
device_map=”auto”,
torch_dtype=torch.float16,
)
model.save_pretrained(“./llama-3.1-8b-gptq-int4”)
tokenizer.save_pretrained(“./llama-3.1-8b-gptq-int4”)
“`
### Walking Through the Key Settings
The `dataset=”c4″` line is doing the real work in this whole snippet: it’s what the model runs forward passes on to collect the activation statistics GPTQ needs to compute its layer-wise error compensation. Using a dataset that resembles your actual traffic tends to produce better real-world results than a generic one.
The `group_size=128` setting controls the granularity of quantization. Smaller groups mean more scaling constants stored, which costs slightly more memory, but in exchange for tighter accuracy. One hundred and twenty-eight is the community-standard middle ground.
The `desc_act=False` flag disables a reordering step that processes the most impactful weight columns first. This improves accuracy marginally but slows down both quantization and, in some serving setups, inference itself. It’s commonly turned off for GPU-serving-first setups where quantization is a one-time cost but inference speed happens on every request.
### GPTQ’s Real Limitation
A January 2026 benchmark running all four major 4-bit formats side by side on the same hardware found GPTQ trailing specifically on code generation tasks, scoring around 46% on standard code benchmarks against competing methods landing near 52%. The likely cause is that GPTQ’s column-by-column error propagation compounds more over the course of a long matrix, which hurts multi-step reasoning tasks like writing correct code more than it hurts simple next-token prediction.
GPTQ remains a solid, mature, widely supported choice, especially if you already have a GPTQ checkpoint working well. It’s just no longer the automatic first pick for a fresh setup in 2026.
—
## Method 3: AWQ (Activation-Aware Weight Quantization)
AWQ, introduced by Lin and colleagues in 2023, takes a different angle on the same underlying problem. Instead of correcting for error after the fact the way GPTQ does, it starts from an observation about which weights actually matter: by watching activations during a short calibration pass, it identifies a small percentage of “salient” weight channels — the ones that consistently produce the largest activation magnitudes and therefore have an outsized effect on the model’s output. Those salient weights get protected with a scaling trick that preserves their effective precision, while everything else gets quantized aggressively.
That targeted protection is a big part of why AWQ has become the default choice for production GPU serving, particularly for instruction-tuned models where a small number of weights carrying real semantic weight can make an outsized difference to output quality.
### Working Code
“`python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = “meta-llama/Llama-3.1-8B-Instruct”
quant_path = “llama-3.1-8b-awq”
quant_config = {
“zero_point”: True, # asymmetric quantization: shifts the zero point
# instead of forcing weights to center on zero
“q_group_size”: 128, # same grouping idea as GPTQ, 128 weights per group
“w_bit”: 4, # 4-bit weights
“version”: “GEMM”, # kernel variant tuned for batched GPU inference
}
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
# quantize() runs the calibration pass, identifies the salient weight
# channels by observing activation magnitudes, and protects them while
# aggressively quantizing everything else
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
“`
### Key Configuration Choices
`zero_point=True` allows the quantized range to shift instead of forcing it to sit symmetrically around zero. This matters because real weight distributions are rarely perfectly centered, and asymmetric quantization captures that shape more faithfully.
`q_group_size=128` plays the identical role it does in GPTQ, controlling the accuracy-versus-memory tradeoff at the group level.
`w_bit=4` is the target precision.
`version=”GEMM”` selects the kernel AWQ compiles against at inference time. GEMM is the variant built for the batched matrix multiplications that happen when a server is handling multiple concurrent requests — which is the exact scenario production serving actually looks like.
### The Numbers Behind AWQ
With the Marlin inference kernel, AWQ runs roughly 1.6 times faster than the original half-precision model while retaining approximately 92% of code generation accuracy. Worth noting honestly: without an optimized kernel behind it, AWQ can actually run slower than plain half-precision, so the format and the serving stack it runs on need to be chosen together, not separately.
—
## Method 4: SparseGPT (One-Shot Structured Pruning)
This is where the article shifts from shrinking numbers to removing weights entirely. SparseGPT, from Frantar and Alistarh’s 2023 paper, was the method that first proved large language models could be pruned aggressively without retraining, at a time when the established wisdom was that this simply didn’t work on models this size. It frames pruning as a layer-wise reconstruction problem: for each layer, it decides which weights to remove and, in that same pass, updates the surviving weights in that layer to compensate for the ones just deleted, using second-order Hessian information similar in spirit to GPTQ’s approach.
The practical detail that matters most here is the sparsity pattern. Unstructured sparsity — zeroing out whatever individual weights score lowest with no pattern to where they sit — saves memory on disk but doesn’t actually speed anything up on standard GPU hardware. The hardware still has to load every weight from memory regardless of whether it’s zero. NVIDIA’s 2:4 structured sparsity pattern, exactly two zeros in every group of four consecutive weights, is what changes that. Specialized hardware on recent GPU architectures can skip the zeroed weights during matrix multiplication entirely, delivering a real, measurable speedup rather than just a smaller file.
### Working Code
“`bash
# Clone the official SparseGPT repository
git clone
cd sparsegpt
# Run one-shot pruning with structured 2:4 sparsity
python llama.py meta-llama/Llama-3.1-8B-Instruct c4
–sparsity 0.5 # target: 50% of weights removed overall
–prunen 2 –prunem 4 # enforce a 2:4 pattern, 2 zeros in every group of 4,
# required for real Sparse Tensor Core speedups
–save llama-3.1-8b-sparsegpt-2-4
“`
The two positional arguments tell the script which model to prune and which calibration dataset to run forward passes on to estimate the Hessian statistics the pruning decisions are based on — functionally the same role calibration data plays for GPTQ. The `–sparsity 0.5` setting sets the overall target: half the weights across pruned layers get removed. The `–prunen 2 –prunem 4` flag pair is the single most important setting in this command if the goal is real inference speedup rather than just a smaller checkpoint on disk. It enforces the structured pattern that hardware can actually exploit.
Expect this to take somewhere in the range of an hour on a single H100 for a 70-billion-parameter model, considerably less for something in the 7- to 8-billion range.
—
## Method 5: Wanda (Pruning by Weights and Activations)
Wanda — short for Pruning by Weights and Activations — from Sun and colleagues’ 2023 paper, takes SparseGPT’s core insight and strips it down to something much simpler. Instead of solving a full layer-wise reconstruction problem with Hessian inversion, Wanda scores each weight using just the product of its magnitude and the L2 norm of its corresponding input activation — a metric that can be computed in a single forward pass through the model. There’s no weight update step afterwards at all; the surviving weights are simply left exactly as they were.
That simplicity translates directly into speed. Because there’s no Hessian to invert and no iterative column-by-column solving, Wanda’s own paper reports it can be roughly 300 times faster to compute than SparseGPT, and separate benchmarking on 70-billion-parameter-class models found it runs 5 to 10 times faster in wall-clock terms with roughly half the peak memory. Quality-wise, the comparison isn’t a clean win for either method across the board. SparseGPT tends to edge out Wanda on smaller models around the 7-billion mark under 2:4 structured sparsity, while Wanda holds up better on larger models, per the original paper’s own reported results.
### Working Code
“`bash
# Clone the official Wanda repository
git clone
cd wanda
# Run one-shot pruning: a single forward pass, no Hessian, no weight update
python main.py
–model meta-llama/Llama-3.1-8B-Instruct
–prune_method wanda # selects the magnitude-times-activation metric
–sparsity_ratio 0.5 # remove 50% of weights overall
–sparsity_type 2:4 # structured pattern for real GPU speedups
–save out/llama-3.1-8b-wanda-2-4
“`
The `–prune_method wanda` flag is what selects this specific scoring approach over the script’s other supported methods, including plain magnitude pruning and SparseGPT itself, since the two are often implemented side by side in the same tooling for direct comparison. The `–sparsity_ratio` and `–sparsity_type` flags mirror SparseGPT’s flags almost exactly.
The practical reason to reach for Wanda specifically over SparseGPT is when the model is large enough — or the calibration set numerous enough — that SparseGPT’s inverse Hessian computation becomes the bottleneck in your workflow rather than the pruning decision itself.
—
## Stacking Them: Pruning and Quantization Together
These five methods aren’t a menu where you pick exactly one. Pruning and quantization attack different parts of the same problem, so they combine directly, and the combined result is bigger than either technique alone. Take a 70-billion-parameter model, prune it first with SparseGPT or Wanda down to 50% structured sparsity, then quantize what’s left with AWQ or GPTQ, and a model that needed 140GB in its original half-precision form can land around 17 to 18GB — small enough to run comfortably on a single high-end consumer GPU.
The order matters, and it’s not arbitrary. Pruning first and quantizing second works because the quantization step calibrates against the model’s actual final weight distribution, including the gaps pruning already introduced. Reverse the order and quantize first, then prune, and the pruning step is now making its removal decisions based on weights that have already been rounded and distorted, compounding two sources of error against each other instead of letting the second step correct cleanly for what the first one changed.
—
## Choosing the Right Method for Your Situation
With five real options on the table, the actual decision usually comes down to what you’re optimizing for, and the comparison table from earlier maps fairly directly onto real-world choices.
If fine-tuning is anywhere in the plan — not just inference — bitsandbytes with QLoRA is the only method on this list built for that from the ground up.
If you’re serving at scale through an inference engine and raw throughput matters most, AWQ with the Marlin kernel is the current default for good reason.
If you already have a GPTQ checkpoint working reliably in production, there’s rarely a strong case to migrate purely for the sake of it, though a new project is better served starting with AWQ today.
If you’re deploying to a laptop, an edge device, or running through tools like Ollama or LM Studio, that world runs on the GGUF format rather than any of the three quantization methods detailed above, since GGUF is built specifically for efficient CPU inference. Most compressed models eventually get converted into this format for the last mile of deployment.
For pruning specifically, the choice usually comes down to model size and how much compute you’re willing to spend on the pruning pass itself. SparseGPT’s extra weight-update step tends to edge out Wanda’s quality on smaller models in the 7-billion range. Wanda’s dramatically lower compute cost makes it the more practical choice as models get larger, when SparseGPT’s Hessian computation starts to become a real bottleneck rather than a rounding error in your timeline.
—
## FAQ
**Q: Do I need a GPU to run quantized models?**
A: Not necessarily. While most quantization methods described here target GPU deployment, formats like GGUF are specifically designed for efficient CPU inference. The choice of format and deployment target should be made together based on your hardware.
**Q: How much accuracy loss should I expect from quantization?**
A: It varies significantly by model, task, and method. Some models tolerate aggressive compression with minimal degradation, while others fall apart quickly. The only reliable way to know is to benchmark the compressed version on tasks that resemble your actual use case, not just check for grammatical correctness in generated text.
**Q: Can I combine pruning and quantization on the same model?**
A: Yes, and they stack cleanly. The recommended order is to prune first and quantize second. Pruning first lets the quantization step calibrate against the model’s final weight distribution, including the structural gaps introduced by pruning. Reversing the order compounds errors from both steps.
**Q: What’s the difference between structured and unstructured sparsity?**
A: Structured sparsity follows a regular pattern — for example, exactly two zeros in every group of four consecutive weights — which hardware can exploit for actual speedups during computation. Unstructured sparsity zeros out whatever individual weights score lowest with no pattern, which saves disk space but doesn’t speed up inference because the hardware still loads every weight from memory regardless.
**Q: Is QLoRA the same as standard bitsandbytes quantization?**
A: QLoRA builds on bitsandbytes quantization by adding a fine-tuning step. While standard 4-bit loading is inference-only, QLoRA allows you to train small low-rank adapter weights on top of a frozen 4-bit base model, giving you the memory benefits of quantization during fine-tuning as well.
**Q: Why does AWQ sometimes run slower than FP16 without the right kernel?**
A: AWQ relies on specialized inference kernels — like the Marlin kernel — to achieve speedups. Without an optimized kernel compiled against the specific format, the overhead of handling 4-bit operations can actually make inference slower than running the model in native half-precision. The format and serving stack should be chosen together.
**Q: At what model size does Wanda become preferable to SparseGPT?**
A: Wanda becomes increasingly practical as models grow larger. For models around 7 billion parameters, SparseGPT’s extra weight-update step tends to produce slightly better quality. For models in the 30-billion to 70-billion range and above, SparseGPT’s Hessian computation becomes a significant bottleneck, and Wanda’s dramatically lower compute cost makes it the more practical choice.
—
## Conclusion
None of these five methods make a model worse in any meaningful sense when done properly. They make it honest. Most large models ship with more precision and more parameters than the task in front of them actually requires — carried over from training runs optimized for a different goal than the one deployment cares about. Quantization and pruning are how you find out what a model genuinely needs to keep doing its job well, and cut the rest.
Start with whichever of these five fits the constraint you’re actually up against right now — memory, latency, hardware you don’t have, or a fine-tuning step you still need to run — rather than chasing the method with the best benchmark number on a task that isn’t yours. Benchmark the result on something that resembles your real traffic before you trust it. That’s the whole discipline here, and it’s a lot more approachable than the size of these models makes it feel.
Thank you for reading
