LLM Inference Optimization: What Actually Works in Production

Inference is where LLM projects die. The model works in testing.

Push it to production and latency climbs, throughput craters, and your GPU budget evaporates. A demo that answers one question in 400ms can turn into a serving system that buckles at 50 concurrent users, and the problem only worsens as traffic grows.

Across vLLM, TensorRT-LLM, and SGLang, I've spent time tuning inference stacks until they hold up under load. The techniques that actually matter fall into four categories: batching, memory management, quantization, and attention kernels.

Here's how they work and where they break.

Batching: The First Lever

Running one request at a time leaves your GPU mostly idle, waiting on the next prompt while thousands of cores do nothing. Batching fixes the idleness by packing multiple requests together, and how you batch matters enormously.

Static batching groups requests at the start of inference. It carries a fatal flaw: you wait for the slowest request in the batch to finish before processing the next one.

Picture a single 2,000-token completion in a batch of mostly 50-token replies. Everyone else sits and waits for that one straggler to drain.

Continuous batching (also called iteration-level scheduling) addresses the straggler problem. Requests enter and leave the batch dynamically, at each token generation step, so a finished sequence frees its slot immediately and a queued request drops in.

Think of it as a bus that picks up and drops off passengers at every stop rather than waiting for the whole bus to empty before the next group can board. Continuous batching is what vLLM uses by default, and it's the reason vLLM typically delivers 10-20x higher throughput than naive batching for real-world request distributions.

Memory is the tradeoff. Continuous batching requires pre-allocating enough memory for the maximum batch size you want to support.

Set max_num_seqs=256 on a workload that rarely sees more than 30 concurrent sequences, and you've reserved VRAM for 226 phantom requests, memory that could have gone toward a longer context window or more model weights.

KV Cache Management

Every token your model generates needs to remember all previous tokens in the sequence. Storing those keys and values in GPU memory is the KV cache, and for long contexts it becomes the dominant memory consumer, often dwarfing the weights themselves.

Run the numbers and the pressure is obvious. Meta's 70B Llama models in BF16 have 80 layers and 8 grouped-query key/value heads of 128 dimensions (Meta's Llama paper), so each token stores 2 × 80 × 8 × 128 × 2 bytes, about 328KB, and one 128K-token sequence needs roughly 43GB just for the KV cache.

At sixteen concurrent sequences that grows to roughly 690GB, which no single GPU holds. Context length and throughput pull against each other for exactly this reason: every extra token of context steals room you could have spent on another concurrent user.

PagedAttention (vLLM's approach) manages the KV cache the way an operating system manages RAM, mapping non-contiguous physical blocks to logical sequence positions instead of demanding one long contiguous slab per request. Fragmentation disappears, and memory utilization gets close to perfect.

PagedAttention typically lets you serve 2-4x more concurrent sequences than naively managed KV caches on the same memory budget.

KV cache eviction becomes critical once memory is full. LRU is the default, though it ignores the fact that some keys get attended to far more often than others.

Recent work on "importance-based" eviction drops tokens the model attended to least, but the operational trade-off still depends on the workload. The KV cache eviction article explains the mechanism rather than supplying benchmark evidence.

Quantization: Trading Precision for Throughput

Quantization reduces the numerical precision of weights and activations (typically from 16-bit or 32-bit floats to 8-bit or 4-bit integers). Fitting more model parameters into the same VRAM is the goal, which buys you larger batch sizes or lets you run the same model on cheaper hardware.

FP8 quantization (8-bit floating point) has native tensor-core support starting with NVIDIA's Hopper and Ada Lovelace architectures, and Ampere GPUs like the A100 have none. FP8 roughly halves the storage of the weights it quantizes compared to BF16, though total VRAM savings depend on how much of the budget the KV cache and other allocations take.

On Hopper and Ada the matrix multiplies run natively in FP8, which adds compute throughput on top of the memory savings. On older GPUs without FP8 tensor cores, the weights get dequantized before computing, so saved memory bandwidth is the main benefit.

INT8 quantization using quantization-aware training or post-training calibration pushes harder. For language tasks, it typically degrades quality by 1-3% on standard benchmarks, acceptable for many production use cases like summarization or chat.

Code generation is where the degradation bites. I've seen 5-8% drops on HumanEval with naive INT8 approaches, the kind of regression that turns a working function into one with an off-by-one bug you only catch in review.

GPTQ and AWQ are the two dominant post-training quantization methods. GPTQ takes longer to apply and produces slightly better quality at 4-bit.

AWQ runs faster and preserves more quality at 4-bit for most tasks, which is why SGLang has defaulted to AWQ for its 4-bit quantization path.

Here is the rule I follow. Reach for FP8 first when your hardware supports it, and test output quality on your own workload before you ship it.

Move to INT8 when memory is the bottleneck and you can tolerate small quality losses. Drop to 4-bit only when you're severely memory-constrained and the quality tradeoff is acceptable for the specific job in front of you.

Attention Kernels: Flash Attention and Beyond

Standard attention materializes the full attention matrix: O(n²) memory for sequence length n. A 4,096 token context produces 16M values.

Stretch that to 128K and you're at 16B values, which alone overflows the memory of most GPUs before the model has generated a single token.

Flash Attention computes attention in tiles that fit in GPU SRAM, streaming through HBM with minimal memory usage. The trick resembles reading a huge spreadsheet column by column instead of loading the entire sheet into memory at once: it processes the matrix in chunks small enough to keep on-chip, so it never has to write the full result back out.

That gets you O(n) memory instead of O(n²) and runs typically 2-4x faster for longer sequences. Any inference stack without Flash Attention leaves real performance on the table, so it's now a baseline expectation.

Flash Attention 2 reworked how attention is split across thread blocks and warps, and the FlashAttention-2 paper reports around a 2x speedup over FA1. Flash Attention 3 (available on H100s with FP8 support) pushes this further with overlapping compute and memory operations.

PagedAttention I already mentioned in the KV cache context, but it's worth noting it combines with Flash Attention. vLLM uses PA by default with FA2 as its attention kernel.

What I Actually Reach For

When I'm optimizing a new inference deployment, I work in this order:

1. Enable continuous batching if not already on: this is the single biggest win for most workloads.
2. Flash Attention 2: upgrade from standard attention if not already present.
3. FP8 quantization: if VRAM is tight, apply it first and check quality before going to INT8.
4. PagedAttention: if running long contexts or high concurrency, this is essential.
5. INT8 or 4-bit: only if the above still don't get you enough headroom.

Speculative decoding earns its place once the baseline is fast enough. It differs from test-time compute, which spends more inference work to improve or select an answer rather than producing the target distribution with fewer sequential steps.

Jumping to the advanced stuff is the common mistake, like wiring up speculative decoding on a server that hasn't even enabled continuous batching yet. It's the equivalent of bolting on a turbocharger and leaving the handbrake engaged.

The draft model adds weeks of work, and a single config flag for continuous batching would have moved throughput far more. The basics deliver more than the sophisticated techniques do.

The Hardware Angle

No software optimization compensates for the wrong hardware choice. Your inference stack matters, and so does the GPU underneath it.

For batch inference with long sequences, H100s with NVLink earn their cost, because the high bandwidth between GPUs lets tensor parallelism scale efficiently across cards. A100s hold up well at smaller scales.

A consumer GPU can suit local development while remaining too small for a large model and long context together. Check the model weights, quantization, and KV-cache budget against available memory before choosing the hardware.

Running on consumer hardware with limited VRAM flips the priority order: quantization becomes your first move rather than a refinement on top of already-sufficient memory. For workloads with repeated system prompts or shared prefixes, like a RAG app that prepends the same 3,000-token instruction block to every query, prompt caching is worth understanding as a complementary technique.

It cuts the compute cost of re-encoding tokens that don't change between requests, and it compounds well with the batching and KV cache strategies above.