GPU Memory Management and KV Cache Optimization for On-Premises LLM Serving

Why GPU memory is the bottleneck in on-premises LLM serving

When serving large language models on-premises, GPU memory is almost always the constraining resource. Unlike cloud environments where you can provision additional GPU instances on demand, on-premises deployments operate within a fixed memory budget. A single NVIDIA A100 provides 80 GB of HBM2e memory. A 13B parameter model in FP16 occupies roughly 26 GB just for weights, leaving 54 GB for everything else: activations, the KV cache, framework overhead, and CUDA context.

The KV cache is where this gets interesting. During autoregressive generation, the model stores key and value tensors from every attention layer for every token in the sequence. For a 13B model with 40 attention layers and a hidden dimension of 5120, each token in the KV cache consumes approximately 800 KB in FP16. A batch of 32 concurrent requests, each generating sequences of 2048 tokens, requires over 50 GB of KV cache alone. This means the KV cache frequently consumes more memory than the model weights themselves.

Poor memory management leads to two outcomes, both unacceptable in production: either you limit concurrency and throughput to avoid out-of-memory errors, or you experience unpredictable OOM crashes that bring down the entire serving process. Effective GPU memory management is what separates a research prototype from a production-grade on-premises LLM deployment.

Understanding KV cache memory dynamics

The KV cache grows dynamically as each request progresses through generation. At the start of inference, a request with a 500-token prompt requires KV cache entries for those 500 tokens across all attention layers. As the model generates each new token, the cache grows by one entry per layer. This creates a memory allocation pattern that is fundamentally different from serving traditional ML models, where memory usage is predictable and static.

The challenge is compounded by variable sequence lengths. In a typical enterprise deployment, request lengths vary enormously. A classification task might use 100 tokens total, while a document summarization request could reach 8192 tokens or more. If you pre-allocate KV cache memory for the maximum possible sequence length for every request, you waste enormous amounts of memory on short requests. If you allocate conservatively, long requests fail mid-generation.

Additionally, KV cache memory is not fungible with model weight memory. Model weights are loaded once and shared across all requests. KV cache is per-request and must be allocated and freed as requests arrive and complete. This makes KV cache management a dynamic memory allocation problem similar to heap management in operating systems, but with the added constraint that GPU memory allocation and deallocation is significantly more expensive than CPU memory operations.

Understanding these dynamics is prerequisite to choosing the right optimization strategy. The techniques below address different aspects of this problem: reducing per-token memory consumption, improving allocation efficiency, and enabling more intelligent sharing of cached computations.

Paged attention: solving memory fragmentation

The most significant advancement in KV cache management is paged attention, introduced by the vLLM project. The core insight is borrowed directly from operating system virtual memory: instead of allocating contiguous memory blocks for each request's KV cache, divide GPU memory into fixed-size pages (typically 16 tokens per page) and map each request's logical KV cache to physical pages that can be scattered anywhere in GPU memory.

Without paged attention, KV cache allocation suffers from external fragmentation. Imagine GPU memory as a long ribbon: as requests arrive and complete, they leave gaps of various sizes. A new request needing 4 GB of contiguous KV cache might fail even though 6 GB is free, because the free memory is scattered in 500 MB chunks. Paged attention eliminates this problem entirely. Any free page can be assigned to any request, regardless of physical location.

The practical impact is substantial. Paged attention reduces KV cache memory waste from internal fragmentation by up to 95% compared to naive contiguous allocation. This translates directly to higher throughput: the same GPU can serve 2 to 4 times more concurrent requests because memory is used more efficiently.

For on-premises deployments, implementing paged attention is straightforward. Frameworks like vLLM, TensorRT-LLM, and SGLang all support paged attention natively. When configuring these frameworks, the key parameters to tune are block size (number of tokens per page) and GPU memory utilization ratio (the fraction of GPU memory reserved for the KV cache versus model weights and overhead). A utilization ratio of 0.85 to 0.90 is a good starting point; higher values risk OOM errors from framework overhead or unexpected memory spikes.

KV cache compression and quantization

Even with perfect memory management, the raw size of the KV cache limits throughput. Compressing the KV cache allows you to fit more concurrent requests into the same GPU memory. The most practical approach is KV cache quantization: storing key and value tensors in lower precision formats.

Storing KV cache in FP8 or INT8 instead of FP16 halves the memory consumption with minimal impact on output quality. Empirical evaluations across multiple model families show that KV cache quantization to 8 bits produces outputs that are virtually indistinguishable from FP16 inference on most tasks. The quality degradation becomes measurable only for tasks requiring very precise numerical reasoning or when quantizing to 4 bits.

A more aggressive technique is grouped-query attention (GQA), which is an architectural feature rather than a post-hoc optimization. Models trained with GQA, such as Llama 3 and Mistral, use fewer key-value heads than query heads, reducing KV cache size by 4 to 8 times compared to standard multi-head attention. When selecting models for on-premises deployment, prioritizing GQA-enabled architectures provides a significant memory advantage that compounds with other optimizations.

For deployments requiring maximum throughput, consider sliding window attention for use cases that can tolerate it. Instead of caching KV pairs for the entire context, only the most recent N tokens are cached. This bounds KV cache memory usage regardless of sequence length, which is particularly useful for streaming or conversational workloads where older context becomes less relevant.

Dynamic memory pooling and preemption strategies

In a multi-tenant on-premises environment, GPU memory must be shared across diverse workloads with varying memory requirements. A dynamic memory pool provides a centralized allocator that manages GPU memory across all active requests and implements policies for handling memory pressure.

The memory pool should implement reservation and limit policies. Each tenant or workload class is assigned a minimum guaranteed memory allocation (reservation) and a maximum limit. Reservations ensure that high-priority workloads always have sufficient memory for basic operation. Limits prevent any single workload from monopolizing GPU memory and starving other tenants.

When memory pressure exceeds available capacity, the system must decide which requests to preempt. The two primary preemption strategies are swapping and recomputation. Swapping offloads a request's KV cache from GPU to CPU memory, preserving the cached state for later resumption. Recomputation simply discards the KV cache and reprocesses the prompt when the request is resumed. Swapping has lower resumption latency but requires sufficient CPU memory and PCIe bandwidth. Recomputation is simpler and works even under CPU memory pressure but costs additional compute.

A practical preemption policy considers request priority, the cost of preemption (proportional to KV cache size and progress through generation), and the expected time until GPU memory becomes available. Priority-based preemption with cost weighting works well: among equally prioritized requests, preempt the one whose KV cache is cheapest to restore. This minimizes the aggregate performance penalty across all affected requests.

For vLLM deployments, the --preemption-mode flag controls this behavior. The swap mode offloads KV cache to CPU memory, while recompute discards and reprocesses. Monitor preemption frequency in your serving metrics; a high preemption rate indicates that you need either more GPU memory, fewer concurrent requests, or shorter maximum sequence lengths.

Monitoring and capacity planning for KV cache

Effective GPU memory management requires continuous monitoring. The key metrics to track are: KV cache utilization (percentage of allocated KV cache pages in use), preemption rate (requests preempted per minute), memory fragmentation ratio (free pages versus largest contiguous free block, relevant for non-paged systems), and peak memory watermark (highest memory usage observed in a given window).

Export these metrics to your existing monitoring stack (Prometheus, Grafana, or equivalent). Set alerts at 85% KV cache utilization for warnings and 95% for critical alerts. The warning threshold gives your team time to investigate whether a workload change is driving increased memory pressure, while the critical threshold triggers automated responses like request queuing or graceful rejection of new requests.

For capacity planning, profile your workload's memory requirements over at least two weeks to capture weekly patterns. Calculate the P95 and P99 peak KV cache usage and provision GPU memory to handle the P99 peak with a 15% headroom margin. This headroom absorbs traffic spikes and prevents cascading failures where memory pressure triggers preemptions, which cause retries, which increase memory pressure further.

Finally, consider the total cost of GPU memory when evaluating model choices and serving configurations. A model that is 10% more accurate but requires 50% more KV cache per request may not be the right choice for your on-premises deployment. Run capacity models that map model selection, quantization choices, and concurrency targets to concrete GPU hardware requirements. This analysis often reveals that a smaller, GQA-enabled model with KV cache quantization delivers better overall system performance than a larger model on the same hardware.

Featured image by alireza edalati on Unsplash.