Hardware-Aware Model Selection: Matching SLMs to Your On-Premises Compute

The Model-Hardware Mismatch Problem

Most teams select AI models the way they select software libraries: they read benchmark comparisons, pick the highest-scoring option, and deploy it. This works well enough when you run on elastic cloud infrastructure that scales to fit any model. It fails entirely on-premises, where your hardware is fixed and your budget for new GPUs competes with every other infrastructure priority.

The result is a common pattern: a team deploys a 70-billion parameter model because it scored highest on a public leaderboard, only to discover that it saturates their GPU memory, serves one request at a time, and responds with latency measured in seconds rather than milliseconds. Meanwhile, a 7-billion parameter model quantized to 4-bit would have met their accuracy requirements while serving ten concurrent users at sub-second latency on the same hardware.

Hardware-aware model selection reverses the decision process. Instead of choosing a model and then figuring out how to run it, you start with your hardware constraints and find the best model that fits within them. This approach consistently produces better outcomes for on-premises deployments because it optimizes for production performance rather than benchmark scores.

Profiling Your Hardware Constraints

Before evaluating any model, build a precise profile of your available compute. The key dimensions are GPU memory, compute throughput, memory bandwidth, and interconnect speed.

GPU memory (VRAM) is the hardest constraint. A model must fit entirely in VRAM (or be split across GPUs with appropriate overhead) to serve inference. An FP16 model requires approximately 2 bytes per parameter, so a 7B model needs roughly 14 GB. Quantization reduces this — 4-bit quantization cuts memory requirements by approximately 4x, putting a 7B model at about 3.5 GB. But you also need memory for the KV cache during inference, which grows with sequence length and batch size. For long-context applications, the KV cache can consume more memory than the model weights themselves.

Compute throughput (measured in TFLOPS) determines how fast the model generates tokens. This matters less for latency-insensitive batch processing and more for interactive applications where users wait for responses. Modern consumer GPUs can offer surprisingly good throughput for smaller models; an NVIDIA RTX 4090 delivers competitive inference speeds for models under 13B parameters.

Memory bandwidth is often the actual bottleneck for LLM inference. Token generation is a memory-bound operation — the GPU spends most of its time reading model weights from VRAM rather than computing. Higher memory bandwidth directly translates to faster token generation. This is why NVIDIA's A100 (2 TB/s bandwidth) significantly outperforms GPUs with similar TFLOPS but lower bandwidth for LLM workloads.

Multi-GPU interconnect matters if you plan to shard models across multiple GPUs. NVLink provides dramatically higher bandwidth than PCIe for inter-GPU communication. If your servers use PCIe-only multi-GPU configurations, the communication overhead of model parallelism may negate the benefit of distributing the model — you may be better off choosing a smaller model that fits on a single GPU.

A Systematic Model Evaluation Process

With your hardware profile defined, evaluate candidate models through a structured process that filters on hardware fit first and task performance second.

Step 1: Compute the memory envelope. For each GPU configuration, calculate the maximum model size you can serve at your target batch size and sequence length. Include the KV cache overhead. This gives you a hard ceiling: any model above this size is immediately disqualified unless you are willing to sacrifice batch size (and therefore throughput) or sequence length.

Step 2: Identify candidate models within the envelope. The SLM landscape is rich. For most enterprise tasks, models in the 1B to 14B parameter range offer excellent performance when properly selected. Families like Mistral, Llama, Phi, Qwen, and Gemma each offer multiple size points with different tradeoffs. Do not limit yourself to a single model family — the best 7B model for code generation may come from a different family than the best 7B model for document summarization.

Step 3: Benchmark on your tasks, not public benchmarks. Public benchmarks (MMLU, HumanEval, MT-Bench) measure general capability, not performance on your specific workload. Create an evaluation dataset from real examples of the tasks your model will handle. If your model will classify support tickets, benchmark it on a labeled sample of your actual support tickets. If it will summarize meeting notes, test it on your actual meeting transcripts. A model that scores 5 points lower on MMLU but 10 points higher on your task-specific benchmark is the better choice.

Step 4: Measure inference performance under realistic conditions. Do not benchmark with a single request on an idle GPU. Measure latency (time to first token and time to complete generation), throughput (requests per second at target batch size), and GPU utilization at your expected concurrent load. Use inference servers like vLLM, TGI, or llama.cpp that support continuous batching and paged attention — these optimizations can double or triple throughput compared to naive serving.

Quantization Strategies for Maximum Hardware Utilization

Quantization is the single most effective technique for fitting better models onto limited hardware. By reducing the precision of model weights from 16-bit floating point to 4-bit or even lower integers, you can often deploy a model that is twice as large — and therefore significantly more capable — within the same memory budget.

GPTQ and AWQ (Activation-aware Weight Quantization) are the most widely supported post-training quantization methods. Both reduce model weights to 4-bit integers with minimal accuracy loss on most tasks. AWQ tends to preserve accuracy slightly better by prioritizing the weights that matter most, based on activation patterns. Both methods are supported by vLLM and TGI, making deployment straightforward.

GGUF format (used by llama.cpp) offers granular control over quantization levels. You can choose from Q2_K through Q8_0, with each level trading memory for accuracy. For tasks where precision matters (structured data extraction, code generation), Q5 or Q6 quantization preserves most accuracy. For tasks that are more forgiving (creative writing, general Q&A), Q4 or even Q3 may be sufficient.

Always benchmark quantized models against your task-specific evaluation set, not just general benchmarks. Some tasks are more sensitive to quantization than others. Mathematical reasoning and code generation tend to degrade more quickly than natural language understanding. If a quantized model drops below your accuracy threshold, try a larger model at the same quantization level rather than increasing precision on the smaller model — you usually get more capability per VRAM byte from a larger-but-more-quantized model.

Consider mixed-precision deployment: serve a heavily quantized model for latency-sensitive interactive queries and a higher-precision version of the same model (or a larger model) for batch processing during off-peak hours. This maximizes hardware utilization across the full daily cycle.

Decision Matrix: Common Hardware Profiles and Recommended Models

While the optimal model depends on your specific tasks, some general patterns hold across common on-premises hardware configurations.

Single consumer GPU (24 GB VRAM, e.g., RTX 4090): Ideal for models up to 14B parameters at Q4 quantization, or 7B parameters at FP16. At this tier, Phi-3 (3.8B) and Llama 3 (8B) offer exceptional performance relative to their size. For code-specific tasks, CodeLlama or DeepSeek Coder in the 7B range perform well. Expect to serve 5-15 concurrent users depending on sequence length.

Single datacenter GPU (40-80 GB VRAM, e.g., A100 or H100): Opens up the 14B-34B parameter range at Q4, or 14B at FP16. Models like Mixtral 8x7B (which uses mixture-of-experts to activate only a fraction of parameters per token) are particularly efficient here. QwQ-32B and similar reasoning-focused models fit well. Concurrent user capacity reaches 30-50 for typical workloads.

Multi-GPU server (2-8 datacenter GPUs): Enables 70B+ models via tensor parallelism. At this tier, the question shifts from "what fits?" to "what is the most efficient allocation?" Consider running multiple smaller models in parallel rather than one large model — three independent 14B models on three GPUs often serve more total throughput than one 70B model sharded across the same GPUs. Reserve the large model for tasks that genuinely require its capability and route simpler requests to the smaller models.

CPU-only servers: Do not dismiss CPU inference for SLMs. Models under 3B parameters with Q4 quantization run at acceptable speeds (5-15 tokens per second) on modern server CPUs with sufficient RAM. For batch processing, document classification, or applications where latency is measured in seconds rather than milliseconds, CPU inference avoids GPU costs entirely. Use llama.cpp or ONNX Runtime for optimized CPU inference.

Continuous Re-Evaluation as Models and Hardware Evolve

Hardware-aware model selection is not a one-time decision. The SLM landscape moves fast — a new model release can shift the performance frontier significantly. Build a process for continuous re-evaluation.

Maintain your task-specific benchmark as a living dataset. Add new examples as your use cases evolve. When a new model family releases a checkpoint in your target size range, run it through your benchmark pipeline. If it outperforms your current model on your tasks at comparable or lower resource consumption, evaluate it for production promotion.

Similarly, when you acquire new hardware, revisit your model choices. A GPU upgrade may unlock a larger model that delivers meaningfully better performance. Conversely, if you decommission hardware, you may need to move to a smaller or more aggressively quantized model.

Track your models' real-world performance over time, not just benchmarks. User satisfaction, downstream task accuracy, and error rates in production are the ultimate measures. A model that benchmarks well but produces outputs that require frequent human correction is costing you more than the benchmark suggests. Hardware-aware model selection is ultimately about finding the model that delivers the most value within your physical constraints — and keeping that choice current as both the model landscape and your hardware evolve.

Featured image by Lilian Do Khac on Unsplash.