LLM Inference Guide: How to Serve Large Language Models at Scale
Complete guide to LLM inference: serving frameworks (vLLM, TensorRT-LLM), optimization, cost reduction, and production deployment.
LLM Inference Challenges
Serving LLMs in production is challenging: (1) Large memory requirements — 70B model = 140GB+ in FP16, (2) Variable latency — autoregressive generation is sequential, (3) High throughput needed — thousands of concurrent users, (4) Cost management — GPUs are expensive, (5) Model updates — deploy new versions without downtime. This guide covers frameworks, optimization, and deployment strategies for production LLM serving.
Memory Requirements by Model Size
LLM memory requirements (FP16, 2 bytes per parameter): (1) 7B model: 14GB → fits on A100 40GB or L40S 48GB, (2) 13B model: 26GB → A100 40GB or H100 80GB, (3) 30B model: 60GB → H100 80GB, (4) 70B model: 140GB → 2x H100 80GB (tensor parallelism) or 1x H200 141GB, (5) 175B model: 350GB → 4-5x H100/H200 (pipeline parallelism). With quantization (INT8): halve memory. With INT4: quarter memory. Choose GPU based on model size and desired batch size.
LLM Serving Frameworks Compared
Top LLM serving frameworks: (1) vLLM — PagedAttention, high throughput, easy to use, open-source, (2) TensorRT-LLM — NVIDIA's optimized engine, fastest inference, FP8 support, (3) Triton Inference Server — production-grade, multi-model, Kubernetes-native, (4) TGI (Text Generation Inference) — HuggingFace, easy model loading, (5) SGLang — structured generation, latest research. Recommendation: vLLM for most use cases, TensorRT-LLM for maximum performance, Triton for production multi-model serving.
vLLM: The Recommended Framework
vLLM is the most popular LLM serving framework: (1) PagedAttention — manages KV cache like virtual memory, 2-4x throughput improvement, (2) Continuous batching — processes new requests without waiting for batch to complete, (3) Tensor parallelism — split model across GPUs, (4) Streaming — token-by-token output, (5) OpenAI-compatible API — drop-in replacement. Install: `pip install vllm`. Serve: `python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b`. Harch Corp provides pre-configured vLLM environments.
TensorRT-LLM: Maximum Performance
TensorRT-LLM is NVIDIA's optimized LLM inference engine: (1) 3-5x faster than vanilla PyTorch, (2) FP8 support on H100 — 2x speedup over FP16, (3) Kernel fusion — combines operations for efficiency, (4) In-flight batching — dynamic batch sizing, (5) Plugin architecture — custom kernels. Use TensorRT-LLM when: (1) You need maximum throughput, (2) You're on H100/H200 (FP8), (3) You have engineering resources to optimize. Harch Corp provides TensorRT-LLM environments with expert support.
Quantization for Inference
Quantization reduces model size and increases speed: (1) FP16 (16-bit) — baseline, 2x smaller than FP32, (2) INT8 (8-bit) — 2x smaller than FP16, 2x faster, minimal accuracy loss, (3) INT4 (4-bit) — 4x smaller than FP16, 3-4x faster, slight accuracy loss, (4) FP8 (8-bit float, H100 only) — 2x faster than FP16, no accuracy loss. Methods: GPTQ, AWQ, SmoothQuant. Use INT8 for most production deployments. Use INT4 for cost-sensitive inference on smaller models.
Throughput vs Latency Optimization
Two modes of LLM serving: (1) Throughput-optimized — maximize tokens/second, batch many requests, good for batch processing, (2) Latency-optimized — minimize time-to-first-token, smaller batches, good for interactive chat. Throughput: use large batch sizes (64+), continuous batching. Latency: use small batch sizes (1-4), speculative decoding. Harch Corp can configure your serving for either mode. Monitor: tokens/second (throughput), time-to-first-token (latency), p99 latency.
Auto-Scaling LLM Serving
Scale LLM serving based on demand: (1) Kubernetes Horizontal Pod Autoscaler — scale based on GPU utilization or request queue, (2) GPU auto-scaling — add/remove GPU instances, (3) Predictive scaling — scale before peak hours, (4) Scale-to-zero — for dev environments. Challenges: (1) Cold start time (loading model takes 30-60s), (2) GPU availability (especially spot), (3) Cost of idle capacity. Harch Corp provides managed Kubernetes with GPU auto-scaling for LLM serving.
Cost Optimization for LLM Serving
LLM serving costs: (1) GPU cost — H100 $2.80/hr, A100 $2.00/hr, (2) Throughput — 70B model on H100: ~500 tokens/sec, (3) Cost per 1M tokens — $1.50-3.00. Optimization: (1) Use spot instances for non-critical serving, (2) Quantize models (INT8 = 2x throughput), (3) Use smaller models when possible (13B instead of 70B), (4) Batch requests (continuous batching), (5) Cache common queries. Harch Corp achieves $1.50/1M tokens for LLaMA 70B — among the cheapest globally.
Production Deployment Checklist
Production LLM deployment checklist: (1) Choose model (size, open-source vs proprietary), (2) Choose framework (vLLM, TensorRT-LLM, TGI), (3) Choose GPU (H100 for 70B+, A100 for 13B-70B, L40S for <13B), (4) Quantize (INT8 for production), (5) Set up auto-scaling, (6) Implement monitoring (Prometheus + Grafana), (7) Set up load balancing, (8) Implement rate limiting, (9) Add content moderation, (10) Set up CI/CD for model updates, (11) Implement fallback (smaller model if primary fails), (12) Load test before production. Harch Corp provides end-to-end LLM deployment services.