Back to Glossary
GPU Cloud
LLM Serving
LLM serving is the deployment of large language models for inference via APIs or applications.
Definition
LLM serving is the process of deploying large language models (LLMs) like GPT, LLaMA, or Claude for inference. Key challenges include: (1) high memory requirements (70B parameter model needs 140GB+ in FP16), (2) low latency for real-time applications, (3) high throughput for concurrent requests. Modern LLM serving frameworks like vLLM, TensorRT-LLM, and Triton Inference Server optimize inference using techniques like PagedAttention, continuous batching, and quantization. Harch Corp provides pre-configured LLM serving environments on H100/H200 GPUs.
Related Keywords
llm servingllm deploymentvllmtensorrt-llmtriton inference
Related Terms
AI Inference
AI inference is the process of using a trained model to make predictions on new data.
GPU Cloud
GPU cloud is a cloud computing service that provides access to graphics processing units (GPUs) on-demand for AI, ML, and HPC workloads.
NVIDIA H100 GPU
The NVIDIA H100 is a datacenter GPU built on the Hopper architecture, designed for AI training and inference at scale.