Back to Glossary
GPU Cloud
AI Inference
AI inference is the process of using a trained model to make predictions on new data.
Definition
AI inference is the second phase of the AI lifecycle, where a trained model is deployed to make predictions or generate outputs on new data. Inference workloads are different from training: they require lower latency, higher throughput, and often use reduced precision (FP16, FP8, INT8). GPU cloud providers like Harch Corp offer specialized inference instances optimized for serving large language models (LLMs), computer vision models, and recommendation engines. Key metrics for inference include tokens/second for LLMs, requests/second for APIs, and p99 latency for real-time applications.
Related Keywords
ai inferencemodel inferencellm inferenceinference gpuinference cost
Related Terms
GPU Cloud
GPU cloud is a cloud computing service that provides access to graphics processing units (GPUs) on-demand for AI, ML, and HPC workloads.
AI Training
AI training is the process of optimizing model parameters using labeled data and compute resources.
LLM Serving
LLM serving is the deployment of large language models for inference via APIs or applications.