Back to Glossary
Artificial Intelligence

Model Quantization

Quantization reduces model precision (FP32→FP16→INT8) to decrease memory and increase speed.

Definition

Model quantization is a technique that reduces the numerical precision of AI model weights and activations to decrease memory usage and increase inference speed. Common quantization levels: FP32 (32-bit float, baseline), FP16 (16-bit, 2x memory saving), BF16 (16-bit brain float), INT8 (8-bit integer, 4x memory saving), INT4 (4-bit, 8x saving). Techniques include: (1) Post-Training Quantization (PTQ) — quantize after training, (2) Quantization-Aware Training (QAT) — train with quantization, (3) GPTQ — quantization for LLMs, (4) AWQ — Activation-aware Weight Quantization.

Related Keywords

quantizationmodel quantizationint8 quantizationgptqawqllm quantization

Related Terms

AI Inference
AI inference is the process of using a trained model to make predictions on new data.
LLM Serving
LLM serving is the deployment of large language models for inference via APIs or applications.
Fine-Tuning
Fine-tuning adapts a pre-trained model to a specific task using domain-specific data.

Explore Harch Corp's GPU Cloud

Leverage model quantization in our carbon-aware GPU cloud infrastructure in Morocco.