Back to Glossary
Artificial Intelligence
Gradient Descent
Gradient descent is an optimization algorithm that minimizes loss by iteratively updating weights.
Definition
Gradient descent is the core optimization algorithm for training neural networks. It works by: (1) Computing the gradient of the loss function with respect to weights (via backpropagation), (2) Updating weights in the opposite direction of the gradient. Variants: (1) SGD (Stochastic Gradient Descent) — uses one sample at a time, (2) Mini-batch SGD — uses small batches (32-256 samples), (3) Adam — adaptive learning rates (most popular), (4) AdamW — Adam with weight decay. Learning rate (typically 1e-5 to 1e-3) controls step size. Too high = divergence, too low = slow convergence.
Related Keywords
gradient descentsgdadam optimizeroptimization algorithmlearning rate
Related Terms
Backpropagation
Backpropagation computes gradients of the loss function with respect to model weights for training.
AI Training
AI training is the process of optimizing model parameters using labeled data and compute resources.
Fine-Tuning
Fine-tuning adapts a pre-trained model to a specific task using domain-specific data.