Back to Glossary
Artificial Intelligence
Attention Mechanism
Attention allows AI models to focus on relevant parts of input when generating output.
Definition
The attention mechanism is a technique in neural networks that allows the model to focus on different parts of the input sequence when producing each part of the output. Self-attention (used in Transformers) computes attention weights between all pairs of tokens in a sequence. Multi-head attention runs multiple attention operations in parallel, capturing different relationship types. Attention is the key innovation behind modern LLMs (GPT, LLaMA, Claude) and enables long-range dependencies in text. The computational cost of attention is O(n²) with sequence length, driving research into efficient attention variants.
Related Keywords
attention mechanismself attentionmulti head attentiontransformer attention
Related Terms
Transformer Architecture
The Transformer is a neural network architecture using self-attention, the basis for modern LLMs.
Large Language Model (LLM)
An LLM is an AI model trained on massive text data to understand and generate human language.
Foundation Model
A foundation model is a large pre-trained AI model that can be adapted to many downstream tasks.