Back to Glossary
Artificial Intelligence
Transformer Architecture
The Transformer is a neural network architecture using self-attention, the basis for modern LLMs.
Definition
The Transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need" by Google. It uses self-attention mechanisms to process sequential data in parallel, replacing the sequential processing of RNNs and LSTMs. The Transformer architecture is the foundation for all modern large language models — GPT, BERT, LLaMA, Claude, and others. Key innovations: (1) Self-attention — each token attends to all others, (2) Multi-head attention — multiple attention patterns, (3) Positional encoding — captures sequence order.
Related Keywords
transformertransformer architectureself attentionattention mechanismai architecture
Related Terms
Large Language Model (LLM)
An LLM is an AI model trained on massive text data to understand and generate human language.
Foundation Model
A foundation model is a large pre-trained AI model that can be adapted to many downstream tasks.
AI Training
AI training is the process of optimizing model parameters using labeled data and compute resources.