Why Transformer Is Needed
An architecture proposed in the 2017 paper "Attention Is All You Need." Earlier RNNs and LSTMs process words in order, so they're slow and bad at long text. Transformer reconciles parallel processing and learning long-range dependencies.
What Is Attention
Attention is a mechanism that learns "to understand the current word, which other words in the sentence to weight, and how much."
Example: "He sat on the bank"
Whether "bank" is a "financial institution" or "river edge" is decided by attention to other words in context ("sat"). Attention expresses inter-word relevance numerically.
Q / K / V (Query, Key, Value)
Three vectors are made for each word.
- Query (Q): "what am I looking for now"
- Key (K): "I can provide this kind of information"
- Value (V): the actual information content
Compute relevance by Q-K dot product → softmax into probabilities → weighted sum of V → output. This is computed in parallel for all words.
Multi-Head Attention
One kind of Attention alone is poor in expressiveness, so run multiple Attentions in parallel (Heads). Each Head learns a different relationship (grammar, meaning, coreference, etc.).
Encoder vs Decoder
| Type | Use | Representative |
|---|---|---|
| Encoder-only | Classification, understanding, embedding | BERT, RoBERTa |
| Decoder-only | Generation (next-word prediction) | GPT, Claude, Llama |
| Encoder-Decoder | Translation, summarization | T5, BART |
Positional Encoding
Since Attention alone loses word-order info, positional encoding (Sinusoidal, RoPE, etc.) embeds "this is the Nth word." RoPE is the current mainstream.
FFN (Feed-Forward Network)
After the Attention layer, a 2-layer neural net (FFN) is inserted. It processes each word independently, increasing expressiveness. Most of an LLM's parameters are FFN.
Mixture of Experts (MoE)
A technique splitting FFN into "experts" and activating only some by input. Adopted by GPT-4, Mistral 8x7B, DeepSeek V3, etc. Total params are huge but inference cost is low.
How It Helps in Practice
- Long-context understanding (Attention's wide range)
- Few-shot prompts (learning example sentences as patterns)
- Tool Use (the tool-call counterpart is also processed as "words")
- Multimodal (image/audio tokens processed by the same Attention)
Summary
Transformer is an architecture that "finds related words with Attention, increases expressiveness with Multi-Head and FFN." Without memorizing formulas, grasping the meaning of Q/K/V and the Multi-Head/MoE mechanism lets you understand even the latest model news.
