Transformer Architecture

What is a Transformer?

The Transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need" (Vaswani et al., Google Brain). It replaced recurrent networks (RNNs/LSTMs) as the dominant architecture for sequence processing by relying entirely on a mechanism called self-attention — allowing every position in a sequence to directly attend to every other position in parallel, rather than processing tokens one-by-one.

Every modern large language model — GPT-4, Claude, LLaMA, Gemini — is built on the Transformer architecture. Understanding it at a mechanistic level is the gateway to understanding generative AI.

Quick Recap: Self-Attention

Transformers are built entirely on attention mechanisms. Let's refresh:

For each position in sequence:
1. Compute Query Q, Key K, Value V
2. Compute attention scores: Q · K^T / √d_k
3. Apply softmax to get weights
4. Multiply weights by V (weighted sum)

Result: Each position "attends to" other positions based on relevance!

Why Transformers Beat RNNs/LSTMs

RNN/LSTM Problems:
- Sequential processing (slow!)
- Gradients vanish over long sequences
- Hard to parallelize

Transformer Benefits:
- Parallel processing (GPU-friendly!)
- Direct paths between distant words
- Better gradients (no recurrence)
- Scales to billions of parameters

The Full Transformer Block

Input
  ↓
Layer Norm
  ↓
Multi-Head Attention (8+ heads)
  ↓
Residual Add (skip connection)
  ↓
Layer Norm
  ↓
Feed-Forward Network (2 dense layers)
  ↓
Residual Add (skip connection)
  ↓
Output

Repeat this N times (6-96 layers depending on model size)

Residual Connections (Why Skip Connections Matter)

Without skip connections:
Output = f(Input)

With skip connections:
Output = Input + f(Input)

Benefits:
- Gradients flow directly through skip connection (easier learning)
- Network learns residuals (differences) instead of full transformations
- Allows much deeper networks (100+ layers)

Positional Encoding (Not Just Position Numbers!)

Naive approach:
Position 1: [1, 0, 0, ...]
Position 2: [2, 0, 0, ...]
Problem: Large position numbers dwarf embedding values!

Sinusoidal Encoding (what transformers use):
PE(pos, 2i) = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Benefit:
- Bounded values (between -1 and 1)
- Model can learn positional relationships easily
- Smooth interpolation