Week 9 · Mar 9–13

Transformers, BERT & GPT

The architecture that changed NLP forever — self-attention, positional encoding, and the pre-trained model revolution.

Module 9 · 4 Lectures Self-Attention BERT & GPT Transfer Learning ⚠️ Quiz 8 Upcoming
🚀

Why Transformers?

History & motivation for moving beyond LSTMs

Problems with LSTMs that Transformers Solve

▼
What

LSTMs had three key bottlenecks that made them hard to scale to the level needed for modern NLP.

ProblemDetailTransformer Fix
Sequential (Slow)Items are fed one at a time — needs h_{t-1} before computing h_t, so can't parallelizeSelf-attention sees all tokens at once — fully parallelizable
Large MemoryMust store all hidden states and gradients through timeAttention over fixed-length key/value stores
No Transfer LearningMust retrain from scratch on every new labeled dataset — costly & slowPre-train once on massive corpus; fine-tune cheaply on any task

Timeline: From Transformers to Modern LLMs

▼
2017
"Attention Is All You Need" — Vaswani et al.

Original Transformer architecture introduced; designed for machine translation.

June 2018
GPT-1 (OpenAI)

First major pre-trained Transformer; Generative Pre-Training using decoder stack.

October 2018
BERT (Google)

Bidirectional Encoder Representations from Transformers; excels at understanding tasks.

2019
DistilBERT

60% faster, 40% lighter than BERT while maintaining comparable accuracy.

2020
GPT-3 (OpenAI)

175 billion parameters; autoregressive model for generating human-like text.

Why Pre-Train Instead of Training from Scratch?

▼
🎯 The Core Idea Train once on massive unlabeled text using self-supervised learning, then fine-tune cheaply on specific tasks.
  • Time: Training from scratch on large corpora takes extremely long — weeks to months on hundreds of GPUs.
  • Cost & Carbon: Pre-training is very expensive and creates a large environmental footprint.
  • Data efficiency: Fine-tuning a pre-trained model requires far less labeled data than training from scratch.
  • Domain knowledge: A pre-trained model already has statistical knowledge of language that transfers to the fine-tuning task.
💡 Self-Supervised Learning Labels are automatically created from the raw text (e.g., masking words, predicting next word) — no human labeling needed!

NLP Tasks Transformers Excel At

▼

🌐 Machine Translation (MT)

Translate text from one natural language to another (e.g., English → French).

💬 Sentiment Analysis

Identify the emotion or opinion in a sentence (positive/negative/neutral).

📝 Text Summarization

Condense a long document into a shorter, meaningful version.

🔲 Fill-in-the-Blank (MLM)

"This course will teach you all about ___ models." → Transformers

🏷️ Named Entity Recognition (NER)

Detect entities in text: people, places, dates, organizations.

🔤 Part-of-Speech (POS) Tagging

Assign grammatical labels: noun, verb, adjective, etc. to each word.

🏗️

Transformer Architecture

Encoder, decoder, and the three model families

Encoder vs. Decoder Overview

▼
Input Tokens
→
Token Embedding
+
Positional Encoding
→
Encoder Block(s)
→
Context Vectors
Context Vector (Bottleneck) The last encoder produces a fixed-length context vector that is passed to the decoder. The decoder attends to this to generate output.

🔵 Encoder

  • Reads the entire input sequence at once (bidirectional)
  • Produces a rich representation for each token
  • Best for understanding tasks (NLU)
  • Models: BERT, RoBERTa, ALBERT

🟠 Decoder

  • Generates tokens one at a time (left-to-right only)
  • Attends only to previously generated tokens (masked)
  • Best for generation tasks (NLG)
  • Models: GPT, GPT-2, GPT-3

🔵+🟠 Encoder–Decoder (Seq2Seq)

Seq2Seq

Uses both components for generative tasks that require input (e.g., translation, summarization). Models: BART, T5.

Inside One Encoder Block (4 Modules)

▼
Input
(Embedding + Pos)
→
Self-Attention
Layer
→
Add & Norm
(Skip Connection)
→
Feed-Forward
Network
→
Add & Norm
(Skip Connection)
→
Output
(to next block)

① Self-Attention

Computes relationships between all words in the sequence simultaneously using Q, K, V matrices.

② Skip (Residual) Connection

Adds the input directly to the attention output: output = LayerNorm(x + Attention(x)). Prevents vanishing gradients.

③ Layer Normalization

Normalizes the distribution of intermediate layers → smoother gradients, faster training, better generalization.

④ Feed-Forward Network (FFN)

Two linear layers with ReLU in between. Transforms attention vectors for the next block. Also has a skip connection + LayerNorm after it.

Positional Encoding

▼
Why

Transformers have no built-in sense of word order (unlike RNNs/CNNs). Positional encoding injects order information into the embeddings so the model knows where each word appears.

// Positional Encoding formula PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) ← even indices PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)) ← odd indices // pos = position in sequence, i = dimension index // d_model = embedding dimension // Final input = Token Embedding + Positional Encoding
💡 Key Point Sine for even indices, cosine for odd indices. The resulting vectors are added to (not concatenated with) the word embeddings.

Multi-Head Attention

▼
What

Instead of a single self-attention computation, multi-head attention runs several attention operations in parallel, each with its own Q, K, V matrices. Outputs are concatenated and projected.

Why Use Multiple Heads? Each head can attend to different kinds of relationships simultaneously — e.g., one head captures syntax, another captures coreference, another captures semantic similarity.
// Multi-Head Attention head_i = Attention(Q·Wᵢᵠ, K·Wᵢᵏ, V·Wᵢᵛ) MultiHead = Concat(head_1, ..., head_h) · Wᴼ // Each head has its own learned projection matrices Wᵠ, Wᵏ, Wᵛ // Processed in parallel → efficient on GPUs
🔍

Self-Attention: The Core Mechanism

Query, Key, Value — understanding how attention scores are computed

Intuition: "It was hungry" — which word does "it" refer to?

▼
The Problem

Consider: "The cat drank the milk because it was hungry." vs "The cat drank the milk because it was sweet."
In the first sentence "it" = cat; in the second "it" = milk. Self-attention lets each word "look at" every other word to resolve this.

💡 Self-Attention Purpose For each word in the input, compute a weighted combination of all other words' representations, where the weights represent how relevant each word is to understanding the current word.

Query, Key, Value (Q, K, V) — Step by Step

▼
🗝️ KEY (K)

What each word offers.
"Here's what I contain."

❓ QUERY (Q)

What the current word is looking for.
"What am I interested in?"

💰 VALUE (V)

The content to be retrieved.
"Here's the actual information."

// Step-by-step self-attention calculation // 1. Project each word embedding into Q, K, V vectors q_i = x_i · W_Q ← query for word i k_j = x_j · W_K ← key for word j v_j = x_j · W_V ← value for word j // 2. Compute raw attention score (dot product) e_ij = q_i · k_jᵀ / √d_k ← scale to avoid large values // 3. Softmax to get attention weights (sum to 1) α_ij = softmax(e_ij) // 4. Weighted sum of values = output for word i z_i = Σⱼ α_ij · v_j // Matrix form (all words at once): Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V
⚠️ Why divide by √d_k? With large embedding dimensions, dot products grow very large → softmax becomes extremely peaked → vanishing gradients. Dividing by √d_k stabilizes training.

🔬 Interactive: Self-Attention Weight Visualizer

Click a query word to see how much attention it pays to each key word in the sentence.
🤖

Model Families: BERT, GPT, T5

Encoder-only, decoder-only, and encoder–decoder architectures

BERT — Encoder-Only (Bidirectional)

NLU Tasks
  • Trained on BooksCorpus (~800M words) + English Wikipedia (~2.5B words)
  • Two pre-training objectives: MLM + NSP
  • Bidirectional: attends to words both before AND after
  • Great for: classification, Q&A, NER, sentiment
  • Related: RoBERTa, ALBERT, DistilBERT (60% faster, 40% lighter)

Masked Language Modeling (MLM)

Mask ~15% of tokens → train model to predict them. Example:
"The man went to the [MASK]." → "store"

Next Sentence Prediction (NSP)

Given two sentences A & B, predict: IsNext or NotNext. Trains the model to understand sentence relationships (important for Q&A, NLI).

# HuggingFace BERT example — fill-mask pipeline from transformers import pipeline pipe = pipeline("fill-mask", model="bert-base-uncased") result = pipe("The man went to the [MASK]. He bought a [MASK] of milk.") # MASK_1 → "store", MASK_2 → "gallon"

GPT — Decoder-Only (Unidirectional, Autoregressive)

NLG Tasks
  • Developed by OpenAI — Generative Pre-Training
  • Autoregressive: predicts the next word given all previous words
  • Unidirectional: only looks left (past tokens)
  • GPT-3: 175 billion parameters

Pre-training Objective

Language modeling: given "The cat sat on the", predict "mat". Repeat for all positions. No labels needed — fully self-supervised.

# HuggingFace GPT-2 example — text generation from transformers import pipeline pipe = pipeline("text-generation", model="gpt2") results = pipe("Help! I'm a language model", max_length=50, num_return_sequences=5) # Generates 5 different continuations of the input text

T5 / BART — Encoder–Decoder (Seq2Seq)

Generative + Understanding
  • Use both encoder stack (to understand input) and decoder stack (to generate output)
  • Best for tasks needing input → output: translation, summarization, question answering with generation
  • Used the original "Attention Is All You Need" architecture (Vaswani et al., 2017)

Complete Architecture Comparison

▼
Feature RNN/LSTM BERT (Encoder) GPT (Decoder) T5/BART (Seq2Seq)
Attention Bahdanau (optional) Bidirectional self-attn Masked self-attn Both
Direction Left→Right Bidirectional ✓ Left→Right only Enc: bi, Dec: L→R
Parallelization Sequential ✗ Full parallel ✓ Parallel (train) ✓ Parallel (train) ✓
Transfer Learning Poor ✗ Excellent ✓ Excellent ✓ Excellent ✓
Best For Short sequences NLU (classification) NLG (generation) Seq-to-seq tasks
Examples LSTM, GRU BERT, RoBERTa, ALBERT GPT-1/2/3, Llama T5, BART, mT5

🎯 Quiz 8 Practice — Transformers, BERT & GPT

Q1. What is the main reason transformers can be trained faster than LSTMs?

Q2. In the self-attention formula Attention(Q, K, V) = softmax(Q·Kᵀ / √d_k) · V, what does dividing by √d_k accomplish?

Q3. BERT is described as "bidirectional." What does this mean?

Q4. Which pre-training objective does BERT use to learn sentence-level relationships?

Q5. What is positional encoding in transformers and why is it needed?

Q6. GPT is an "autoregressive" model. What does that mean during generation?

← Week 7: LSTM & Attention 🏠 All Weeks
🔬 Beyond the Slides · Graduate Depth

PEFT & LoRA: Fine-Tuning LLMs Without Breaking the Bank

Full fine-tuning a 70B parameter model requires 280+ GB of GPU memory. In 2026, virtually no research lab does this. LoRA and QLoRA are the dominant paradigm for adapting large models — and every NLP practitioner must understand the math and tradeoffs. This section covers what the slides skip.

🧠 Why Parameter-Efficient Fine-Tuning? Critical

📖 The Problem

Full fine-tuning updates all W parameters. For LLaMA-3 70B, that's ~70 billion floats × 4 bytes × 3 (weights + gradients + optimizer states) ≈ 840 GB GPU memory — well beyond any single GPU. PEFT methods update only a tiny adapter while freezing the pretrained weights, achieving near-full-finetune quality with 100–1000× fewer parameters.

The PEFT Family

  • Adapter Layers (Houlsby 2019): bottleneck FFN inserted between transformer sub-layers
  • Prefix Tuning (Li & Liang 2021): prepend trainable virtual tokens to K/V
  • Prompt Tuning (Lester 2021): soft prompt at input layer only
  • LoRA (Hu et al. 2021): low-rank weight decomposition ★ Dominant method
  • QLoRA (Dettmers 2023): LoRA on 4-bit quantized base model

Why LoRA Dominates

  • No inference latency — adapters merge into W at deployment
  • Works on any linear layer (Q, K, V, O, FFN)
  • 80%+ of HuggingFace fine-tuning jobs use LoRA/QLoRA
  • Rank r as low as 4 often matches full fine-tune quality
  • Can train 70B model on a single A100 80GB with QLoRA

📐 LoRA Mathematics PhD

LoRA's key insight: the weight update ΔW from full fine-tuning has low intrinsic rank. So instead of updating the full d×k matrix W, decompose the update into two small matrices:

Pretrained weight: W ∈ ℝ^{d×k} (frozen) LoRA update: ΔW = B·A B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k} rank r << min(d, k) Forward pass: h = Wx + BAx · (α/r) ↑ base ↑ adapter (scaled) Parameters saved: Full: d×k LoRA: d×r + r×k = r(d+k) Example: d=4096, k=4096, r=8 Full: 16.7M params LoRA: 65K params → 256× fewer!

Initialization

  • A initialized with Gaussian noise (random)
  • B initialized to zero (so ΔW=0 at start)
  • Training starts from original pretrained behavior
  • Scaling factor α/r normalizes the adapter magnitude

Hyperparameter Intuition

  • r=4: minimal, for tasks close to pretraining
  • r=16: common default, good quality/cost balance
  • r=64+: approaching full finetune for domain shift
  • α: usually set to r (ratio α/r = 1.0)
  • Layers: apply to Q,K,V,O or all linear layers

🔬 QLoRA: 4-Bit Fine-Tuning 2023 Breakthrough

QLoRA (Dettmers et al., 2023) combines three innovations to fine-tune 65B models on a single 48GB GPU:

The Three Innovations

  1. 4-bit NormalFloat (NF4): quantization format derived from Normal distribution — optimal for normally-distributed neural weights
  2. Double Quantization: quantize the quantization constants themselves — saves ~0.37 bits/param
  3. Paged Optimizers: use NVIDIA unified memory to page optimizer states from GPU to CPU RAM when GPU runs out

QLoRA in Practice

# HuggingFace QLoRA setup from transformers import BitsAndBytesConfig from peft import get_peft_model, LoraConfig bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, ) model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-3-8b", quantization_config=bnb_config ) lora_cfg = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"]) model = get_peft_model(model, lora_cfg)

📊 PEFT Methods Comparison

MethodTrainable ParamsGPU Memory (7B)Quality vs. Full FTInference Overhead
Full Fine-tuning100% (~7B)~80GB (fp16)BaselineNone
LoRA (r=16)~0.1% (~7M)~18GB (fp16)95–99%None (merge at deploy)
QLoRA (r=16, 4-bit)~0.1% (~7M)~6GB (int4)93–97%Small (dequantize)
Prefix Tuning0.1–1%~16GB85–90%Yes (longer KV)
Adapter Layers0.5–3%~20GB88–94%Yes (sequential)

GPU memory figures are approximate for Llama-3 7B with typical batch sizes. Quality percentages relative to full fine-tuning on standard benchmarks.