


2.1 Statistical Language Modeling 2.1.1 Statistical Language Modeling















2.1 Statistical Language Modeling 2.1.2 Building an N-gram Language Model


















2.1 Statistical Language Modeling 2.1.3 Generalization to Unseen Words










2.1 Statistical Language Modeling 2.1.4 Generation










2.1 Statistical Language Modeling 2.1.5 Evaluation











Unigram | Bigram | Trigram | |
Perplexity | 962 | 170 | 109 |



2.2 Transformer 2.2.1 Background: Neural Machine Translation and RNNs


















2.2 Transformer 2.2.2 Background: Attention

































Feature | Self-Attention | Traditional RNN |
Processing Style | ✅ Processes all tokens simultaneously (parallel) | ❌Processes tokens one by one (sequential) |
Information Flow | ✅ Any two tokens connect in 1 step | ❌Long-range dependencies require many steps |
Parallelism | ✅ Fully parallelizable (fast) | ❌ Inherently sequential (slow) |
Dependency Modeling | ✅ Excellent at long-range dependencies | ❌Poor at long-range dependencies (vanishing gradients) |
Interpretability | ✅ Attention weights show what is focused on | ❌ Hard to interpret hidden states |
Linearity | ❌ Linear | ✅ Nonlinear |
Order Sensitive | ❌ No | ✅ Yes |

2.2 Transformer 2.2.3 Transformer Overview















2.2 Transformer 2.2.4 Transformer Blocks






2.2 Transformer 2.2.5 Self-Attention





























2.2 Transformer 2.2.6 Feed-Forward Networks





2.2 Transformer 2.2.7 Positional Encoding















2.2 Transformer 2.2.8 Masking the Future





2.2 Transformer 2.2.9 Training Stability







2.2 Transformer 2.2.10 Transformer Block Summary




2.2 Transformer 2.2.11 Transformer Architecture











2.2 Transformer 2.2.12 Position Representations









2.2 Transformer 2.2.13 Context Length










2.2 Transformer 2.2.14 Transferability of Modifications





2.2 Transformer 2.2.15 Suggested Readings







2.3 Pretrained Language Models 2.3.1 Language Models and Word Embeddings























2.3 Pretrained Language Models 2.3.2 Why Pretrained Models














Perspective | Pretraining Benefit |
Representation Learning | Narrows hypothesis space, induces task-relevant structure |
Sample Complexity | Reduces labeled data needed for good generalization |
Optimization Theory | Provides better initialization for gradient descent |
Bayesian Inference | Acts as a learned prior, guiding parameter estimation |
Information Bottleneck | Encourages compressed but expressive representations |










2.3 Pretrained Language Models 2.3.3 GPT: Autoregressive Pretraining












GPT-1 | GPT-2 | GPT-3 | |
Parameters | 117 Million | 1.5 Billion | 175 Billion |
Decoder Layers | 12 | 48 | 96 |
Context Token Size | 512 | 1024 | 2048 |
Hidden Layer | 768 | 1600 | 12288 |
Batch Size | 64 | 512 | 3.2M |





2.3 Pretrained Language Models 2.3.4 BERT: Masked Language Modeling






























2.3 Pretrained Language Models 2.3.5 T5: Encoder–Decoder Pretraining





























2.3 Pretrained Language Models 2.3.6 Pretrained Model Gallery











2.3 Pretrained Language Models 2.3.7 Suggested Readings







2.4 Language Understanding Tasks 2.4.1 Word Sense Disambiguation















2.4 Language Understanding Tasks 2.4.2 Sentiment Analysis





















2.4 Language Understanding Tasks 2.4.3 Semantic Textual Similarity




Task | Goal | Difference from STS |
Paraphrase Detection | Are the sentences semantically equivalent? | Binary decision (yes/no) |
NLI / Textual Entailment | Does one sentence entail another? | Focuses on logical inference |
Text Matching | Do the texts match in form/content? | Often used in retrieval |












2.4 Language Understanding Tasks 2.4.4 Text Summarization



Type | Description | Example |
Extractive | Selects and combines important sentences from the original text | Sentence selection |
Abstractive | Generates new sentences using understanding and rephrasing | Paraphrased summary |












Model | Copy Ability | Generate New Words | OOV Handling | Use Case Examples |
Seq2Seq + Attn | ❌ | ✅ | Poor | General NLG |
Pointer Network | ✅ (positions) | ❌ | Good | Span selection, TSP |
CopyNet | ✅ (tokens) | ✅ | Excellent | Summarization, QA, NLG |



2.4 Language Understanding Tasks 2.4.5 Information Retrieval




























2.4 Language Understanding Tasks 2.4.6 Question Answering













2.4 Language Understanding Tasks 2.4.7 Dialogue Systems


































2.5 Large Language Models 2.5.1 Pretraining














Capability | PLM (Early Models) | LLM (Modern Models) |
Model Size | ~100M – 1B parameters | 10B – 500B+ parameters |
Training Data | Mid-sized, domain-limited | Diverse, web-scale corpora |
Task Generalization | Weak, narrow | Strong, task-agnostic |
Emergent Abilities | ✘ None observed | ✔ Appear with scale |
Language Understanding | Task-specific pattern matching | Signs of general competence |


















































2.5 Large Language Models 2.5.2 Supervised Instruction Fine-Tuning



























2.5 Large Language Models 2.5.3 Reinforcement Learning from Human Feedback




























































2.5 Large Language Models 2.5.4 Decoding Strategies


![XCS224N] Lecture 8 – Translation, Seq2Seq, Attention - mx's blog](assets/ppt-elements/12-image79.png)

























2.5 Large Language Models 2.5.5 Suggested Readings



