


2.3 Pretrained Language Models 2.3.1 Language Models and Word Embeddings























2.3 Pretrained Language Models 2.3.2 Why Pretrained Models














Perspective | Pretraining Benefit |
Representation Learning | Narrows hypothesis space, induces task-relevant structure |
Sample Complexity | Reduces labeled data needed for good generalization |
Optimization Theory | Provides better initialization for gradient descent |
Bayesian Inference | Acts as a learned prior, guiding parameter estimation |
Information Bottleneck | Encourages compressed but expressive representations |










2.3 Pretrained Language Models 2.3.3 GPT: Autoregressive Pretraining












GPT-1 | GPT-2 | GPT-3 | |
Parameters | 117 Million | 1.5 Billion | 175 Billion |
Decoder Layers | 12 | 48 | 96 |
Context Token Size | 512 | 1024 | 2048 |
Hidden Layer | 768 | 1600 | 12288 |
Batch Size | 64 | 512 | 3.2M |





2.3 Pretrained Language Models 2.3.4 BERT: Masked Language Modeling






























2.3 Pretrained Language Models 2.3.5 T5: Encoder–Decoder Pretraining





























2.3 Pretrained Language Models 2.3.6 Pretrained Model Gallery











2.3 Pretrained Language Models 2.3.7 Suggested Readings



