


2.5 Large Language Models 2.5.1 Pretraining














Capability | PLM (Early Models) | LLM (Modern Models) |
Model Size | ~100M – 1B parameters | 10B – 500B+ parameters |
Training Data | Mid-sized, domain-limited | Diverse, web-scale corpora |
Task Generalization | Weak, narrow | Strong, task-agnostic |
Emergent Abilities | ✘ None observed | ✔ Appear with scale |
Language Understanding | Task-specific pattern matching | Signs of general competence |


















































2.5 Large Language Models 2.5.2 Supervised Instruction Fine-Tuning



























2.5 Large Language Models 2.5.3 Reinforcement Learning from Human Feedback




























































2.5 Large Language Models 2.5.4 Decoding Strategies


![XCS224N] Lecture 8 – Translation, Seq2Seq, Attention - mx's blog](assets/ppt-elements/12-image79.png)

























2.5 Large Language Models 2.5.5 Suggested Readings



