


2.2 Transformer 2.2.1 Background: Neural Machine Translation and RNNs


















2.2 Transformer 2.2.2 Background: Attention

































Feature | Self-Attention | Traditional RNN |
Processing Style | ✅ Processes all tokens simultaneously (parallel) | ❌Processes tokens one by one (sequential) |
Information Flow | ✅ Any two tokens connect in 1 step | ❌Long-range dependencies require many steps |
Parallelism | ✅ Fully parallelizable (fast) | ❌ Inherently sequential (slow) |
Dependency Modeling | ✅ Excellent at long-range dependencies | ❌Poor at long-range dependencies (vanishing gradients) |
Interpretability | ✅ Attention weights show what is focused on | ❌ Hard to interpret hidden states |
Linearity | ❌ Linear | ✅ Nonlinear |
Order Sensitive | ❌ No | ✅ Yes |

2.2 Transformer 2.2.3 Transformer Overview















2.2 Transformer 2.2.4 Transformer Blocks






2.2 Transformer 2.2.5 Self-Attention





























2.2 Transformer 2.2.6 Feed-Forward Networks





2.2 Transformer 2.2.7 Positional Encoding















2.2 Transformer 2.2.8 Masking the Future





2.2 Transformer 2.2.9 Training Stability







2.2 Transformer 2.2.10 Transformer Block Summary




2.2 Transformer 2.2.11 Transformer Architecture











2.2 Transformer 2.2.12 Position Representations









2.2 Transformer 2.2.13 Context Length










2.2 Transformer 2.2.14 Transferability of Modifications





2.2 Transformer 2.2.15 Suggested Readings



