Natural Language UnderstandingTransformer


原 PPT 独立图片资源
Natural Language
Understanding

Transformer
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Background
Transformer Overview
Transformer Block
Transformer Architecture
Challenges and Solutions

2.2 Transformer 2.2.1 Background: Neural Machine Translation and RNNs



原 PPT 独立图片资源

2.2 Transformer 2.2.1 Background: Neural Machine Translation and RNNs

原 PPT 独立图片资源

Machine Translation


原 PPT 独立图片资源
Machine Translation
Machine Translation (MT) is the task of translating a sentence 𝑥 from one language (the source language) to a sentence 𝑦 in another language (the target language).

x: L'homme est né libre, et partout il est dans les fers

y: Man is born free, but everywhere he is in chains
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

History


原 PPT 独立图片资源
History
Early Stage (1950s): Basically just simple rule-based systems doing word substitution. Problem soon appeared intractable.
Statistical Machine Translation (1990s-2010s):
Learn a probabilistic model from data:
𝑎𝑟𝑔𝑚𝑎𝑥𝑦𝑃(𝑦|𝑥)=𝑎𝑟𝑔𝑚𝑎𝑥𝑦𝑃(𝑥|𝑦)𝑃(𝑦)
The best systems were extremely complex, with many separately-designed subcomponents, which requires lots of feature engineering and lots of human effort to maintain
Neural Machine Translation (2010s-)
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

Neural Machine Translation


原 PPT 独立图片资源
Neural Machine Translation
In 2014, Ilya Sutskever applied Sequence to Sequence RNN model to MT, starting the NMT.
NMT learns a Conditional Language Model 𝑃(𝑦|𝑥):
Language Model: the decoder is predicting the next word of the target sentence 𝑦 
Conditional: its predictions are also conditioned on the source sentence 𝑥

𝑃(𝑦|𝑥)=𝑃(𝑦1|𝑥)𝑃(𝑦2|𝑦1,𝑥)𝑃(𝑦3|𝑦2,𝑦1,𝑥)⋯𝑃(𝑦𝑇|𝑦1,⋯,𝑦𝑇−1,𝑥)
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

Sequence to Sequence RNN


原 PPT 独立图片资源
Sequence to Sequence RNN

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

Training the System


原 PPT 独立图片资源
Training the System
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

Sequence to Sequence Learning in NMT


原 PPT 独立图片资源
Sequence to Sequence Learning in NMT
Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to Sequence Learning with Neural Networks.
原 PPT 独立图片资源
原 PPT 独立图片资源

Bottlenecks


原 PPT 独立图片资源
Bottlenecks
Does the architecture limit the learning ability?
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

Limitations: Linear Interaction Distance


原 PPT 独立图片资源
Limitations: Linear Interaction Distance
O(sequence length) steps for distant word pairs to interact means:
Hard to learn long-distance dependencies (because gradient problems!)
Linear order of words is “baked in”; we already know linear order isn’t the right way to think about sentences…
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

Limitations: Lack of Parallelizability


原 PPT 独立图片资源
Limitations: Lack of Parallelizability
Forward/backward have O(sequence length) unparallelizable operations
Inhibits training on very large datasets!
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

2.2 Transformer 2.2.2 Background: Attention



原 PPT 独立图片资源

2.2 Transformer 2.2.2 Background: Attention

原 PPT 独立图片资源

Attention: Core Idea


原 PPT 独立图片资源
Attention: Core Idea
Attention provides a solution to above limitations

On each step of the decoder, use direct connection to the encoder to focus on a particular part of the source sequence.
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

Starting from Meaning-Pooling for RNNs


原 PPT 独立图片资源
Starting from Meaning-Pooling for RNNs
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

Attention is weighted averaging


原 PPT 独立图片资源
Attention is weighted averaging
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源
In attention, the query matches all keys softly,
to a weight between 0 and 1. The keys’ values
are multiplied by the weights and summed.
In a lookup table, we have a table of keys that map to values. The query matches one of the keys, returning its value.

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Attention: Formulation


原 PPT 独立图片资源
Attention: Formulation
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Given encoder hidden states: 𝒉1,⋯,𝒉𝑁∈ℝh
Decoder hidden state on timestep 𝑡: 𝒔𝑡∈ℝh
The attention scores for this step: 𝒆𝑡=[𝒔𝑡𝑇𝒉1,⋯,𝒔𝑡𝑇𝒉𝑁]∈ℝ𝑁
The attention distribution: 𝜶𝑡=softmax(𝒆𝑡)∈ℝ𝑁
The attention output by weighted sum: 𝒂𝑡=Σ𝑖=1𝑁𝛼𝑖𝑡𝒉𝑖∈ℝh
Final decoder hidden state: [𝒂𝑡;𝒔𝑡]∈ℝ2h

Attention solves Limitations of RNN


原 PPT 独立图片资源
Attention solves Limitations of RNN
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Number of unparallelizable operations does not increase with sequence length.
Maximum interaction distance: O(1), since all words interact at every layer
原 PPT 独立图片资源

Attention becomes a General Technique


原 PPT 独立图片资源
Attention becomes a General Technique

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
More general definition of attention:
Given a set of vector values, and a vector query, attention is a technique to compute a weighted sum of the values, dependent on the query.

Intuition:
The weighted sum is a selective summary of the information contained in the values, where the query determines which values to focus on.
Attention is a way to obtain a fixed-size representation of an arbitrary set of representations (the values), dependent on some other representation (the query).

Attention vs RNN


原 PPT 独立图片资源
Attention vs RNN

Feature
Self-Attention
Traditional RNN
Processing Style
✅ Processes all tokens simultaneously (parallel)
❌Processes tokens one by one (sequential)
Information Flow
✅ Any two tokens connect in 1 step
❌Long-range dependencies require many steps
Parallelism
✅ Fully parallelizable (fast)
❌ Inherently sequential (slow)
Dependency Modeling
✅ Excellent at long-range dependencies
❌Poor at long-range dependencies (vanishing gradients)
Interpretability
✅ Attention weights show what is focused on
❌ Hard to interpret hidden states
Linearity
❌ Linear
✅ Nonlinear
Order Sensitive
❌ No
✅ Yes

2.2 Transformer 2.2.3 Transformer Overview



原 PPT 独立图片资源

2.2 Transformer 2.2.3 Transformer Overview

原 PPT 独立图片资源

Motivation


原 PPT 独立图片资源
Motivation
Attention had to be used with RNN, so the architecture RNN+Attention still suffers from RNN’s limitation;
Key Idea:
Why Not Just Use Attention?
Difficulties:
How can Attention used as Encoder?
Attention is Linear
Attention omits the order information
The Solution:
Transformer Architecture

Attention is All You Need (2017)


原 PPT 独立图片资源
Attention is All You Need (2017)

原 PPT 独立图片资源

NMT Task


原 PPT 独立图片资源
NMT Task

原 PPT 独立图片资源
https://jalammar.github.io/illustrated-transformer/

Decoder-Encoder


原 PPT 独立图片资源
Decoder-Encoder

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Stack of Layers


原 PPT 独立图片资源
Stack of Layers

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Encoder Layer


原 PPT 独立图片资源
Encoder Layer

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Decoder Layer


原 PPT 独立图片资源
Decoder Layer

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

2.2 Transformer 2.2.4 Transformer Blocks



原 PPT 独立图片资源

2.2 Transformer 2.2.4 Transformer Blocks

原 PPT 独立图片资源

Input Embedding


原 PPT 独立图片资源
Input Embedding

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Embedding in the Block


原 PPT 独立图片资源
Embedding in the Block

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

2.2 Transformer 2.2.5 Self-Attention



原 PPT 独立图片资源

2.2 Transformer 2.2.5 Self-Attention

原 PPT 独立图片资源

Self-Attention


原 PPT 独立图片资源
Self-Attention

https://jalammar.github.io/illustrated-transformer/
”The animal didn't cross the street because it was too tired”
For each word, what is the relationship between me and other words?
原 PPT 独立图片资源

Mechanism Design: Q, K, V


原 PPT 独立图片资源
Mechanism Design: Q, K, V

Each token acts like a “questioner,” using its own Query to “ask” the Keys of other tokens, and then obtains a weighted combination of their Values — which becomes the final attention output.
📌 Query (Q)
Represents the type of information the token at the current position is seeking.
For example, if the current token is “ate,” it might want to know: “Who did the eating?”
📌 Key (K)
Represents the “tag” or “description” of the information each token provides.
For instance, the token “He” might have a key representing “subject-type pronoun,” so it will be matched accordingly.
📌 Value (V)
Is the actual content or information provided by each token, and will be selected as part of the final output.
The token “He” might provide the actual semantic content like “male third-person singular subject” — this is the Value.
Q and K determine who you pay attention to, but you ultimately receive is the information provided by those you pay attention to — that is, the Values (V).

Mechanism Design: Q, K, V


原 PPT 独立图片资源
Mechanism Design: Q, K, V

Let 𝒘1:𝑛 be a sequence of words in vocabulary 𝑉, like Zuko made his uncle tea.
For each 𝒘𝑖 , let 𝒙𝑖 = 𝐸𝒘𝒊 , where 𝐸 ∈ ℝ𝑑×|𝑉|is an embedding matrix.
Transform each word embedding with weight matrices 𝑄, 𝐾, 𝑉 , each in ℝ𝑑×𝑑
𝒒𝑖 = 𝑄𝒙𝒊 (𝑞𝑢𝑒𝑟𝑖𝑒𝑠)       𝒌𝑖 = 𝐾𝒙𝒊 (𝑘𝑒𝑦𝑠)        𝒗𝑖 = 𝑉𝒙𝒊 (𝑣𝑎𝑙𝑢𝑒𝑠)
Compute pairwise similarities between keys and queries; normalize with softmax
𝒆𝑖𝑗 = 𝒒𝒊⊤𝒌𝒋       𝜶𝑖𝑗 =exp⁡(𝒆𝑖𝑗)/Σ𝑗′ exp⁡(𝒆𝑖𝑗′ ) 
Compute output for each word as weighted sum of values
𝒐𝑖 =Σ𝒋 𝜶𝑖𝑗 𝒗𝑖�

Implementation


原 PPT 独立图片资源
Implementation

原 PPT 独立图片资源
https://jalammar.github.io/illustrated-transformer/

Implementation


原 PPT 独立图片资源
Implementation

原 PPT 独立图片资源
https://jalammar.github.io/illustrated-transformer/

Implementation


原 PPT 独立图片资源
Implementation

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Implementation


原 PPT 独立图片资源
Implementation

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Matrix Calculation


原 PPT 独立图片资源
Matrix Calculation

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Matrix Calculation


原 PPT 独立图片资源
Matrix Calculation

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源
原 PPT 独立图片资源

2.2 Transformer 2.2.6 Feed-Forward Networks



原 PPT 独立图片资源

2.2 Transformer 2.2.6 Feed-Forward Networks

原 PPT 独立图片资源

Feed-Forward


原 PPT 独立图片资源
Feed-Forward

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Feed-Forward


原 PPT 独立图片资源
Feed-Forward

𝑚𝑖 = 𝑀𝐿𝑃(output𝑖)
= 𝑊2 ∗ 𝑅𝑒𝐿𝑈( 𝑊1 output𝑖 + 𝑏1) + 𝑏2
Attention decides “who is important”, while Feedforward decides “how to interpret this information.”
Expanding dimensions gives the model more room for transformation
Reducing dimensions back to the original size for residual connection

2.2 Transformer 2.2.7 Positional Encoding



原 PPT 独立图片资源

2.2 Transformer 2.2.7 Positional Encoding

原 PPT 独立图片资源

From hidden Order Info to Position Feature


原 PPT 独立图片资源
From hidden Order Info to Position Feature

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

The Embedding of Position


原 PPT 独立图片资源
The Embedding of Position

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源
Consider representing each sequence index as a vector:
𝒑𝑖 ∈ ℝ𝑑 , for 𝑖 ∈ {1,2, … , 𝑛}
Easy to incorporate into our self-attention block: just add the 𝒑𝑖 to our inputs!
Recall that 𝒙𝑖 is the embedding of the word at index 𝑖. The positioned embedding is:
𝒙𝑖 = 𝒙𝑖 + 𝒑

The Embedding of Position


原 PPT 独立图片资源
The Embedding of Position

https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/
原 PPT 独立图片资源

Sinusoidal Position Representations


原 PPT 独立图片资源
Sinusoidal Position Representations

https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/
原 PPT 独立图片资源
原 PPT 独立图片资源

Why this Form?


原 PPT 独立图片资源
Why this Form?

https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/
原 PPT 独立图片资源

Visualization


原 PPT 独立图片资源
Visualization

https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/
原 PPT 独立图片资源
The positional encoding matrix for n=10,000, d=512, sequence length=100

2.2 Transformer 2.2.8 Masking the Future



原 PPT 独立图片资源

2.2 Transformer 2.2.8 Masking the Future

原 PPT 独立图片资源

Decoder: Mask the Future


原 PPT 独立图片资源
Decoder: Mask the Future

原 PPT 独立图片资源

原 PPT 独立图片资源
When applying self-attention in decoder, we must ensure we cannot depend on future.

2.2 Transformer 2.2.9 Training Stability



原 PPT 独立图片资源

2.2 Transformer 2.2.9 Training Stability

原 PPT 独立图片资源

Residual and Normalization


原 PPT 独立图片资源
Residual and Normalization

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Residual and Normalization


原 PPT 独立图片资源
Residual and Normalization

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源
原 PPT 独立图片资源

2.2 Transformer 2.2.10 Transformer Block Summary



原 PPT 独立图片资源

2.2 Transformer 2.2.10 Transformer Block Summary

原 PPT 独立图片资源

Building Blocks


原 PPT 独立图片资源
Building Blocks

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源
Self-attention:
the basis of the method.
Position representations:
Specify the sequence order, since self-attention is an unordered function of its inputs.
Nonlinearities:
At the output of the self-attention block
Frequently implemented as a simple feedforward network.
Masking:
Parallelize operations while not looking at the future.
Keeps information about the future from “leaking” to the past.

2.2 Transformer 2.2.11 Transformer Architecture



原 PPT 独立图片资源

2.2 Transformer 2.2.11 Transformer Architecture

原 PPT 独立图片资源

Encoder and Decoder


原 PPT 独立图片资源
Encoder and Decoder

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源
Encoder
Decoder
原 PPT 独立图片资源
No Masking!

Encoder-Decoder


原 PPT 独立图片资源
Encoder-Decoder

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源

The embedding produced by Decoder should be conditional on the Encoder embedding
The Cross Attention:
Encoder Embedding K, V;
Decoder Embedding Q

Encoder-Decoder Illustration


原 PPT 独立图片资源
Encoder-Decoder Illustration

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Encoder-Decoder Illustration


原 PPT 独立图片资源
Encoder-Decoder Illustration

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

2.2 Transformer 2.2.12 Position Representations



原 PPT 独立图片资源

2.2 Transformer 2.2.12 Position Representations

原 PPT 独立图片资源

Sinusoidal Position Representations


原 PPT 独立图片资源
Sinusoidal Position Representations

原 PPT 独立图片资源
Pros:
Periodicity indicates that maybe “absolute position” isn’t as important
Maybe can extrapolate to longer sequences as periods restart!
Cons:
Not learnable; also the extrapolation doesn’t really work!

Learnable Position Representations


原 PPT 独立图片资源
Learnable Position Representations

Let all 𝒑𝑖 be learnable parameters! Learn a matrix 𝒑 ∈ ℝ𝑑×𝑛 , and let each 𝒑𝑖 be a column of that matrix!
Pros:
Flexibility: each position gets to be learned to fit the data
Cons:
Definitely can’t extrapolate to indices outside 1, … , 𝑛.
Widely used in many system;
From absolute position to relative position: 𝒑𝑖,𝒑𝑗→𝒓𝑖𝑗
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf

RoPE –Rotary Position Embedding


原 PPT 独立图片资源
RoPE –Rotary Position Embedding

Relative position should be some 𝑓(𝑥, 𝑖) s.t.
<𝑓(𝑥, 𝑖), 𝑓(𝑦,𝑗)> = 𝑔(𝑥, 𝑦, 𝑖 − 𝑗) 
That is, the attention function only gets to depend on the relative position (𝑖−𝑗) .
Relative position equivariance → Rotation
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源

RoPE in High Dimension


原 PPT 独立图片资源
RoPE in High Dimension

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源

2.2 Transformer 2.2.13 Context Length



原 PPT 独立图片资源

2.2 Transformer 2.2.13 Context Length

原 PPT 独立图片资源

Beyond Fixed-Length Context


原 PPT 独立图片资源
Beyond Fixed-Length Context

The matrix computations in Transformer need Fixed-Length Context
Limited context hampers performance on long texts
Key use cases:
Long documents
Conversations
Streaming text
Agents
…
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf

Approaches


原 PPT 独立图片资源
Approaches

Why Transformer cannot cope with very long context?
The attention complexity is 𝑂(𝑛2)
Approaches:
Sparse Attention
Linear approximation of Attention: Back of RNN

Sparse Attention


原 PPT 独立图片资源
Sparse Attention

原 PPT 独立图片资源
https://research.google/blog/constructing-transformers-for-longer-sequences-with-sparse-attention-methods/

Linear Approximation


原 PPT 独立图片资源
Linear Approximation

https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state
原 PPT 独立图片资源
Scaling Laws for Mamba vs other Language Models
https://thegradient.pub/mamba-explained/

2.2 Transformer 2.2.14 Transferability of Modifications



原 PPT 独立图片资源

2.2 Transformer 2.2.14 Transferability of Modifications

原 PPT 独立图片资源

Do Transformer Modifications Transfer?


原 PPT 独立图片资源
Do Transformer Modifications Transfer?

"Surprisingly, we find that most modifications do not meaningfully improve performance."
原 PPT 独立图片资源
原 PPT 独立图片资源

2.2 Transformer 2.2.15 Suggested Readings



原 PPT 独立图片资源

2.2 Transformer 2.2.15 Suggested Readings

原 PPT 独立图片资源

Papers and Articles


原 PPT 独立图片资源
Papers and Articles
Attention Is All You Need https://arxiv.org/abs/1706.03762

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源