Natural Language UnderstandingStatistical Language Modeling


原 PPT 独立图片资源
Natural Language
Understanding

Statistical Language Modeling
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Statistical Language Modeling
Building a N-Gram Language Model
Generalization to Unseen Words
Generation
Evaluation

2.1 Statistical Language Modeling 2.1.1 Statistical Language Modeling



原 PPT 独立图片资源

2.1 Statistical Language Modeling 2.1.1 Statistical Language Modeling

原 PPT 独立图片资源

Problem: Is this Document Natural?


原 PPT 独立图片资源
Problem: Is this Document Natural?

Symbolic Approach:
Decompose the document into sentences and paragraphs;
Check whether each sentence is generated by valid generative gramma;
Check whether the former and latter sentences/paragraphs satisfy coherent relations;
Check whether the knowledge stated in the document be consistent with the real world.
…
Statistical Approach
Check whether 𝑝(𝐷)>𝜃

Language Modeling


原 PPT 独立图片资源
Language Modeling

Probabilistic Sequence Model:



The sentence/document is generated by Next-Word-Generation: generate 𝑤𝑖 given 𝑤1,⋯, 𝑤𝑖−1.
原 PPT 独立图片资源

Predicting Next Words


原 PPT 独立图片资源
Predicting Next Words

The water of Walden Pond is beautifully ...
blue
green
clear
refrigerator
that
tomorrow

Examples


原 PPT 独立图片资源
Examples

𝑃(“𝑇ℎ𝑒 𝑤𝑎𝑡𝑒𝑟 𝑜𝑓 𝑊𝑎𝑙𝑑𝑒𝑛 𝑃𝑜𝑛𝑑”)= 𝑃(𝑇ℎ𝑒)
× 𝑃(𝑤𝑎𝑡𝑒𝑟|𝑇ℎ𝑒)
×  𝑃(𝑜𝑓|𝑇ℎ𝑒 𝑤𝑎𝑡𝑒𝑟) 
×  𝑃(𝑊𝑎𝑙𝑑𝑒𝑛|𝑇ℎ𝑒 𝑤𝑎𝑡𝑒𝑟 𝑜𝑓) 
×  𝑃(𝑃𝑜𝑛𝑑|𝑇ℎ𝑒 𝑤𝑎𝑡𝑒𝑟 𝑜𝑓 𝑊𝑎𝑙𝑑𝑒𝑛)

How to estimate these probabilities


原 PPT 独立图片资源
How to estimate these probabilities

Could we just count and divide?





No! Too many possible sentences!
We’ll never see enough data for estimating these
原 PPT 独立图片资源
原 PPT 独立图片资源
=

Markov Assumption


原 PPT 独立图片资源
Markov Assumption

原 PPT 独立图片资源
原 PPT 独立图片资源
≈
Simplify the problem by applying Markov Assumption:

Bigram Model


原 PPT 独立图片资源
Bigram Model

原 PPT 独立图片资源
原 PPT 独立图片资源

N-gram Model


原 PPT 独立图片资源
N-gram Model

原 PPT 独立图片资源
Generalize the Markov Assumption to Higher Order Markov Assumption:
Given the previous 𝑛−1 words, probability of  𝑛-th word is independent of words prior to 𝑛−𝑁+1 words.

Simplest Case: Unigram Model


原 PPT 独立图片资源
Simplest Case: Unigram Model

To him swallowed confess hear both . Which . Of save on trail for are ay device and rote life have
Hill he late speaks ; or ! a more to leg less first you enter
Months the my and issue of year foreign new exchange’s September
were recession exchange new endorsed a acquire to six executives
原 PPT 独立图片资源

Example of Bigram Model


原 PPT 独立图片资源
Example of Bigram Model

Why dost stand forth thy canopy, forsooth; he is this palpable hit the King Henry. Live king. Follow.
What means, sir. I confess she? then all sorts, he is trim, captain.
Last December through the way to preserve the Hudson corporation N. B. E. C. Taylor would seem to complete the major central planners one gram point five percent of U. S. E. has already old M. X. corporation of living
on information such as more frequently fishing to keep her
原 PPT 独立图片资源

Pros and Cons of N-gram Model


原 PPT 独立图片资源
Pros and Cons of N-gram Model

Pros:
Make Statistical Language Model feasible, greatly promote the research;
Cons:
N-grams do not modeling semantic relationships.
N-grams can't handle long-distance dependencies.
“The soups that I made from that new cookbook I bought yesterday were amazingly delicious."

Application of N-Gram Model


原 PPT 独立图片资源
Application of N-Gram Model

Build a N-Gram Model from a Corpus, and use it to:
Calculate the probability of a sequence:
Select most possible (in ASR) /natural (in MT) result;
Generate a sequence:
Input completion in Input Method/Code Editing/Search Query;

2.1 Statistical Language Modeling 2.1.2 Building an N-gram Language Model



原 PPT 独立图片资源

2.1 Statistical Language Modeling 2.1.2 Building an N-gram Language Model

原 PPT 独立图片资源

Build a N-Gram Model from Corpus


原 PPT 独立图片资源
Build a N-Gram Model from Corpus

Language distributions are different among problem domains:
If task-specific (legal or medical), use a training corpus that has a similar genre to your task.
Variety is important:
Make sure to cover different kinds of language patterns in the task domain.

Estimating Bigram Probabilities


原 PPT 独立图片资源
Estimating Bigram Probabilities

<s> I am Sam </s>
<s> Sam I am </s>
<s> I do not like green eggs and ham </s>
原 PPT 独立图片资源
原 PPT 独立图片资源
sam.tiff

More examples: Berkeley Restaurant Project


原 PPT 独立图片资源
More examples:
Berkeley Restaurant Project

can you tell me about any good cantonese restaurants close by
tell me about chez panisse
i’m looking for a good place to eat breakfast
when is caffe venezia open during the day

Raw Bigram Counts: 9222 Sentences


原 PPT 独立图片资源
Raw Bigram Counts: 9222 Sentences

berp1

Raw Bigram Probabilities


原 PPT 独立图片资源
Raw Bigram Probabilities

berp3
berp2

Bigram estimates of sentence probabilities


原 PPT 独立图片资源
Bigram estimates of sentence probabilities

𝑃(<s> 𝐼 𝑤𝑎𝑛𝑡 𝑒𝑛𝑔𝑙𝑖𝑠ℎ 𝑓𝑜𝑜𝑑 </𝑠>) 
=    𝑃(𝐼|<𝑠>)  
×  𝑃(𝑤𝑎𝑛𝑡|𝐼) 
×  𝑃(𝑒𝑛𝑔𝑙𝑖𝑠ℎ|𝑤𝑎𝑛𝑡)
×  𝑃(𝑓𝑜𝑜𝑑|𝑒𝑛𝑔𝑙𝑖𝑠ℎ)   
×  𝑃(</𝑠>|𝑓𝑜𝑜𝑑)
=  .000031

Dealing with scale in large n-grams


原 PPT 独立图片资源
Dealing with scale in large n-grams

Log Probabilities: avoids underflow from multiplying many small numbers
原 PPT 独立图片资源

Larger ngrams: 4-grams, 5-grams


原 PPT 独立图片资源
Larger ngrams: 4-grams, 5-grams

N-grams from Corpus of Contemporary American English (COCA) 1 billion words (Davies 2020)
Google Web 5-grams (Franz and Brants 2006) 1 trillion words)

Infini-grams (∞-grams)


原 PPT 独立图片资源
Infini-grams (∞-grams)

The biggest n-gram LM ever built to date.
Trained on 5 trillion tokens;
Contains about 5 quadrillion (or 5 thousand trillion, or 5x10^15) unique n-grams.

Larger-N is Better


原 PPT 独立图片资源
Larger-N is Better

原 PPT 独立图片资源
https://infini-gram.io/

But the world always surprise you


原 PPT 独立图片资源
But the world always surprise you

Shakespeare Corpus: 884,647 words , voc size =29,066;
Shakespeare produced 300,000 bigram types out of 844 million possible bigrams;
So 99.96% of the possible bigrams were never seen in the corpus ;
That sparsity is even worse for 4-grams;
But you may see them in applications.

2.1 Statistical Language Modeling 2.1.3 Generalization to Unseen Words



原 PPT 独立图片资源

2.1 Statistical Language Modeling 2.1.3 Generalization to Unseen Words

原 PPT 独立图片资源

Zero Probability Problem


原 PPT 独立图片资源
Zero Probability Problem

Training:
… ate lunch
… ate dinner
… ate a
… ate the
𝑃(𝑏𝑟𝑒𝑎𝑘𝑓𝑎𝑠𝑡 | 𝑎𝑡𝑒) = 0

Application: Probability

𝑃(𝐼 𝑎𝑡𝑒 𝑏𝑟𝑒𝑎𝑘𝑓𝑎𝑠𝑡) = 0
Application: Generation
Cannot generate breakfast

Smoothing


原 PPT 独立图片资源
Smoothing

We have sparse statistics

allegations
reports
claims
attack
request
man
outcome
…

allegations


attack

man
outcome
…
allegations
reports
claims
request



Steal probability mass to generalize better

Add-one Estimation (Laplace smoothing)


原 PPT 独立图片资源
Add-one Estimation (Laplace smoothing)

Pretend we saw each word one more time than we did
Just add one to all the counts!

MLE estimate:

Add-1 estimate:
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Backoff and Interpolation


原 PPT 独立图片资源
Backoff and Interpolation

More context length results in more Sparsity;
Less context reduces the Zero probability problem;
Condition on less context for contexts you know less about
Backoff:
use trigram if you have good evidence,
otherwise bigram, otherwise unigram
Interpolation:
mix unigram, bigram, trigram
Interpolation works better

Linear Interpolation


原 PPT 独立图片资源
Linear Interpolation

Simple interpolation


Lambdas conditional on context:
原 PPT 独立图片资源
原 PPT 独立图片资源
interp2

Backoff


原 PPT 独立图片资源
Backoff

Suppose you want:
𝑃(𝑝𝑎𝑛𝑐𝑎𝑘𝑒𝑠| 𝑑𝑒𝑙𝑖𝑐𝑖𝑜𝑢𝑠 𝑠𝑜𝑢𝑓𝑓𝑙é)
If the trigram probability is 0, use the bigram
𝑃(𝑝𝑎𝑛𝑐𝑎𝑘𝑒𝑠| 𝑠𝑜𝑢𝑓𝑓𝑙é)
If the bigram probability is 0, use the unigram
𝑃(𝑝𝑎𝑛𝑐𝑎𝑘𝑒𝑠)
Complication: need to discount the higher-order ngram so probabilities don't sum higher than 1 (e.g., Katz backoff)

Stupid Backoff


原 PPT 独立图片资源
Stupid Backoff

Backoff without discounting (not a true probability)
原 PPT 独立图片资源
原 PPT 独立图片资源

2.1 Statistical Language Modeling 2.1.4 Generation



原 PPT 独立图片资源

2.1 Statistical Language Modeling 2.1.4 Generation

原 PPT 独立图片资源

Generating Language using N-Gram Model


原 PPT 独立图片资源
Generating Language using N-Gram Model

The Shannon (1948) Visualization Method
Unigram



Bigram
REPRESENTING AND SPEEDILY IS AN GOOD APT OR COME CAN DIFFERENT NATURAL HERE HE THE A IN CAME THE TO OF TO EXPERT GRAY COME TO FURNISHES THE LINE MESSAGE HAD BE THESE.
THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED.
原 PPT 独立图片资源

How Shannon sampled those words in 1948


原 PPT 独立图片资源
How Shannon sampled those words in 1948

"Open a book at random and select a letter at random on the page. This letter is recorded. The book is then opened to another page and one reads until this letter is encountered. The succeeding letter is then recorded. Turning to another page this second letter is searched for and the succeeding letter recorded, etc."
Stack of open books

Sampling a word from a distribution


原 PPT 独立图片资源
Sampling a word from a distribution

原 PPT 独立图片资源

Visualizing Bigrams the Shannon Way


原 PPT 独立图片资源
Visualizing Bigrams the Shannon Way

Choose a random bigram (<𝑠>, 𝑤) according to its probability 𝑝(𝑤|<𝑠>);
Now choose a random bigram (𝑤, 𝑥)  according to its probability 𝑝(𝑥|𝑤);
And so on until we choose </𝑠>;
Then string the words together
<s> I
I want
want to
to eat
eat Chinese
Chinese food
food </s>
I want to eat Chinese food

Approximating Shakespeare


原 PPT 独立图片资源
Approximating Shakespeare

原 PPT 独立图片资源

The Wall Street Journal


原 PPT 独立图片资源
The Wall Street Journal

原 PPT 独立图片资源

2.1 Statistical Language Modeling 2.1.5 Evaluation



原 PPT 独立图片资源

2.1 Statistical Language Modeling 2.1.5 Evaluation

原 PPT 独立图片资源

Evaluation of N-Gram Model


原 PPT 独立图片资源
Evaluation of N-Gram Model

Extrinsic Evaluation:
Evaluate the model on external Task
Intrinsic Evaluation:
Directly measures language model performance at predicting words.
Doesn't necessarily correspond with real application performance
But gives us a single general metric for language models

Recall: Evaluating the Machine Learning Algorithms


原 PPT 独立图片资源
Recall: Evaluating the Machine Learning Algorithms

Training Data: Used to train a machine given a hyper-parameter setting;
Development Data: Evaluate the trained machines to select the best hyper-parameter setting;
Testing Data: Evaluate the real performance of the selected machine;
Datasets should be with the same distribution with the final task;
Datasets should be no overlap. Do not test on training data!
Train Data
Dev Data
Test Data

What is a Good Language Model


原 PPT 独立图片资源
What is a Good Language Model

A good LM prefers the next word that actually occurs;
Assign higher probability to “time” given “Once upon a _”;
Assign near zero probability to “and” given “Once upon a _”;
A good LM prefers “real” sentences:
Assign higher probability to “real” or “frequently observed” sentences
Assigns lower probability to “word salad” or “rarely observed” sentences

How to Evaluate N-Gram Model on a Dataset


原 PPT 独立图片资源
How to Evaluate N-Gram Model on a Dataset

The test set represents the real sentences;
The best LM assigns high probability to the entire test set.
A good performance metric for LM:
𝑝(𝑡𝑒𝑠𝑡 𝑠𝑒𝑡)
When comparing two LMs, A and B
We compute 𝑃𝐴(𝑡𝑒𝑠𝑡 𝑠𝑒𝑡) and 𝑃𝐵(𝑡𝑒𝑠𝑡 𝑠𝑒𝑡)
The one with larger probability is better.

The Problem of Probability as a Metric


原 PPT 独立图片资源
The Problem of Probability as a Metric

Probability of test set depends on the number of tokens in it:
Sentence probability gets smaller for longer sentences
Dataset probability gets smaller for larger dataset;
We’d prefer a metric that is per-word, normalized by length, so we could compare across texts of different lengths.

Perplexity


原 PPT 独立图片资源
Perplexity

Perplexity is the inverse probability of the test set, normalized by the number of words;





Minimizing perplexity is the same as maximizing probability
原 PPT 独立图片资源
pp2
pp3
Bigrams:
Chain rule:

Perplexity for Sentence and Corpus


原 PPT 独立图片资源
Perplexity for Sentence and Corpus

Compute the log likelihood for each sentence
Given a sentence 𝑆 = 𝑤1, 𝑤2, …, 𝑤𝑁, the log perplexity per word is:
𝐿𝑜𝑔 𝑃𝑃𝐿(𝑆)= −1𝑁∑𝑖=1𝑁  𝐿𝑜𝑔 𝑃(𝑤𝑖 |𝑤1, …, 𝑤𝑁−1)
Compute the perplexity for each sentence
𝑃𝑃𝐿(𝑆) = 𝑒𝑥𝑝(−1𝑁∑𝑖=1𝑁𝐿𝑜𝑔 𝑃(𝑤𝑖 |𝑤1, …, 𝑤𝑁−1))
Aggregate over all sentences in the corpus
Compute the total log-likelihood across all tokens then normalize by the total number of tokens:
𝑃𝑃𝐿𝐶𝑜𝑟𝑝𝑢𝑠 =𝑒𝑥𝑝( −1𝑁𝑡𝑜𝑡𝑎𝑙∑𝑆∈𝐶𝑜𝑟𝑝𝑢𝑠∑𝑖=1𝑁𝑆𝐿𝑜𝑔 𝑃(𝑤𝑖 |𝑤1, …, 𝑤𝑁𝑆−1)) 
where 𝑁𝑡𝑜𝑡𝑎𝑙 is the total number of tokens in the dataset.

Empirical Studies


原 PPT 独立图片资源
Empirical Studies

WSJ Dataset:
the Wall Street Journal newspaper
Training 38 million words, test 1.5 million words

Unigram
Bigram
Trigram
Perplexity
962
170
109

Natural Language UnderstandingTransformer


原 PPT 独立图片资源
Natural Language
Understanding

Transformer
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Background
Transformer Overview
Transformer Block
Transformer Architecture
Challenges and Solutions

2.2 Transformer 2.2.1 Background: Neural Machine Translation and RNNs



原 PPT 独立图片资源

2.2 Transformer 2.2.1 Background: Neural Machine Translation and RNNs

原 PPT 独立图片资源

Machine Translation


原 PPT 独立图片资源
Machine Translation
Machine Translation (MT) is the task of translating a sentence 𝑥 from one language (the source language) to a sentence 𝑦 in another language (the target language).

x: L'homme est né libre, et partout il est dans les fers

y: Man is born free, but everywhere he is in chains
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

History


原 PPT 独立图片资源
History
Early Stage (1950s): Basically just simple rule-based systems doing word substitution. Problem soon appeared intractable.
Statistical Machine Translation (1990s-2010s):
Learn a probabilistic model from data:
𝑎𝑟𝑔𝑚𝑎𝑥𝑦𝑃(𝑦|𝑥)=𝑎𝑟𝑔𝑚𝑎𝑥𝑦𝑃(𝑥|𝑦)𝑃(𝑦)
The best systems were extremely complex, with many separately-designed subcomponents, which requires lots of feature engineering and lots of human effort to maintain
Neural Machine Translation (2010s-)
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

Neural Machine Translation


原 PPT 独立图片资源
Neural Machine Translation
In 2014, Ilya Sutskever applied Sequence to Sequence RNN model to MT, starting the NMT.
NMT learns a Conditional Language Model 𝑃(𝑦|𝑥):
Language Model: the decoder is predicting the next word of the target sentence 𝑦 
Conditional: its predictions are also conditioned on the source sentence 𝑥

𝑃(𝑦|𝑥)=𝑃(𝑦1|𝑥)𝑃(𝑦2|𝑦1,𝑥)𝑃(𝑦3|𝑦2,𝑦1,𝑥)⋯𝑃(𝑦𝑇|𝑦1,⋯,𝑦𝑇−1,𝑥)
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

Sequence to Sequence RNN


原 PPT 独立图片资源
Sequence to Sequence RNN

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

Training the System


原 PPT 独立图片资源
Training the System
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

Sequence to Sequence Learning in NMT


原 PPT 独立图片资源
Sequence to Sequence Learning in NMT
Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to Sequence Learning with Neural Networks.
原 PPT 独立图片资源
原 PPT 独立图片资源

Bottlenecks


原 PPT 独立图片资源
Bottlenecks
Does the architecture limit the learning ability?
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

Limitations: Linear Interaction Distance


原 PPT 独立图片资源
Limitations: Linear Interaction Distance
O(sequence length) steps for distant word pairs to interact means:
Hard to learn long-distance dependencies (because gradient problems!)
Linear order of words is “baked in”; we already know linear order isn’t the right way to think about sentences…
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

Limitations: Lack of Parallelizability


原 PPT 独立图片资源
Limitations: Lack of Parallelizability
Forward/backward have O(sequence length) unparallelizable operations
Inhibits training on very large datasets!
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

2.2 Transformer 2.2.2 Background: Attention



原 PPT 独立图片资源

2.2 Transformer 2.2.2 Background: Attention

原 PPT 独立图片资源

Attention: Core Idea


原 PPT 独立图片资源
Attention: Core Idea
Attention provides a solution to above limitations

On each step of the decoder, use direct connection to the encoder to focus on a particular part of the source sequence.
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf

Starting from Meaning-Pooling for RNNs


原 PPT 独立图片资源
Starting from Meaning-Pooling for RNNs
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

Attention is weighted averaging


原 PPT 独立图片资源
Attention is weighted averaging
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源
In attention, the query matches all keys softly,
to a weight between 0 and 1. The keys’ values
are multiplied by the weights and summed.
In a lookup table, we have a table of keys that map to values. The query matches one of the keys, returning its value.

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Sequence-to-Sequence With Attention


原 PPT 独立图片资源
Sequence-to-Sequence With Attention
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Core Idea: use direct connection to the encoder to focus on a particular part of the source sequence.
原 PPT 独立图片资源

Attention: Formulation


原 PPT 独立图片资源
Attention: Formulation
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Given encoder hidden states: 𝒉1,⋯,𝒉𝑁∈ℝh
Decoder hidden state on timestep 𝑡: 𝒔𝑡∈ℝh
The attention scores for this step: 𝒆𝑡=[𝒔𝑡𝑇𝒉1,⋯,𝒔𝑡𝑇𝒉𝑁]∈ℝ𝑁
The attention distribution: 𝜶𝑡=softmax(𝒆𝑡)∈ℝ𝑁
The attention output by weighted sum: 𝒂𝑡=Σ𝑖=1𝑁𝛼𝑖𝑡𝒉𝑖∈ℝh
Final decoder hidden state: [𝒂𝑡;𝒔𝑡]∈ℝ2h

Attention solves Limitations of RNN


原 PPT 独立图片资源
Attention solves Limitations of RNN
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
Number of unparallelizable operations does not increase with sequence length.
Maximum interaction distance: O(1), since all words interact at every layer
原 PPT 独立图片资源

Attention becomes a General Technique


原 PPT 独立图片资源
Attention becomes a General Technique

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
More general definition of attention:
Given a set of vector values, and a vector query, attention is a technique to compute a weighted sum of the values, dependent on the query.

Intuition:
The weighted sum is a selective summary of the information contained in the values, where the query determines which values to focus on.
Attention is a way to obtain a fixed-size representation of an arbitrary set of representations (the values), dependent on some other representation (the query).

Attention vs RNN


原 PPT 独立图片资源
Attention vs RNN

Feature
Self-Attention
Traditional RNN
Processing Style
✅ Processes all tokens simultaneously (parallel)
❌Processes tokens one by one (sequential)
Information Flow
✅ Any two tokens connect in 1 step
❌Long-range dependencies require many steps
Parallelism
✅ Fully parallelizable (fast)
❌ Inherently sequential (slow)
Dependency Modeling
✅ Excellent at long-range dependencies
❌Poor at long-range dependencies (vanishing gradients)
Interpretability
✅ Attention weights show what is focused on
❌ Hard to interpret hidden states
Linearity
❌ Linear
✅ Nonlinear
Order Sensitive
❌ No
✅ Yes

2.2 Transformer 2.2.3 Transformer Overview



原 PPT 独立图片资源

2.2 Transformer 2.2.3 Transformer Overview

原 PPT 独立图片资源

Motivation


原 PPT 独立图片资源
Motivation
Attention had to be used with RNN, so the architecture RNN+Attention still suffers from RNN’s limitation;
Key Idea:
Why Not Just Use Attention?
Difficulties:
How can Attention used as Encoder?
Attention is Linear
Attention omits the order information
The Solution:
Transformer Architecture

Attention is All You Need (2017)


原 PPT 独立图片资源
Attention is All You Need (2017)

原 PPT 独立图片资源

NMT Task


原 PPT 独立图片资源
NMT Task

原 PPT 独立图片资源
https://jalammar.github.io/illustrated-transformer/

Decoder-Encoder


原 PPT 独立图片资源
Decoder-Encoder

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Stack of Layers


原 PPT 独立图片资源
Stack of Layers

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Encoder Layer


原 PPT 独立图片资源
Encoder Layer

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Decoder Layer


原 PPT 独立图片资源
Decoder Layer

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

2.2 Transformer 2.2.4 Transformer Blocks



原 PPT 独立图片资源

2.2 Transformer 2.2.4 Transformer Blocks

原 PPT 独立图片资源

Input Embedding


原 PPT 独立图片资源
Input Embedding

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Embedding in the Block


原 PPT 独立图片资源
Embedding in the Block

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

2.2 Transformer 2.2.5 Self-Attention



原 PPT 独立图片资源

2.2 Transformer 2.2.5 Self-Attention

原 PPT 独立图片资源

Self-Attention


原 PPT 独立图片资源
Self-Attention

https://jalammar.github.io/illustrated-transformer/
”The animal didn't cross the street because it was too tired”
For each word, what is the relationship between me and other words?
原 PPT 独立图片资源

Mechanism Design: Q, K, V


原 PPT 独立图片资源
Mechanism Design: Q, K, V

Each token acts like a “questioner,” using its own Query to “ask” the Keys of other tokens, and then obtains a weighted combination of their Values — which becomes the final attention output.
📌 Query (Q)
Represents the type of information the token at the current position is seeking.
For example, if the current token is “ate,” it might want to know: “Who did the eating?”
📌 Key (K)
Represents the “tag” or “description” of the information each token provides.
For instance, the token “He” might have a key representing “subject-type pronoun,” so it will be matched accordingly.
📌 Value (V)
Is the actual content or information provided by each token, and will be selected as part of the final output.
The token “He” might provide the actual semantic content like “male third-person singular subject” — this is the Value.
Q and K determine who you pay attention to, but you ultimately receive is the information provided by those you pay attention to — that is, the Values (V).

Mechanism Design: Q, K, V


原 PPT 独立图片资源
Mechanism Design: Q, K, V

Let 𝒘1:𝑛 be a sequence of words in vocabulary 𝑉, like Zuko made his uncle tea.
For each 𝒘𝑖 , let 𝒙𝑖 = 𝐸𝒘𝒊 , where 𝐸 ∈ ℝ𝑑×|𝑉|is an embedding matrix.
Transform each word embedding with weight matrices 𝑄, 𝐾, 𝑉 , each in ℝ𝑑×𝑑
𝒒𝑖 = 𝑄𝒙𝒊 (𝑞𝑢𝑒𝑟𝑖𝑒𝑠)       𝒌𝑖 = 𝐾𝒙𝒊 (𝑘𝑒𝑦𝑠)        𝒗𝑖 = 𝑉𝒙𝒊 (𝑣𝑎𝑙𝑢𝑒𝑠)
Compute pairwise similarities between keys and queries; normalize with softmax
𝒆𝑖𝑗 = 𝒒𝒊⊤𝒌𝒋       𝜶𝑖𝑗 =exp⁡(𝒆𝑖𝑗)/Σ𝑗′ exp⁡(𝒆𝑖𝑗′ ) 
Compute output for each word as weighted sum of values
𝒐𝑖 =Σ𝒋 𝜶𝑖𝑗 𝒗𝑖�

Implementation


原 PPT 独立图片资源
Implementation

原 PPT 独立图片资源
https://jalammar.github.io/illustrated-transformer/

Implementation


原 PPT 独立图片资源
Implementation

原 PPT 独立图片资源
https://jalammar.github.io/illustrated-transformer/

Implementation


原 PPT 独立图片资源
Implementation

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Implementation


原 PPT 独立图片资源
Implementation

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Matrix Calculation


原 PPT 独立图片资源
Matrix Calculation

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Matrix Calculation


原 PPT 独立图片资源
Matrix Calculation

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Multi-Head


原 PPT 独立图片资源
Multi-Head

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源
原 PPT 独立图片资源

2.2 Transformer 2.2.6 Feed-Forward Networks



原 PPT 独立图片资源

2.2 Transformer 2.2.6 Feed-Forward Networks

原 PPT 独立图片资源

Feed-Forward


原 PPT 独立图片资源
Feed-Forward

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Feed-Forward


原 PPT 独立图片资源
Feed-Forward

𝑚𝑖 = 𝑀𝐿𝑃(output𝑖)
= 𝑊2 ∗ 𝑅𝑒𝐿𝑈( 𝑊1 output𝑖 + 𝑏1) + 𝑏2
Attention decides “who is important”, while Feedforward decides “how to interpret this information.”
Expanding dimensions gives the model more room for transformation
Reducing dimensions back to the original size for residual connection

2.2 Transformer 2.2.7 Positional Encoding



原 PPT 独立图片资源

2.2 Transformer 2.2.7 Positional Encoding

原 PPT 独立图片资源

From hidden Order Info to Position Feature


原 PPT 独立图片资源
From hidden Order Info to Position Feature

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

The Embedding of Position


原 PPT 独立图片资源
The Embedding of Position

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源
Consider representing each sequence index as a vector:
𝒑𝑖 ∈ ℝ𝑑 , for 𝑖 ∈ {1,2, … , 𝑛}
Easy to incorporate into our self-attention block: just add the 𝒑𝑖 to our inputs!
Recall that 𝒙𝑖 is the embedding of the word at index 𝑖. The positioned embedding is:
𝒙𝑖 = 𝒙𝑖 + 𝒑

The Embedding of Position


原 PPT 独立图片资源
The Embedding of Position

https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/
原 PPT 独立图片资源

Sinusoidal Position Representations


原 PPT 独立图片资源
Sinusoidal Position Representations

https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/
原 PPT 独立图片资源
原 PPT 独立图片资源

Why this Form?


原 PPT 独立图片资源
Why this Form?

https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/
原 PPT 独立图片资源

Visualization


原 PPT 独立图片资源
Visualization

https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/
原 PPT 独立图片资源
The positional encoding matrix for n=10,000, d=512, sequence length=100

2.2 Transformer 2.2.8 Masking the Future



原 PPT 独立图片资源

2.2 Transformer 2.2.8 Masking the Future

原 PPT 独立图片资源

Decoder: Mask the Future


原 PPT 独立图片资源
Decoder: Mask the Future

原 PPT 独立图片资源

原 PPT 独立图片资源
When applying self-attention in decoder, we must ensure we cannot depend on future.

2.2 Transformer 2.2.9 Training Stability



原 PPT 独立图片资源

2.2 Transformer 2.2.9 Training Stability

原 PPT 独立图片资源

Residual and Normalization


原 PPT 独立图片资源
Residual and Normalization

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Residual and Normalization


原 PPT 独立图片资源
Residual and Normalization

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源
原 PPT 独立图片资源

2.2 Transformer 2.2.10 Transformer Block Summary



原 PPT 独立图片资源

2.2 Transformer 2.2.10 Transformer Block Summary

原 PPT 独立图片资源

Building Blocks


原 PPT 独立图片资源
Building Blocks

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源
Self-attention:
the basis of the method.
Position representations:
Specify the sequence order, since self-attention is an unordered function of its inputs.
Nonlinearities:
At the output of the self-attention block
Frequently implemented as a simple feedforward network.
Masking:
Parallelize operations while not looking at the future.
Keeps information about the future from “leaking” to the past.

2.2 Transformer 2.2.11 Transformer Architecture



原 PPT 独立图片资源

2.2 Transformer 2.2.11 Transformer Architecture

原 PPT 独立图片资源

Encoder and Decoder


原 PPT 独立图片资源
Encoder and Decoder

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源
Encoder
Decoder
原 PPT 独立图片资源
No Masking!

Encoder-Decoder


原 PPT 独立图片资源
Encoder-Decoder

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源

The embedding produced by Decoder should be conditional on the Encoder embedding
The Cross Attention:
Encoder Embedding K, V;
Decoder Embedding Q

Encoder-Decoder Illustration


原 PPT 独立图片资源
Encoder-Decoder Illustration

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

Encoder-Decoder Illustration


原 PPT 独立图片资源
Encoder-Decoder Illustration

https://jalammar.github.io/illustrated-transformer/
原 PPT 独立图片资源

2.2 Transformer 2.2.12 Position Representations



原 PPT 独立图片资源

2.2 Transformer 2.2.12 Position Representations

原 PPT 独立图片资源

Sinusoidal Position Representations


原 PPT 独立图片资源
Sinusoidal Position Representations

原 PPT 独立图片资源
Pros:
Periodicity indicates that maybe “absolute position” isn’t as important
Maybe can extrapolate to longer sequences as periods restart!
Cons:
Not learnable; also the extrapolation doesn’t really work!

Learnable Position Representations


原 PPT 独立图片资源
Learnable Position Representations

Let all 𝒑𝑖 be learnable parameters! Learn a matrix 𝒑 ∈ ℝ𝑑×𝑛 , and let each 𝒑𝑖 be a column of that matrix!
Pros:
Flexibility: each position gets to be learned to fit the data
Cons:
Definitely can’t extrapolate to indices outside 1, … , 𝑛.
Widely used in many system;
From absolute position to relative position: 𝒑𝑖,𝒑𝑗→𝒓𝑖𝑗
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf

RoPE –Rotary Position Embedding


原 PPT 独立图片资源
RoPE –Rotary Position Embedding

Relative position should be some 𝑓(𝑥, 𝑖) s.t.
<𝑓(𝑥, 𝑖), 𝑓(𝑦,𝑗)> = 𝑔(𝑥, 𝑦, 𝑖 − 𝑗) 
That is, the attention function only gets to depend on the relative position (𝑖−𝑗) .
Relative position equivariance → Rotation
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源

RoPE in High Dimension


原 PPT 独立图片资源
RoPE in High Dimension

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf
原 PPT 独立图片资源

2.2 Transformer 2.2.13 Context Length



原 PPT 独立图片资源

2.2 Transformer 2.2.13 Context Length

原 PPT 独立图片资源

Beyond Fixed-Length Context


原 PPT 独立图片资源
Beyond Fixed-Length Context

The matrix computations in Transformer need Fixed-Length Context
Limited context hampers performance on long texts
Key use cases:
Long documents
Conversations
Streaming text
Agents
…
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture08-transformers.pdf

Approaches


原 PPT 独立图片资源
Approaches

Why Transformer cannot cope with very long context?
The attention complexity is 𝑂(𝑛2)
Approaches:
Sparse Attention
Linear approximation of Attention: Back of RNN

Sparse Attention


原 PPT 独立图片资源
Sparse Attention

原 PPT 独立图片资源
https://research.google/blog/constructing-transformers-for-longer-sequences-with-sparse-attention-methods/

Linear Approximation


原 PPT 独立图片资源
Linear Approximation

https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state
原 PPT 独立图片资源
Scaling Laws for Mamba vs other Language Models
https://thegradient.pub/mamba-explained/

2.2 Transformer 2.2.14 Transferability of Modifications



原 PPT 独立图片资源

2.2 Transformer 2.2.14 Transferability of Modifications

原 PPT 独立图片资源

Do Transformer Modifications Transfer?


原 PPT 独立图片资源
Do Transformer Modifications Transfer?

"Surprisingly, we find that most modifications do not meaningfully improve performance."
原 PPT 独立图片资源
原 PPT 独立图片资源

2.2 Transformer 2.2.15 Suggested Readings



原 PPT 独立图片资源

2.2 Transformer 2.2.15 Suggested Readings

原 PPT 独立图片资源

Papers and Articles


原 PPT 独立图片资源
Papers and Articles
Attention Is All You Need https://arxiv.org/abs/1706.03762

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源

Natural Language UnderstandingNeural Pretrained Language Modeling


原 PPT 独立图片资源
Natural Language
Understanding

Neural Pretrained
Language Modeling
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
ELMO: When Language Model Meets Word Embedding
GPT: Language Model + Transformer
BERT: Masked Language Model + Transformer
T5: Pretrained Encoder-Decoder Transformer
Pretrained Language Models Gallery

2.3 Pretrained Language Models 2.3.1 Language Models and Word Embeddings



原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.1 Language Models and Word Embeddings

原 PPT 独立图片资源

Recall: Word Embedding


原 PPT 独立图片资源
Recall: Word Embedding
Word2Vec: Predict occurrence of neighborhood;
GloVe: Approximate the co-occurrence information:
Each word has an embedding vector;
Representing the meaning learned from the global environment.
原 PPT 独立图片资源
原 PPT 独立图片资源

Recall: Word Embedding


原 PPT 独立图片资源
Recall: Word Embedding
Global Environment:
Language is bag of words.
Co-occurrence is the statistical connection.
Polysemy words need local environment.
Apple (fruit/company)
打(打游戏/打人/打酱油)
More thorough modeling of language?
原 PPT 独立图片资源
原 PPT 独立图片资源

Recall: Probabilistic Language Model


原 PPT 独立图片资源
Recall: Probabilistic Language Model
The more thorough language modeling over Bag of Words?
Probabilistic Language Model:



Maximize the likelihood of language by optimizing the word embedding!
How to do that?
原 PPT 独立图片资源

Recall: Sequence to Sequence RNN


原 PPT 独立图片资源
Recall: Sequence to Sequence RNN
Sequence to Sequence RNN (e.g. LSTM or Transformer) is proposed to perform sequence transform (e.g. From French to English);
It maps each word/token into a local embedding (dependent on its neighborhood /history words);
But this embedding is task dependent, not for the accurate meaning representation in language, so not usable for other tasks.
We need to focus on language itself, not specific tasks.

Learning Context-dependent Meaning Embeddings


原 PPT 独立图片资源
Learning Context-dependent Meaning Embeddings

原 PPT 独立图片资源
https://www.cs.utexas.edu/~gdurrett/courses/fa2022/lectures/lec20-1pp.pdf

ELMO: Embeddings from Language Models (AllenAI, 2018)


原 PPT 独立图片资源
ELMO: Embeddings from Language Models
(AllenAI, 2018)
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture06-fancy-rnn.pdf
原 PPT 独立图片资源

ELMO - Training


原 PPT 独立图片资源
ELMO - Training
https://jalammar.github.io/illustrated-bert/
原 PPT 独立图片资源
Training for 10 epochs on the 1B Word Benchmark

ELMO – Embedding


原 PPT 独立图片资源
ELMO – Embedding
https://jalammar.github.io/illustrated-bert/
原 PPT 独立图片资源

ELMO – Supervised Finetuning


原 PPT 独立图片资源
ELMO – Supervised Finetuning
https://naturale0.github.io/2021/02/24/Understanding-ELMo
原 PPT 独立图片资源

ELMO - Performance


原 PPT 独立图片资源
ELMO - Performance
https://arxiv.org/pdf/1802.05365v2
原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.2 Why Pretrained Models



原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.2 Why Pretrained Models

原 PPT 独立图片资源

Recall: Pretraining


原 PPT 独立图片资源
Recall: Pretraining
原 PPT 独立图片资源
1.Hinton, G. E. & Salakhutdinov, R. R. Reducing the Dimensionality of Data with Neural Networks. Science 313, 504–507 (2006).

Pretraining


原 PPT 独立图片资源
Pretraining
Unsupervised (or Self-supervised) Learn from a data-set;
Aims to recover the prior-distribution of the data;
Boltzmann Model for Image
Language Model of Language
Do not apply any further supervision! They harm the generalization!
Train a neural network to learn the distribution and save the weight.
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf

Pretraining-Finetuning Paradigm


原 PPT 独立图片资源
Pretraining-Finetuning Paradigm
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf
原 PPT 独立图片资源

Why Pretrained Model?


原 PPT 独立图片资源
Why Pretrained Model?
https://arxiv.org/pdf/1802.05365v2
Representation Learning
Sample Complexity
Optimization Theory
Bayesian Inference
Information Bottleneck

Representation Learning


原 PPT 独立图片资源
Representation Learning
Pretraining serves as a form of unsupervised or self-supervised representation learning, which helps a model learn useful features from large-scale data.
In terms of statistical learning theory, this process reduces the hypothesis space the model must explore during downstream tasks.
Let 𝐻 be the hypothesis space of a model.
After pretraining, the model selects a subspace 𝐻pretrained⊂ 𝐻 that is better aligned with natural data distributions.
The learning problem becomes easier and requires fewer labeled examples to generalize well.
This aligns with the bias-variance tradeoff: pretraining imposes a useful inductive bias (i.e., structural prior about the task or data distribution).

Data Efficiency and Transfer Learning


原 PPT 独立图片资源
 Data Efficiency and Transfer Learning
In settings with limited labeled data, training from scratch leads to overfitting due to high model complexity.
Pretrained models already encode generic knowledge, so fine-tuning requires only small amounts of data.
This can be analyzed through PAC-learning or sample complexity:
Pretraining reduces the sample complexity required to learn a task to within a desired generalization error 𝜖 .
Formally, the number of labeled examples needed scales with the effective capacity of the hypothesis class consistent with the pretrained features, which is often much smaller.

Optimization Landscape Improvement


原 PPT 独立图片资源
Optimization Landscape Improvement
Deep neural networks suffer from highly non-convex loss surfaces with many local minima and saddle points. Pretraining provides a good initialization:
Gradient descent from a pretrained state converges faster and often to better local minima.
This is linked to ideas from meta-learning: the pretrained model is already near a “task manifold” in parameter space.
The benefit can also be seen from the perspective of energy-based models where pretraining shapes the energy landscape so that desirable modes (i.e., good solutions) are more easily reachable.

Bayesian View: Prior Knowledge


原 PPT 独立图片资源
Bayesian View: Prior Knowledge
From a Bayesian perspective, pretraining can be interpreted as providing a prior distribution over model parameters 𝑝(𝜃),learned from large unlabeled data.
Fine-tuning becomes a posterior inference given a new (supervised) task and its data D:
𝑝(𝜃| 𝐷)∝ 𝑝(𝐷∣𝜃) 𝑝(𝜃)
The pretrained model provides a strong informative prior that narrows the posterior and improves generalization on new tasks.
This view also supports the empirical success of few-shot and zero-shot learning in pretrained models like GPT and BERT.

Information Bottleneck Theory


原 PPT 独立图片资源
Information Bottleneck Theory
The Information Bottleneck (IB) principle suggests learning representations that compress the input while retaining task-relevant information.
Pretraining can be viewed as maximizing mutual information 𝐼(𝑧; 𝑥) between inputs x and intermediate representations z, while minimizing 𝐼(𝑧; 𝑦) only later in fine-tuning.
This staged process can improve generalization by avoiding overfitting in early stages.

Why Pretrained Model?


原 PPT 独立图片资源
Why Pretrained Model?
Perspective
Pretraining Benefit
Representation Learning
Narrows hypothesis space, induces task-relevant structure
Sample Complexity
Reduces labeled data needed for good generalization
Optimization Theory
Provides better initialization for gradient descent
Bayesian Inference
Acts as a learned prior, guiding parameter estimation
Information Bottleneck
Encourages compressed but expressive representations

Think: When we Need Pretraining?


原 PPT 独立图片资源
Think: When we Need Pretraining?
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf
AGI unifies all task in Dialog/QA scenario.
Can we just train a supervised model using all Dialog/QA data?
原 PPT 独立图片资源

2018: Year of Pretrained Model


原 PPT 独立图片资源
2018: Year of Pretrained Model
AI2 made ELMo in spring 2018;
GPT (transformer-based ELMo) was released in summer 2018;
BERT came out October 2018.

The Pretrain Revolution


原 PPT 独立图片资源
The Pretrain Revolution
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf

The Pretrain Revolution


原 PPT 独立图片资源
The Pretrain Revolution
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf
原 PPT 独立图片资源

The Pretrain Revolution


原 PPT 独立图片资源
The Pretrain Revolution
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf
原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.3 GPT: Autoregressive Pretraining



原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.3 GPT: Autoregressive Pretraining

原 PPT 独立图片资源

GPT (Open AI, 2018)


原 PPT 独立图片资源
GPT (Open AI, 2018)
https://medium.com/data-science/understanding-the-evolution-of-gpt-part-1-an-in-depth-look-at-gpt-1-and-what-inspired-it-b7388a32e87d
原 PPT 独立图片资源
Trained on BooksCorpus dataset (7000 books), 3 epochs.

GPT - Training


原 PPT 独立图片资源
GPT - Training
Back of the fixed context window – k:


Trained on BooksCorpus dataset (7000 books), 3 epochs.
原 PPT 独立图片资源

GPT- Supervised Finetuning


原 PPT 独立图片资源
GPT- Supervised Finetuning
Improving Language Understanding by Generative Pre-Training, Open AI, 2018
原 PPT 独立图片资源

GPT- Performance


原 PPT 独立图片资源
GPT- Performance
Improving Language Understanding by Generative Pre-Training, Open AI, 2018
原 PPT 独立图片资源

GPT: Pretraining + Transformer


原 PPT 独立图片资源
原 PPT 独立图片资源
GPT: Pretraining + Transformer
Improving Language Understanding by Generative Pre-Training, Open AI, 2018


GPT 1, GPT 2, and GPT 3


原 PPT 独立图片资源
GPT 1, GPT 2, and GPT 3
https://360digitmg.com/blog/types-of-gpt-in-artificial-intelligence
 
GPT-1
GPT-2
GPT-3
Parameters
117 Million
1.5 Billion
175 Billion
Decoder Layers
12
48
96
Context Token Size
512
1024
2048
Hidden Layer
768
1600
12288
Batch Size
64
512
3.2M

GPT 2: Increasingly convincing generations


原 PPT 独立图片资源
GPT 2: Increasingly convincing generations
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf

GPT 3: Very large models and In-context learning


原 PPT 独立图片资源
GPT 3: Very large models and In-context learning
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf
Perform some kind of learning without gradient steps simply from examples you provide within their contexts.
原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.4 BERT: Masked Language Modeling



原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.4 BERT: Masked Language Modeling

原 PPT 独立图片资源

Lets focus on Embedding


原 PPT 独立图片资源
Lets focus on Embedding
https://www.baeldung.com/cs/word-embeddings-cbow-vs-skip-gram

原 PPT 独立图片资源

Language Modeling and Embedding


原 PPT 独立图片资源
Language Modeling and Embedding
Recall: Words are defined by their environments (the words around them);
CBOW and Skip-Gram(Word2Vec): Full environment;
Language Model: Partial Environment (Only History)
Language Model + RNN/Transformer = ELMO/GPT
CBOW/Skip-Gram + RNN/Transformers = ?

From CBOW/SG to Masked Language Model


原 PPT 独立图片资源
From CBOW/SG to Masked Language Model
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/

Masked Language Model


原 PPT 独立图片资源
Masked Language Model
https://slds-lmu.github.io/seminar_nlp_ss20/transfer-learning-for-nlp-ii.html
Autoencoding

原 PPT 独立图片资源

BERT (Bidirectional Encoder Representations from Transformers) = MLM+ Transformers


原 PPT 独立图片资源
BERT (Bidirectional Encoder Representations from Transformers) = MLM+ Transformers
BERT implement following two tasks using Transformer:
Masked Language Modeling


Next Sentence Prediction
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/

Input Representation


原 PPT 独立图片资源
Input Representation
原 PPT 独立图片资源
https://d2l.ai/chapter_natural-language-processing-pretraining/bert.html

Masked Token Prediction


原 PPT 独立图片资源
Masked Token Prediction
https://jalammar.github.io/illustrated-bert/
原 PPT 独立图片资源

How to Mask?


原 PPT 独立图片资源
How to Mask?
15% of tokens in each sequence will be randomly masked.
How to mask?
80% of the time: Replace the word with the [MASK] token, e.g., my dog is hairy ⇒ my dog is [MASK]
10% of the time: Replace the word with a random word, e.g., my dog is hairy ⇒ my dog is apple.
10% of the time: Keep the word unchanged, e.g., my dog is hairy ⇒ my dog is hairy.
Why last two operators?

Next Sentence Prediction


原 PPT 独立图片资源
Next Sentence Prediction
原 PPT 独立图片资源
https://jalammar.github.io/illustrated-bert/

Training the Pretrain Model


原 PPT 独立图片资源
Training the Pretrain Model
Training Corpus:
BooksCorpus (800M words)
English Wikipedia (2,500M words)
Versions (L stands for the number of layers, H stands for the hidden size, A stands for the number of self-attention heads):
BERT-Base: L = 12, H = 768, A = 12, Total parameters = 110M
BERT-Large: L = 24, H = 1024, A = 16, Total parameters = 340M

Finetuning for Tasks


原 PPT 独立图片资源
Finetuning for Tasks
BERT Task Specific Models
https://jalammar.github.io/illustrated-bert/
BERT Task Specific Models

Bert for Feature Extraction


原 PPT 独立图片资源
Bert for Feature Extraction
https://jalammar.github.io/illustrated-bert/
原 PPT 独立图片资源

Bert for Feature Extraction


原 PPT 独立图片资源
Bert for Feature Extraction
https://jalammar.github.io/illustrated-bert/
原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.5 T5: Encoder–Decoder Pretraining



原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.5 T5: Encoder–Decoder Pretraining

原 PPT 独立图片资源

Transformers  Pretrained Models


原 PPT 独立图片资源
Transformers → Pretrained Models
原 PPT 独立图片资源
原 PPT 独立图片资源
No Masking!
原 PPT 独立图片资源

BERT
Encoder
Decoder
GPT
Encoder-Decoder
?

Pretrained Encoder-Decoder Model


原 PPT 独立图片资源
Pretrained Encoder-Decoder Model
原 PPT 独立图片资源
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf

Prefix Language Model


原 PPT 独立图片资源
Prefix Language Model
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR, 2020
原 PPT 独立图片资源
原 PPT 独立图片资源

Why Encoder-Decoder Denoising Model


原 PPT 独立图片资源
Why Encoder-Decoder Denoising Model
原 PPT 独立图片资源
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR, 2020

What is the Pretraining Output?


原 PPT 独立图片资源
What is the Pretraining Output?
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR, 2020
原 PPT 独立图片资源

Choose Which One?


原 PPT 独立图片资源
Choose Which One?
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR, 2020
原 PPT 独立图片资源

Choose Which One?


原 PPT 独立图片资源
Choose Which One?
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR, 2020
原 PPT 独立图片资源
原 PPT 独立图片资源

T5 (Text-to-Text Transfer Transformer, Google, 2021)


原 PPT 独立图片资源
T5 (Text-to-Text Transfer Transformer, Google, 2021)
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf
原 PPT 独立图片资源
原 PPT 独立图片资源

原 PPT 独立图片资源
原 PPT 独立图片资源
Uses a “span corruption” objective, where spans of text are randomly masked out and replaced by a mask token.
The model then must reconstruct these masked spans from the unmasked text.
Encoder-Decoder Framework

Multi-Task Pretraining


原 PPT 独立图片资源
Multi-Task Pretraining
T5 converts every task into a text-in → text-out format. For example:
Sentiment classification:
Input: "sst2 sentence: I liked the movie.”
Output: "positive”
Summarization:
Input: "summarize: [article text]”
Output: "[short summary]”
Translation:
Input: "translate English to German: How are you?”
Output: "Wie geht es dir?"
By unifying all tasks under this text-to-text scheme, T5 uses the same encoder-decoder architecture and the same loss function for everything, rather than having specialized heads for classification, QA, etc.

T5: Inspiration of AGI


原 PPT 独立图片资源
T5: Inspiration of AGI
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR, 2020
原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.6 Pretrained Model Gallery



原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.6 Pretrained Model Gallery

原 PPT 独立图片资源

ULMFiT (2018)


原 PPT 独立图片资源
ULMFiT (2018)
What:
Another pretrain–fine-tune architecture for NLP using LSTM-based language models at the same time with ELMO.

Why it’s important:
Historical significance: demonstrated that language model fine-tuning can drastically improve downstream tasks.
Bridges older RNN-based approaches and modern transformer-based methods.

RoBERTa (2019)


原 PPT 独立图片资源
RoBERTa (2019)
What:
A robustly optimized BERT variant from Facebook (Meta AI) that removes Next Sentence Prediction (NSP) and trains with longer sequences, dynamic masking, and more data.

Why it’s important:
Demonstrates that improved training protocols can yield better performance without architectural changes.
One of the first to show NSP might not be crucial in BERT-like models.

SpanBERT (2019)


原 PPT 独立图片资源
SpanBERT (2019)
What:
A variant of BERT that masks and predicts contiguous spans of text rather than individual tokens.

Why it’s important:
By focusing on phrase-level (span-level) context, SpanBERT better captures how multiple tokens interact to form entities or expressions.
This improvement leads to better performance on tasks such as question answering, coreference resolution, and information extraction—where understanding multi-token entities is crucial.

XLNet (2019)


原 PPT 独立图片资源
XLNet (2019)
What:
A generalized auto-regressive pretraining method that uses permutation language modeling rather than strictly left-to-right or masked token prediction.

Why it’s important:
Addresses some of the limitations of BERT’s masked language modeling by learning bidirectional context while still using an auto-regressive objective.
Showed strong performance on a variety of NLP benchmarks at the time.

BART (2019)


原 PPT 独立图片资源
BART (2019)
What:
An encoder-decoder (seq2seq) model that corrupts text input (via masking, shuffling, etc.) and learns to reconstruct the original text.
Why it’s important:
Demonstrates a strong denoising autoencoder approach for generation tasks (summarization, etc.).
Similar architecture to T5 but with different corruption strategies.

MASS (2019)


原 PPT 独立图片资源
MASS (2019)
What:
An encoder-decoder model that masks a fragment of the input and trains the decoder to reconstruct it.

Why it’s important:
Early encoder-decoder approach for tasks like machine translation.
A direct precursor to frameworks like BART/T5 in seq2seq pretraining.

ALBERT (2019)


原 PPT 独立图片资源
ALBERT (2019)
What:
A “lite” version of BERT that uses parameter-sharing across layers and factorized embeddings to reduce the model size.

Why it’s important:
Illustrates parameter efficiency in large Transformers.
Achieves near-state-of-the-art performance at the time with far fewer parameters.

DistilBERT (2019)


原 PPT 独立图片资源
DistilBERT (2019)
What:
A distilled (smaller) version of BERT that retains much of its performance while being faster and lighter.

Why it’s important:
Showcases knowledge distillation for compressing large PLMs.
Vital for real-world deployment constraints (latency, memory).

XLM/mBERT/XLM-R (2019)


原 PPT 独立图片资源
XLM/mBERT/XLM-R (2019)
What:
Multilingual and cross-lingual variants of BERT-like models.

Why it’s important:
Demonstrates how to handle many languages within a single model.
Cross-lingual transfer (fine-tune in one language, apply to another).

2.3 Pretrained Language Models 2.3.7 Suggested Readings



原 PPT 独立图片资源

2.3 Pretrained Language Models 2.3.7 Suggested Readings

原 PPT 独立图片资源

Papers and Articles


原 PPT 独立图片资源
Papers and Articles
Deep Contextualized Word Representations https://aclanthology.org/N18-1202/
Improving Language Understanding by Generative Pre-Training https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding https://aclanthology.org/N19-1423/
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer: https://jmlr.org/papers/v21/20-074.html

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源

Natural Language UnderstandingLanguage Understanding Tasks


原 PPT 独立图片资源
Natural Language
Understanding

Language Understanding Tasks
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Word Sense Disambiguation
Sentiment Analysis
Semantic Textual Similarity
Text Summarization
Information Retrieval
Question Answering
Dialogue Systems

2.4 Language Understanding Tasks 2.4.1 Word Sense Disambiguation



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.1 Word Sense Disambiguation

原 PPT 独立图片资源

Recall: Word Sense and Contextual Embedding


原 PPT 独立图片资源
Recall: Word Sense and Contextual Embedding
World has different Sense in different environment;
Bat (noun) — Animal
Bat (noun) — Sports equipment
Bat (verb) — To hit with a bat
Bat (verb) — To flutter or blink quickly
World has different contextual Embedding in different environment;
原 PPT 独立图片资源
https://www.cs.utexas.edu/~gdurrett/courses/fa2022/lectures/lec20-1pp.pdf

Recall: Word Sense and Contextual Embedding


原 PPT 独立图片资源
Recall: Word Sense and Contextual Embedding
https://web.stanford.edu/~jurafsky/slp3/11.pdf
原 PPT 独立图片资源

Word Sense Disambiguation (WSD)


原 PPT 独立图片资源
Word Sense Disambiguation (WSD)
Given:
a word in context;
a set of fixed potential word senses;
Output:
Identify the correct sense of the word in the context.
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/11.pdf

Problem Modeling


原 PPT 独立图片资源
Problem Modeling
Target: Matching a word in context to a set of sense;
What we have:
We can get contextual embedding of the word given context through a pre-trained BERT;
But we have no embedding of senses.
How to get the embedding of senses?
A sentence embedding from its definition?
Other method?

How to compute the Sense Embedding


原 PPT 独立图片资源
How to compute the Sense Embedding
Sense embedding from its definition:
Plan I: Use sentence-BERT to compute the phrase/sentence embedding.
Plan II: Average embedding of words in definition (using the same Bert of computing the contextual embedding of the target word)
Pros and Cons?

How to compute the Sense Embedding


原 PPT 独立图片资源
How to compute the Sense Embedding
If we have a sense-labeled dataset (SemCore or SenseEval):
The words in each sentence are labeled with it correct sense in the context.
When applying BERT in each sentence, we obtain the corresponding contextual embedding of that word, hence the corresponding sense.
So, for each sense, we gather a collection of contextual embeddings.
原 PPT 独立图片资源

In Inference Time


原 PPT 独立图片资源
In Inference Time
Just apply the nearest neighbor classification:
原 PPT 独立图片资源
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/11.pdf

2.4 Language Understanding Tasks 2.4.2 Sentiment Analysis



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.2 Sentiment Analysis

原 PPT 独立图片资源

Sentiment Analysis


原 PPT 独立图片资源
Sentiment Analysis
Movie reviews
Product comments
Online Twitter and Weibo
News
Financial report
unbelievably disappointing
Full of zany characters and richly applied satire, and some great plot twists
this is the greatest screwball comedy ever filmed
It was pathetic. The worst part about it was the boxing scenes.

Why Sentiment Analysis


原 PPT 独立图片资源
Why Sentiment Analysis

原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/slides/7_Sent.pdf

Why Sentiment Analysis


原 PPT 独立图片资源
Why Sentiment Analysis

https://web.stanford.edu/~jurafsky/slp3/slides/7_Sent.pdf
原 PPT 独立图片资源

Why Sentiment Analysis


原 PPT 独立图片资源
Why Sentiment Analysis

From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series. In ICWSM-2010
原 PPT 独立图片资源

Why Sentiment Analysis


原 PPT 独立图片资源
Why Sentiment Analysis
原 PPT 独立图片资源
原 PPT 独立图片资源
Bollen et al. (2011)
Twitter mood predicts the stock market, 2011

Other names of Sentiment Analysis


原 PPT 独立图片资源
Other names of Sentiment Analysis
Opinion extraction
Opinion mining
Sentiment mining
Subjectivity analysis

Sentiment Analysis Task


原 PPT 独立图片资源
Sentiment Analysis Task
Sentiment analysis is the detection of attitudes:
“enduring, affectively colored beliefs, dispositions towards objects or persons”
Holder (source) of attitude
Target (aspect) of attitude
Type of attitude
From a set of types: Like, love, hate, value, desire, etc.
Or (more commonly) simple weighted polarity: positive, negative, neutral, together with strength
Text containing the attitude: Sentence or entire document

Sentiment Analysis Task


原 PPT 独立图片资源
Sentiment Analysis Task
Simple task
Is the attitude of this text positive or negative?
Rank the attitude of this text from 1 to 5
Advanced:
Aspect based sentiment analysis

Aspect based Sentiment Analysis


原 PPT 独立图片资源
Aspect based Sentiment Analysis
Phrase 1: Aspect Term Extraction: extract explicit aspect expressions on which users express opinions in the given text.
The pizza is delicious, but the service is terrible. → “pizza”, “service”
Phrase 2: Aspect Sentiment Classification: predict the sentiment polarity for a specific aspect within a sentence
The pizza is delicious, but the service is terrible.
Aspect: pizza → Positive
Aspect: service → Negative

Algorithm: Simple Task


原 PPT 独立图片资源
Algorithm: Simple Task
Global classification problem.
Fine-tuning BERT with the right paradigm.
原 PPT 独立图片资源

Algorithm: Aspect based Sentiment Analysis


原 PPT 独立图片资源
Algorithm: Aspect based Sentiment Analysis
原 PPT 独立图片资源
原 PPT 独立图片资源
Aspect Term Extraction
Aspect Sentiment Classification
Aspect Term, left and right context

2.4 Language Understanding Tasks 2.4.3 Semantic Textual Similarity



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.3 Semantic Textual Similarity

原 PPT 独立图片资源

Semantic Textual Similarity (STS)


原 PPT 独立图片资源
Semantic Textual Similarity (STS)
Definition: STS measures the degree of semantic equivalence between two text snippets.
Input: A pair of sentences.
Output: A similarity score (typically on a scale from 0 to 5).
Example:
Sentence A: “A man is playing a guitar.”
Sentence B: “A person is performing music.”
Similarity Score: 4.5

Why STS Matters


原 PPT 独立图片资源
Why STS Matters
Core NLP task for understanding meaning.
Used in:
Duplicate question detection
Semantic search
Paraphrase identification
Dialogue understanding
Helps systems move beyond surface matching to deeper meaning comparison.

STS vs. Related Tasks


原 PPT 独立图片资源
STS vs. Related Tasks
Task
Goal
Difference from STS
Paraphrase Detection
Are the sentences semantically equivalent?
Binary decision (yes/no)
NLI / Textual Entailment
Does one sentence entail another?
Focuses on logical inference
Text Matching
Do the texts match in form/content?
Often used in retrieval
But their solutions may share the same framework.

Candidate Solution


原 PPT 独立图片资源
Candidate Solution
Pairwise sentence relation identification using BERT.
But if we want to cluster a set of sentences, we need to run 𝑁2 times of BERT.
How to do it efficiently?
原 PPT 独立图片资源

Make complex decision via embedding computation!


原 PPT 独立图片资源
Make complex decision via embedding computation!
Sentence-BERT(SBERT) : embed each sentence into a vector!
原 PPT 独立图片资源
https://arxiv.org/pdf/1908.10084

Training


原 PPT 独立图片资源
Training
Trained on 1M sentence pairs of NLI task of three classes: contradiction, eintailment, and neutral.
Finetuned with a 3-way softmax classifier objective function for one epoch.
The default pooling strategy is MEAN.

Performances


原 PPT 独立图片资源
Performances
The performances answers following questions:
Why not just average the BERT embedding?
Why use NLI dataset?
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.4 Text Summarization



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.4 Text Summarization

原 PPT 独立图片资源

Text Summarization


原 PPT 独立图片资源
Text Summarization
Definition: Automatically generating a shorter version of a text that preserves its key information.
Input: Long document or passage
Output: Concise summary
Goal: Retain the most important ideas in fewer words
Input: "Artificial Intelligence is transforming industries such as healthcare, finance, and education..."
Summary: "AI is revolutionizing major industries."

Road Map


原 PPT 独立图片资源
Road Map
Type
Description
Example
Extractive
Selects and combines important sentences from the original text
Sentence selection
Abstractive
Generates new sentences using understanding and rephrasing
Paraphrased summary

Abstractive Approach


原 PPT 独立图片资源
Abstractive Approach
We can apply Encoder-Decoder Pretrained Model + Fine tuning to do the task – BART or T5;
But we will introduce two more interesting models:
Summarization-Focused Pretrained Model: PEGASUS by Google;
Copy or Generation? CopyNet

PEGASUS


原 PPT 独立图片资源
PEGASUS
Mask and Predicting Important Sentences:
Important Sentences: Most similar with other part of the document measured by metric ROUGE;
Pretrained on large amount of data, finetuned on summarization tasks;
原 PPT 独立图片资源
https://research.google/blog/pegasus-a-state-of-the-art-model-for-abstractive-text-summarization/

PEGASUS: Performance


原 PPT 独立图片资源
PEGASUS: Performance
New state-of-the-art results as measured by automatic metrics, while using only 5% of the number of parameters of T5. 
Strong performance with 1000 finetuning samples;
原 PPT 独立图片资源
原 PPT 独立图片资源

Copy Net


原 PPT 独立图片资源
Copy Net
In ancient times:
Model cannot honestly repeat details of source document;
The algorithm still suffers from the Out-of-Vocabulary problem.
Abstractive Summarization is not reliable.
原 PPT 独立图片资源
https://arxiv.org/pdf/1603.06393

Copy Net


原 PPT 独立图片资源
Copy Net
Hybrid Generation + Copying:
At each decoding step, the model decides whether to generate a word from the vocabulary or copy a word from the input
Unified probability:
Final output probability is a combination of generation and copy probabilities:
原 PPT 独立图片资源
原 PPT 独立图片资源
https://arxiv.org/pdf/1603.06393

Copy Net


原 PPT 独立图片资源
Copy Net
Model
Copy Ability
Generate New Words
OOV Handling
Use Case Examples
Seq2Seq + Attn
❌
✅
Poor
General NLG
Pointer Network
✅ (positions)
❌
Good
Span selection, TSP
CopyNet
✅ (tokens)
✅
Excellent
Summarization, QA, NLG

Evaluation Metric


原 PPT 独立图片资源
Evaluation Metric
Text Generation Task generally measured by the n-gram overlapping ratio of the generated text to the reference/ground truth text.





BLEU: Precision, better for Machine Translation
ROUGE: Recall, better for For Summarization
Bleu score bigrams
https://clementbm.github.io/theory/2021/12/23/rouge-bleu-scores.html

2.4 Language Understanding Tasks 2.4.5 Information Retrieval



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.5 Information Retrieval

原 PPT 独立图片资源

Recall: The Bert Paradigm for QA


原 PPT 独立图片资源
Recall: The Bert Paradigm for QA
原 PPT 独立图片资源

Why Information Retrieval


原 PPT 独立图片资源
Why Information Retrieval
What if the paragraphs is not available?
T5 (and current LLM) can answer the question with their own memory.
But the memory is outdated and not reliable.
We need to retrieve the relevant paragraphs!

Information Retrieval


原 PPT 独立图片资源
Information Retrieval
Information Retrieval (IR) is the process of obtaining relevant information from a large repository based on a user’s query.
Examples:
Web search (e.g., Google, Baidu)
Searching a digital library
Product search on e-commerce platforms
Core Objective:
Find documents relevant to a user’s information need.

Key Concepts


原 PPT 独立图片资源
Key Concepts
Document Collection: The database of information (e.g., webpages, articles).
Query: A user-issued question or phrase.
Relevance: How well a document meets the user’s information need.
Indexing: Efficient organization of documents for fast retrieval.
Ranking: Ordering search results by relevance.

System Overview


原 PPT 独立图片资源
System Overview
原 PPT 独立图片资源

Retrieval vs Ranking


原 PPT 独立图片资源
Retrieval vs Ranking
Modern large scale system generally involve two stage:
Retrieval Stage: For Efficiency:
Narrow down the massive document collection to a small candidate set (e.g., top 1000 docs) that are potentially relevant.
Fast and scalable
Ranking Stage: For Quality:
Take the small set of candidates from the retrieval stage and finely rank them using more sophisticated and computationally expensive methods.
Precision over speed.

How to Retrieve


原 PPT 独立图片资源
How to Retrieve
Inverted index + BM25
Embedding Vector + ANN Search

Invert Index + BM25 Retrieve


原 PPT 独立图片资源
Invert Index + BM25 Retrieve
GitHub - Noureldin2303/Inverted-index-python: Writing a simple Inverted  Index in Python
https://github.com/Noureldin2303/Inverted-index-python
Query has a set of terms {𝑞1,𝑞2,⋯, 𝑞𝑛};
A large amount of documents can be filtered by invert index;
Compute BM25 between query and each remaining document;
Most of the component can be pre-computed and cached.

BM25 (Best Matching 25): Improved TFIDF


原 PPT 独立图片资源
BM25 (Best Matching 25): Improved TFIDF
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
→
→
Widely used in industry level retrieval system, like elastic search.

Embedding Vector + ANN Search


原 PPT 独立图片资源
Embedding Vector + ANN Search
Documents/Chunks are embedded into vectors using Embedding Language Models (like SBERT);
When a query is coming, embed it into an embedding vector;
Find the k-nearest neighbors of query-embedding from the collection of document embeddings.
How to find the 1000 nearest neighbors out of 1 billion document embeddings in 0.2 seconds?
We need Approximate Nearest Neighbor (ANN) Search

ANN Search: Local Sensitive Hash


原 PPT 独立图片资源
ANN Search: Local Sensitive Hash
Efficient Neural Networks Training through Locality Sensitive Hashing | by  LightOn | Medium
https://medium.com/@LightOnIO/efficient-neural-networks-training-through-locality-sensitive-hashing-93dc37077a89

ANN Search: KD-Tree


原 PPT 独立图片资源
ANN Search: KD-Tree

https://www.baeldung.com/cs/k-d-trees
A visual example of a simple K-D tree and its partitions.
Error In this Graph?

ANN Search: HNSW (Hierarchical Navigable Small Worlds )


原 PPT 独立图片资源
ANN Search: HNSW (Hierarchical Navigable Small Worlds )

https://www.pinecone.io/learn/series/faiss/hnsw/
The search process through the multi-layer structure of an HNSW graph.

How to Rank


原 PPT 独立图片资源
How to Rank
After initial retrieval, a dense re-ranker takes the query and candidate documents and computes a more accurate relevance score, usually with deep neural networks.
原 PPT 独立图片资源
https://medium.com/@mhammadkhan/neural-re-ranking-models-c0a67278f626

2.4 Language Understanding Tasks 2.4.6 Question Answering



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.6 Question Answering

原 PPT 独立图片资源

Question Answering Task


原 PPT 独立图片资源
Question Answering Task
Classification by Questions:
Factoid (e.g., “Who is the president of France?”)
Descriptive (e.g., “Why is the sky blue?”)
Procedural (e.g., “How to reset a router?”)
Classification by Resources:
Open-Domain QA:
No context is given; system must retrieve relevant docs then answer
Two-step pipeline: retrieval + reading
Closed-Book QA:
Model answers from its internal knowledge only

Road Maps


原 PPT 独立图片资源
Road Maps
Extractive QA:
Answer is a span in the provided context
Example datasets: SQuAD
Abstractive/Generative QA:
Answer is generated in free-form language
Example datasets: MS MARCO 

Extractive QA


原 PPT 独立图片资源
Extractive QA
原 PPT 独立图片资源
https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/Warsaw.html

Model for Extractive QA: DeBERTa


原 PPT 独立图片资源
Model for Extractive QA: DeBERTa
https://towardsdatascience.com/large-language-models-deberta-decoding-enhanced-bert-with-disentangled-attention-90016668db4b/
Embedding construction in BERT and DeBERTa. Instead of storing all the information in a single vector, DeBERTa uses two separate vectors to store word and position embeddings.
Enhanced mask decoder in DeBERTa. Image adopted by the author.

Abstractive/Generative QA


原 PPT 独立图片资源
Abstractive/Generative QA
{
"query":". what is a corporation?",
"passages":[
{"url":"http:\/\/www.wisegeek.com\/what-is-a-corporation.htm",
“passage_text”:“A company is incorporated in a specific nation, often within the bounds of a smaller subset of that nation, such as a state or province. The corporation is then governed by the laws of incorporation in that state. A corporation may issue stock, either private or public, or may be classified as a non-stock corporation. If stock is issued, the corporation will usually be governed by its shareholders, either directly or indirectly.”},
…],
"answers":["A corporation is a company or group of people authorized to act as a single entity and recognized as such in law."],
}

Model for Abstractive QA: T5


原 PPT 独立图片资源
Model for Abstractive QA: T5
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR, 2020
原 PPT 独立图片资源

Evaluation Metrics


原 PPT 独立图片资源
Evaluation Metrics
Extractive QA:
Exact Match (EM): Measures whether the prediction exactly matches any one ground truth answer.
F1 Score: Measures overlap at the token level between prediction and ground truth.
Abstractive/Generative QA:
ROUGE: Measures n-gram overlap between prediction and reference.

2.4 Language Understanding Tasks 2.4.7 Dialogue Systems



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.7 Dialogue Systems

原 PPT 独立图片资源

Types of Dialogue


原 PPT 独立图片资源
Types of Dialogue
Task-oriented: Help users achieve specific goals (e.g., book flights, tech support)
Open-domain (chit-chat): Free-form conversation on any topic.

Dialogue System Components


原 PPT 独立图片资源
Dialogue System Components
Requirement Understanding:
Intent detection
Slot filling
Dialogue Management:
Keeps track of dialogue state
Determines next action (e.g., via rules or policy learning)
Natural Language Generation:
Converts system actions into human-readable text

Dialogue System Components


原 PPT 独立图片资源
Dialogue System Components
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Requirement Understanding


原 PPT 独立图片资源
Requirement Understanding
Task Domains: Design ontology for each domain via a set of Frame.
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Requirement Understanding


原 PPT 独立图片资源
Requirement Understanding
Domain Classification: What is the domain of the user‘s domain?
Detect the domain;
Example: “Book me a flight” → travel
Intent/Frame Detection: What does the user want to do?
Detect the Frame type;
Example: “Book me a flight” → BookFlight
Slot Filling: Extract arguments or parameters.
Fill the Frame
Example: “from New York to London tomorrow” → ORIG CITY=NYC, DEST CITY=London, DEPA TIME=tomorrow

Requirement Understanding


原 PPT 独立图片资源
Requirement Understanding
Show me morning flights from Boston to San Francisco on Tuesday



Wake me tomorrow at 6
原 PPT 独立图片资源
原 PPT 独立图片资源

Requirement Understanding


原 PPT 独立图片资源
Requirement Understanding
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Define Dialog Acts
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Label examples of Dialog Acts
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Dialog State Tracking: Maintain memory of the dialogue so far.
https://web.stanford.edu/~jurafsky/slp3/15.pdf
原 PPT 独立图片资源

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Dialogue Policy: Which act to generate
Decide when to answer user’s questions, when to instead ask the user a clarification question, and so on
原 PPT 独立图片资源
原 PPT 独立图片资源

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Rule based Policy
Book a hotel in Paris for 2 nights starting April 20.
IF "intent" = "book_hotel" AND "location" IS NOT filled
THEN system action = Ask for location

ELSE IF "location" IS filled AND "nights" IS NOT filled
THEN system action = Ask for number of nights

ELSE IF all slots are filled
THEN system action = Confirm booking

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Learning based Policy
Book a hotel in Paris for 2 nights starting April 20.
The policy is represented as a function: 𝜋(𝑠) = 𝑎
where 𝑠 is the current dialogue state, and a is the next system action.
Example:
𝑠 = {𝑖𝑛𝑡𝑒𝑛𝑡=𝐵𝑜𝑜𝑘𝐻𝑜𝑡𝑒𝑙, 𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛=𝑃𝑎𝑟𝑖𝑠, 𝑛𝑖𝑔h𝑡𝑠=𝑁𝑜𝑛𝑒}⇒ 𝑎 = 𝐴𝑠𝑘(𝑛𝑖𝑔h𝑡𝑠)
The agent learns this mapping by receiving rewards (e.g., +1 for task success, -1 for confusion or errors).

Natural Language Generation


原 PPT 独立图片资源
Natural Language Generation
Once a dialogue act has been chosen, we need to generate the text of the response to the user.
https://web.stanford.edu/~jurafsky/slp3/15.pdf
原 PPT 独立图片资源

Natural Language Generation


原 PPT 独立图片资源
Natural Language Generation
Generation with Placeholders
https://web.stanford.edu/~jurafsky/slp3/15.pdf
原 PPT 独立图片资源
原 PPT 独立图片资源

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源

Natural Language UnderstandingLarge Language Model


原 PPT 独立图片资源
Natural Language
Understanding

Large Language Model
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Large Language Model
Pretraining
Supervised Instruction Finetuning
Reinforcement Learning from Human Preference

2.5 Large Language Models 2.5.1 Pretraining



原 PPT 独立图片资源

2.5 Large Language Models 2.5.1 Pretraining

原 PPT 独立图片资源

Pretraining


原 PPT 独立图片资源
Pretraining
Definition: Pretraining refers to the process of training a language model on a large corpus of raw text using self-supervised learning.
Objective: Learn statistical patterns of language, including grammar, semantics, and world knowledge.
Output: A general-purpose model that can be adapted (via fine-tuning or prompting) to downstream tasks.

Recall: Pretrained Language Model


原 PPT 独立图片资源
Recall: Pretrained Language Model
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture09-pretraining.pdf
原 PPT 独立图片资源

Recall: Pretrained Language Model


原 PPT 独立图片资源
Recall: Pretrained Language Model
原 PPT 独立图片资源
Language Model:




Masked Language Model:
原 PPT 独立图片资源

Recall: Pretrained Language Model


原 PPT 独立图片资源
Recall: Pretrained Language Model
原 PPT 独立图片资源
原 PPT 独立图片资源
No Masking!
原 PPT 独立图片资源

BERT
Encoder
Decoder
GPT
Encoder-Decoder
Bart, T5

Why Large Pretrained Language Model?


原 PPT 独立图片资源
Why Large Pretrained Language Model?
Pretrained Language Model:
Trained on mid-sized datasets with mid-sized models
Require fine-tuning for specific tasks (e.g., classification, QA)
Limited generalization: perform well on tasks they are tuned for
Do not exhibit general language understanding
Example: BERT, GPT-2, T5

Why Large Pretrained Language Model?


原 PPT 独立图片资源
Why Large Pretrained Language Model?
Large Pretrained Language Model:
Trained on massive datasets with billions of parameters
Surprising capabilities emerge during pretraining
Demonstrate general language understanding (though imperfect)
Able to perform zero-shot, few-shot, and chain-of-thought reasoning
Represent a new paradigm in language understanding
Example: GPT-3, PaLM, LLaMA, GPT-4

Why Large Pretrained Language Model?


原 PPT 独立图片资源
Why Large Pretrained Language Model?

Capability
PLM (Early Models)
LLM (Modern Models)
Model Size
~100M – 1B parameters
10B – 500B+ parameters
Training Data
Mid-sized, domain-limited
Diverse, web-scale corpora
Task Generalization
Weak, narrow
Strong, task-agnostic
Emergent Abilities
✘ None observed
✔ Appear with scale
Language Understanding
Task-specific pattern matching
Signs of general competence

GPT3: The Beginning of Large Pretrained Language Model


原 PPT 独立图片资源
GPT3: The Beginning of Large Pretrained Language Model
GPT-3 model by OpenAI
https://blog.accubits.com/getting-started-with-gpt-3-model-by-openai/

Few-Shot In-Context Learning


原 PPT 独立图片资源
Few-Shot In-Context Learning
T. B. Brown, et.al , Language Models are Few-Shot Learners, 2020
原 PPT 独立图片资源
Machine Translation Task
Traditional Fine-tuning Approach →
Let‘s see Few shot In-Context Learning Approach by GPT 3

Few Shot In-Context Learning


原 PPT 独立图片资源
Few Shot In-Context Learning
原 PPT 独立图片资源
T. B. Brown, et.al , Language Models are Few-Shot Learners, 2020

Few Shot In-Context Learning


原 PPT 独立图片资源
Few Shot In-Context Learning
T. B. Brown, et.al , Language Models are Few-Shot Learners, 2020
原 PPT 独立图片资源

Few Shot In-Context Learning


原 PPT 独立图片资源
Few Shot In-Context Learning
T. B. Brown, et.al , Language Models are Few-Shot Learners, 2020
原 PPT 独立图片资源

More Examples


原 PPT 独立图片资源
More Examples
Sentiment analysis
Classify the sentiment of each sentence as Positive, Negative, or Neutral.

"I love this movie!" → Positive
 
"This is the worst product I've ever bought." → Negative  

"It's an average experience." → Neutral  

"The customer service was excellent!" → 
Math Word Problem
Solve the problem and write the answer as a number.

Q: If you have 5 apples and you give away 2, how many are left?  
A: 3  

Q: John has 12 pencils. He buys 5 more. How many pencils does he have now?  
A: 17  

Q: Sarah read 7 books last month and 6 books this month. How many books did she read in total?  
A: 
Named Entity Recognition
Extract named entities and classify them as PERSON, LOCATION, or ORGANIZATION.

Sentence: "Barack Obama visited Paris."  
Entities: Barack Obama [PERSON], Paris [LOCATION]  

Sentence: "Google opened a new office in London."  
Entities: Google [ORGANIZATION], London [LOCATION]  

Sentence: "Elon Musk met with the mayor of New York."  
Entities: 

Performances


原 PPT 独立图片资源
Performances
原 PPT 独立图片资源
T. B. Brown, et.al , Language Models are Few-Shot Learners, 2020
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Where the Ability Comes From? Scaling Law


原 PPT 独立图片资源
Where the Ability Comes From? Scaling Law
原 PPT 独立图片资源
原 PPT 独立图片资源
T. B. Brown, et.al , Language Models are Few-Shot Learners, 2020

Historical Discovery on Scaling


原 PPT 独立图片资源
Historical Discovery on Scaling
原 PPT 独立图片资源
 “Trends and Developments in Deep Learning Research” Jeff Dean, Google, 2017
原 PPT 独立图片资源

Scaling Law on LLM


原 PPT 独立图片资源
Scaling Law on LLM
原 PPT 独立图片资源
J. Kaplan, Scaling Laws for Neural Language Models, 2020

Scaling Law on LLM


原 PPT 独立图片资源
Scaling Law on LLM
J. Kaplan, Scaling Laws for Neural Language Models, 2020
原 PPT 独立图片资源

Data Scaling


原 PPT 独立图片资源
Data Scaling
Statistical Learning
原 PPT 独立图片资源
原 PPT 独立图片资源

Scaling Laws for Nonparametric Learning


原 PPT 独立图片资源
Scaling Laws for Nonparametric Learning
原 PPT 独立图片资源

Model Scaling


原 PPT 独立图片资源
Model Scaling
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Data & Model Joint Scaling


原 PPT 独立图片资源
Data & Model Joint Scaling
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Data & Model Joint Scaling is Accurate


原 PPT 独立图片资源
Data & Model Joint Scaling is Accurate
原 PPT 独立图片资源

Importance of Scaling Law


原 PPT 独立图片资源
Importance of Scaling Law
Predictive Design Principle
Allows researchers to forecast model performance before training. → Helps allocate compute/data/model size optimally.
Drives Paradigm Shift
Shifts focus from architecture engineering to scale-driven performance. → “More data + bigger models + longer training” became a viable recipe.
Foundation for Efficient Training
Informs compute-optimal model design (e.g., Chinchilla Law): → For a fixed compute budget, what’s the best model/data ratio?
Unifies Empirical Trends Across Models
Observed in transformers, language models, vision models, multimodal models… → Suggests a universal scaling behavior in deep learning.

When the Ability Occur? By Emergence


原 PPT 独立图片资源
When the Ability Occur? By Emergence
原 PPT 独立图片资源
J. Wei, Emergent Abilities of Large Language Models, 2022

Emergence


原 PPT 独立图片资源
Emergence
Emergent abilities are capabilities that appear suddenly at a specific scale of model size, data, or compute — not present in smaller models and not predictable by extrapolation.
Key Observations by Wei et al., 2022
Signature Trait: Abrupt, non-linear improvements in task performance as model size increases
Suggests a qualitative shift in general-purpose intelligence with sufficient scale.

Emergence: Critiques


原 PPT 独立图片资源
Emergence: Critiques
“Are Emergent Abilities a Mirage?” (Schaeffer et al., 2023)
Apparent emergence may result from misleading evaluation scales (e.g., log accuracy)
When plotted properly, performance follows smooth scaling for most tasks
“What Makes In-Context Learning Work?” (Min et al., 2022)
Prompt formatting and input distribution strongly affect observed abilities
Abilities are sensitive, not necessarily emergent
Other Key Critiques:
Some tasks never exhibit emergence
Emergence may reflect training data overlap, not true generalization
Different abilities emerge at different scales — no unified threshold

Emergence


原 PPT 独立图片资源
Emergence
Emergence is not entirely an illusion, but it requires more precise definitions and measurement methodologies.
It is more likely a complex phenomenon that arises from the interaction among task type, model scale, prompt design, and data distribution, rather than being controlled by a single factor.
Emergent abilities remain an important but debated phenomenon. To harness them, we need better diagnostics, metrics, and theory.

Why the Ability of LLM?


原 PPT 独立图片资源
Why the Ability of LLM?
From the Language Model: The model learns to predict the next word given a sequence of previous words.
During training, the model is exposed to a vast variety of textual patterns, including:
Instructions + answers
Lists, tables, dialogue turns
Textbook examples, Q&A pairs, code examples, etc.
🧠 Key Insight: In-context learning is not explicitly trained, but rather emerges because the model memorizes and generalizes over millions of task-format patterns in natural data.

Why the Ability of LLM?


原 PPT 独立图片资源
Why the Ability of LLM?
Large LMs perform implicit meta-learning:
🔬 Idea:
During pretraining, the model sees many sequences of the form:
"Input A → Output A, Input B → Output B, Input C → ?"
The model learns a general mechanism:
“Given a few demonstrations, infer the mapping rule and apply it to the new input.”
This is analogous to meta-learning (“learning to learn”) in traditional ML:
Instead of explicitly optimizing for task adaptation , GPT-3 internalizes the meta-structure of many tasks through exposure to patterns.

2.5 Large Language Models 2.5.2 Supervised Instruction Fine-Tuning



原 PPT 独立图片资源

2.5 Large Language Models 2.5.2 Supervised Instruction Fine-Tuning

原 PPT 独立图片资源

Why Supervised Instruction Fine-Tuning (SFT)


原 PPT 独立图片资源
Why Supervised Instruction Fine-Tuning (SFT)
Why do we need instruction fine-tuning?
LLMs are pretrained on diverse web-scale corpora: code, books, Wikipedia, etc.
Pretrained LLMs are Good at Text continuation task, including few shot in-context learning;
But no notion of task-specific instruction.
Example: “Summarize the following paragraph…” → Base LLM often fails.
Goal: Align model behavior with human-intended instructions.

Instruction Fine-Tuning


原 PPT 独立图片资源
Instruction Fine-Tuning
Definition: Supervised fine-tuning of a pretrained LLM on (instruction, response) pairs.
Also known as supervised fine-tuning (SFT) stage in alignment pipeline.
Shift from pretraining objective to task-oriented alignment.
Learn to map:
(𝐼𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛, 𝐼𝑛𝑝𝑢𝑡)→ 𝑂𝑢𝑡𝑝𝑢𝑡
Ex: Instruction: “Translate to French”, Input: “Hello” → Output: “Bonjour”

Type of Instruction Data


原 PPT 独立图片资源
Type of Instruction Data
Natural language instructions + labeled outputs.
Sources:
NLP benchmarks (SuperGLUE, SQuAD, etc.)
Crowd-sourced instruction sets (e.g., Self-Instruct)
Synthetic instructions from models themselves.

Recall: Multi-Task in T5


原 PPT 独立图片资源
Recall: Multi-Task in T5
T5 converts every task into a text-in → text-out format. For example:
Sentiment classification:
Input: "sst2 sentence: I liked the movie.”
Output: "positive”
Summarization:
Input: "summarize: [article text]”
Output: "[short summary]”
Translation:
Input: "translate English to German: How are you?”
Output: "Wie geht es dir?”
Fixed Labels, not flexible nature instructions.

Algorithm: Supervised Finetuning


原 PPT 独立图片资源
Algorithm: Supervised Finetuning
Objective:
𝐿𝑆𝐹𝑇 =∑{(𝑥,𝑦)∈ 𝐷}log 𝑃(𝑦 | 𝑥;𝜃)  
where 𝑥 = 𝑖𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛 + 𝑖𝑛𝑝𝑢𝑡,  𝑦 = 𝑑𝑒𝑠𝑖𝑟𝑒𝑑 𝑜𝑢𝑡𝑝𝑢𝑡
Can be applied in various pretrained models: T5, GPT 3 …

Historical Mile-Stones


原 PPT 独立图片资源
Historical Mile-Stones
FLAN (2021): Unified instruction tuning on 60+ tasks.
T0 (2021): Zero-shot generalization via multitask instruction tuning.
InstructGPT (2022): Supervised + RLHF training for instruction following.

FLAN


原 PPT 独立图片资源
FLAN
原 PPT 独立图片资源
J. Wei, Finetuned Language Models Are Zero-Shot Learners, 2022

FLAN


原 PPT 独立图片资源
FLAN
J. Wei, Finetuned Language Models Are Zero-Shot Learners, 2022
原 PPT 独立图片资源

FLAN


原 PPT 独立图片资源
FLAN
J. Wei, Finetuned Language Models Are Zero-Shot Learners, 2022
原 PPT 独立图片资源

T0


原 PPT 独立图片资源
T0
V. Sanh, et.al, Multitask Prompted Training Enables Zero-Shot Task Generalization, 2022
原 PPT 独立图片资源

T0


原 PPT 独立图片资源
T0
V. Sanh, et.al, Multitask Prompted Training Enables Zero-Shot Task Generalization, 2022
原 PPT 独立图片资源

T0


原 PPT 独立图片资源
T0
V. Sanh, et.al, Multitask Prompted Training Enables Zero-Shot Task Generalization, 2022
原 PPT 独立图片资源

T0


原 PPT 独立图片资源
T0
V. Sanh, et.al, Multitask Prompted Training Enables Zero-Shot Task Generalization, 2022
原 PPT 独立图片资源

Instruct-GPT


原 PPT 独立图片资源
Instruct-GPT
L. Ouyang, Training language models to follow instructions with human feedback, 2022.
原 PPT 独立图片资源
原 PPT 独立图片资源

Summary


原 PPT 独立图片资源
Summary

All methods create natural language instructed supervised data:
From existing NLP tasks;
Human Label (Open AI)
A pretrained LLM are fine-tuned in this dataset;
The zero-shot performances over supervised tasks are improved;
But more importantly, the finetuned models all show promising generalization to instructions outside of the finetuning distribution.
They learn the meaning of natural language instructions.

Gap to Our Goal – Human Intelligence


原 PPT 独立图片资源
Gap to Our Goal -- Human Intelligence

SFT cannot fully achieve our goal -- Human Preference
Learns to Mimic, Not to Optimize Preferences
Cannot Resolve Ambiguities or Trade-offs
Passive and Non-Interactive
Overfits to Demonstration Distribution
Cannot Penalize Undesirable Outputs

2.5 Large Language Models 2.5.3 Reinforcement Learning from Human Feedback



原 PPT 独立图片资源

2.5 Large Language Models 2.5.3 Reinforcement Learning from Human Feedback

原 PPT 独立图片资源

How to achieve Human Preference?


原 PPT 独立图片资源
How to achieve Human Preference?
First, we need to model human preference, that is, tell how well the result satisfies humans requests.
Human preference is expensive and slow!
Human preference hard to model by rules on language responses;
How about build a machine learning model?
Given each response, the human preference model will give a score indicating the quality of the response;
Second, we need to optimize our LLM to generate responses with high scores given by the human preference model:
Reinforcement Learning is the best choice.

Architecture of ChatGPT


原 PPT 独立图片资源
Architecture of ChatGPT

index/chatgpt > Methods > Media > Chatgpt Diagram Light
https://openai.com/index/chatgpt/


Instruction Finetuning
RLHF

History of RLHFEarly Developments in Human-in-the-Loop Reinforcement Learning


原 PPT 独立图片资源
History of RLHF
Early Developments in Human-in-the-Loop Reinforcement Learning
Early approaches had humans directly guiding agent behavior via rewards or corrections.
TAMER (2008) allowed humans to provide scalar feedback signals (positive/negative) in real time to train an agent. This taught agents by shaping a reward model from human evaluations, rather than a predefined reward.
COACH (2017) introduced an actor-critic method where human feedback adjusted the critic’s advantage estimates. These early works established that humans can meaningfully shape an agent’s policy during learning.
Preference-based Reinforcement Learning (PbRL, 2011) : humans express preferences between two outcomes, and the agent learns a policy that produces outcomes ranking higher in human preference. This set the stage for modern RLHF – learning from comparative feedback rather than absolute reward signals.

History of RLHFEmergence of RLHF in the Deep Learning Era (2016–2018)


原 PPT 独立图片资源
History of RLHF
Emergence of RLHF in the Deep Learning Era (2016–2018)
The Proximal Policy Optimization (PPO, 2017) algorithm provided a stable and efficient way to fine-tune policies in large parameter spaces.
The modern wave of RLHF research took off with the success of deep reinforcement learning. By the late 2010s, RLHF techniques were being tested in complex domains, especially in games.
In Christiano et al. (OpenAI, 2017) humans compared pairs of short video clips of an agent’s behavior , and a neural reward model was trained from these comparisons.
OpenAI Five (which played Dota 2) and DeepMind’s AlphaStar (which played StarCraft II) incorporated human demonstrations or preference-based fine-tuning.

History of RLHFRLHF for Language Models (2019–2021)


原 PPT 独立图片资源
History of RLHF
RLHF for Language Models (2019–2021)
Early large language models could produce fluent text but often in ways misaligned with user intent (e.g. going off-topic or being inappropriate).
Ziegler et al. (2019) and Stiennon et al. (2020) fine-tuned GPT-2 on tasks like summarization and dialogue using human preference comparisons.
OpenAI’s WebGPT (2021) trained a model to answer questions using a web browser, with human evaluators rewarding answers that were correct and had proper citations.
DeepMind’s GopherCite (2022) similarly used human feedback to encourage a language model to provide sources for its claims.

RLHF provides gains over pretraining + finetuning


原 PPT 独立图片资源
RLHF provides gains over pretraining + finetuning
原 PPT 独立图片资源
https://arxiv.org/pdf/2009.01325

History of RLHFInstruction-Following and the ChatGPT Era


原 PPT 独立图片资源
History of RLHF
Instruction-Following and the ChatGPT Era
OpenAI’s InstructGPT (Ouyang et al., 2022) demonstrated that RLHF can make models dramatically more useful.
InstructGPT also exhibited improvements in truthfulness and reductions in toxic or harmful content compared to the base model – a significant proof that RLHF can enhance both helpfulness and safety of AI systems.
OpenAI’s ChatGPT (2022) is fine-tuned with human feedback for conversational dialogue.
By 2023, RLHF became a standard component for state-of-the-art language models across the AI industry: OpenAI’s GPT-4 (2023) , Anthropic’s Claude, LLaMA-2 (Meta, 2023) , Google’s dialogue models (e.g. LaMDA, Bard).

Incorporating Human Preference


原 PPT 独立图片资源
Incorporating Human Preference
Human Preference is expensive and slow!
Train an LM 𝑅𝑀𝜙 𝑠 to predict human preferences from an annotated dataset, then optimize for 𝑅𝑀𝜙 instead.
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture10-instruction-tunining-rlhf.pdf

Incorporating Human Preference


原 PPT 独立图片资源
Incorporating Human Preference
Human judgments are noisy and miscalibrated!
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture10-instruction-tunining-rlhf.pdf
原 PPT 独立图片资源

Incorporating Human Preference


原 PPT 独立图片资源
Incorporating Human Preference
Solution: instead of asking for direct ratings, ask for pairwise comparisons, which can be more reliable
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture10-instruction-tunining-rlhf.pdf

Reward Model In RLHF


原 PPT 独立图片资源
Reward Model In RLHF
Start from the SFT model (e.g., GPT-style autoregressive model).
Add a scalar head on top of the final hidden state ℎ𝑇 of the output sequence
𝑟𝜙(𝑥, 𝑦) = 𝑓𝜙(ℎ𝑇 )
Train a rank model
原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture10-instruction-tunining-rlhf.pdf

Recall: Reinforcement Learning


原 PPT 独立图片资源
Recall: Reinforcement Learning
Agent learns to make decisions by interacting with an environment to maximize a reward signal over time.
Exploration vs. Exploitation:
Exploration refers to trying new actions to discover potentially better rewards.
Exploitation means choosing the action that gives the highest known reward based on past experience.
The agent interacts with the environment by taking actions, receiving rewards, and updating its policy to maximize future rewards.

Recall: Reinforcement Learning


原 PPT 独立图片资源
Recall: Reinforcement Learning
原 PPT 独立图片资源

https://www.slideshare.net/slideshow/the-fundamentals-of-machine-learning/238653218

Reinforcement Learning


原 PPT 独立图片资源
Reinforcement Learning
https://medium.com/@binaydhakal35/mdp-markov-decision-process-rl-reinforcement-learning-bc85e5d25031
Markov Decision Process

Reinforcement Learning Markov Decision Process


原 PPT 独立图片资源
Reinforcement Learning
Markov Decision Process
An MDP is defined by a tuple (𝑆,𝐴,𝑃,𝑅,𝛾) ,where:
𝑆 (State Space): A set of states representing the environment.
𝐴 (Action Space): A set of actions available to the agent in each state.
𝑃 (Transition Probability): A probability function 𝑃(𝑠′ | 𝑠, 𝑎) that defines the likelihood of transitioning from state s to s' after taking action a.
𝑅 (Reward Function): A function 𝑅(𝑠, 𝑎, 𝑠′) that assigns a reward for moving from state s to s' via action a.
𝛾 (Discount Factor): A value in the range [0,1] that determines the importance of future rewards.

Reinforcement LearningMarkov Property


原 PPT 独立图片资源
Reinforcement Learning
Markov Property
MDP follows the Markov Property, which states that the next state depends only on the current state and action, not on past states.
Mathematically,
𝑃(𝑠′|𝑠,𝑎,𝑠𝑡−1,𝑎𝑡−1,…)=𝑃(𝑠′|𝑠,𝑎)
This property ensures that MDPs can be efficiently solved using mathematical techniques like dynamic programming.

Reinforcement LearningPolicy and Value


原 PPT 独立图片资源
Reinforcement Learning
Policy and Value
The policy is defined as a probability distribution over actions given a state.

The optimal policy is the one that maximizes the expected cumulative reward, denoted as:

Reinforcement Learning is to find the optimal policy.
原 PPT 独立图片资源
原 PPT 独立图片资源

Reinforcement Learning for LLM


原 PPT 独立图片资源
Reinforcement Learning for LLM
Train the language model 𝜋𝜃 to maximize the reward model score:


This is a deep reinforcement learning setting:
Policy: 𝜋𝜃 :a deep autoregressive language model
Reward: scalar from 𝑟𝜙(𝑥) , given at the end of generation
原 PPT 独立图片资源

The Policy Gradient Approach


原 PPT 独立图片资源
The Policy Gradient Approach
Optimize the objective by gradient ascent:
原 PPT 独立图片资源

Why not Policy Gradient


原 PPT 独立图片资源
Why not Policy Gradient
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
The gradient is modulated by the total return, regardless of how good the action was relative to the state, leading to high variance in gradient estimates.

Actor-Critic


原 PPT 独立图片资源
Actor-Critic
Subtracting a state-dependent baseline would greatly reduce the variance
原 PPT 独立图片资源
原 PPT 独立图片资源
How to choose a good baseline? – Value and Advantage;
原 PPT 独立图片资源
原 PPT 独立图片资源

Actor-Critic: Advantage


原 PPT 独立图片资源
Actor-Critic: Advantage
Monte Calo (MC) Advantage:


One-Step Time Difference Advantage:


 Generalized Advantage Estimation
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Actor-Critic: Algorithm


原 PPT 独立图片资源
Actor-Critic: Algorithm
Components:
Actor: the policy 𝜋𝜃(𝑎|𝑠) , responsible for action selection
Critic: the value function 𝑉𝑤(𝑠), evaluates how good a state is
Procedure:
Actor chooses actions based on 𝜋𝜃(𝑎|𝑠) 
Critic computes 𝑉𝑤(𝑠)
Collect trajectory (𝑠𝑡, 𝑎𝑡, 𝑟𝑡, 𝑠𝑡+1)
Critic updates 𝑤 by minimizing:
ℒ𝑉(𝑤)= (𝑟𝑡 +𝛾 𝑉𝑤(𝑠𝑡+1)− 𝑉𝑤(𝑠𝑡))2
Actor updates 𝜃 using: ∇𝜃 log  𝜋𝜃(𝑎𝑡|𝑠𝑡)𝐴𝑡

The Actor-Critic Approach


原 PPT 独立图片资源
The Actor-Critic Approach
https://www.researchgate.net/figure/The-Markov-decision-process-of-reinforcement-learning-According-to-the-reward-the_fig1_368578437
| The Markov decision process of reinforcement learning. According to the reward, the agent learns to evaluate the value of each action and gain an optimal policy that maximizes the expected return.

Why not Actor-Critic?


原 PPT 独立图片资源
Why not Actor-Critic?
Problem in Actor-Critic:
Policy updates can be too large, causing performance collapse
No explicit constraint on how much the policy changes
Trust Region Policy Optimization (TRPO):
Solves this via a KL-divergence constraint
Effective but complex (second-order method)
PPO: Proposes a simple alternative to TRPO with first-order optimization

The PPO Approach


原 PPT 独立图片资源
The PPO Approach
https://spinningup.openai.com/en/latest/algorithms/ppo.html
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Advantage is positive: the new policy does not benefit by going far away from the old policy. Advantage is negative: the new policy does not benefit by going far away from the old policy.


原 PPT 独立图片资源
Advantage is positive: the new policy does not benefit by going far away from the old policy.


Advantage is negative: the new policy does not benefit by going far away from the old policy.
The PPO Approach
https://spinningup.openai.com/en/latest/algorithms/ppo.html
原 PPT 独立图片资源
原 PPT 独立图片资源

The PPO Approach


原 PPT 独立图片资源
The PPO Approach
https://spinningup.openai.com/en/latest/algorithms/ppo.html
原 PPT 独立图片资源

PPO In RLHF


原 PPT 独立图片资源
PPO In RLHF
Actor: Policy Network
The actor model starts from the SFT model:
𝜋𝜃←𝜋𝑆𝐹𝑇
Outputs a distribution over tokens:
𝜋𝜃(𝑦𝑡∣ 𝑥, 𝑦<𝑡) = 𝑆𝑜𝑓𝑡𝑚𝑎𝑥(𝑊ℎ𝑡)
Critic: Value Network
Also initialized from the SFT model:
𝑉𝜓←𝜋𝑆𝐹𝑇
A scalar value head is added to the transformer outputs:
𝑉𝜓(𝑠_𝑡) = 𝑓𝜓(ℎ𝑡)

PPO In RLHF


原 PPT 独立图片资源
PPO In RLHF
Reward: KL divergence is added to the human preference reward as a regularization term, to avoid catastrophic drift from the supervised fine-tuned (SFT) model, which serves as the original “safe” baseline.
原 PPT 独立图片资源
原 PPT 独立图片资源

RLHF


原 PPT 独立图片资源
RLHF
Finally, we have everything we need:
A pretrained (possibly instruction-finetuned) LM 𝑝𝑃𝑇 (𝑠)
A reward model 𝑅𝑀𝜙(𝑠) that produces scalar rewards for LM outputs, trained on a dataset of human comparisons
A method for optimizing LM parameters towards an arbitrary reward function.
Now to do RLHF:
Initialize the policy network;
Initialize the value network;
Build the regularized reward model;
Optimize the reward with PPO.

Summary


原 PPT 独立图片资源
Summary
index/chatgpt > Methods > Media > Chatgpt Diagram Light
https://openai.com/index/chatgpt/

2.5 Large Language Models 2.5.4 Decoding Strategies



原 PPT 独立图片资源

2.5 Large Language Models 2.5.4 Decoding Strategies

原 PPT 独立图片资源

Decoding for Searching the Desired Sequence


原 PPT 独立图片资源
Decoding for Searching the Desired Sequence
Search: Find desired Sequences that maximize some objectives, calculated using certain information;
Naïve Solution -- Exhaustive Search:
Search Space: All possible Sequences;
Not feasible with complexity of
Need efficient algorithm!
XCS224N] Lecture 8 – Translation, Seq2Seq, Attention - mx's blog
Image Source: https://x-wei.github.io/xcs224n-lecture8.html
原 PPT 独立图片资源

Decoding Problem for Sequence Generation


原 PPT 独立图片资源
Decoding Problem for Sequence Generation
Given: A trained LLM;
Problem: Generating a sequence to maximize certain objective using available information;
Objective
What to maximize
Probability, Reward, Variety, …
Information
What can be utilized;
Step probability, value function, knowledge, …
Algorithm
Computationally feasible/efficient;
Capabilities;

I. For Probability Maximization


原 PPT 独立图片资源
I. For Probability Maximization
Considering generating a sequence of items {𝑦1,⋯,𝑦𝑇} from a vocabulary 𝒴 given a condition 𝑐;
Information:
Objective: Maximize
原 PPT 独立图片资源
原 PPT 独立图片资源

Greedy Search


原 PPT 独立图片资源
Greedy Search
At each step, select the token with the highest conditional probability;
Efficient but sub-optimal;
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
Image Source: https://d2l.ai/chapter_recurrent-modern/beam-search.html

Beam Search


原 PPT 独立图片资源
Beam Search
Keep 𝐾 (beam size) sequence in memory;
At each step, consider K|𝒴| candidates, and select the top 𝐾 sequence with conditional sequence probability;
Time complexity: 𝑂(𝑇|𝒴|𝐾 logK);
原 PPT 独立图片资源
Image Source: https://d2l.ai/chapter_recurrent-modern/beam-search.html

II. For Natural Language


原 PPT 独立图片资源
II. For Natural Language
Image Source: Ari Holtzman, et.al, The Curious Case of Neural Text DeGeneration, ICLR, 2020
The failure of previous methods in generating open-ended texts
原 PPT 独立图片资源

Caused by Objective or Local Minima?


原 PPT 独立图片资源
Caused by Objective or Local Minima?
“Natural Language does not maximize probability”[1]
“the per-token probability of natural text is, on average, much lower than text generated by beam search”
Natural languages favor lower-probability but more informative and non-repetitive tokens.
“Grice’s Maxims of Communication (Grice, 1975) show that people optimize against stating the obvious. Thus, making every word as predictable as possible will be disfavored.”
原 PPT 独立图片资源
[1] Ari Holtzman, et.al, The Curious Case of Neural Text DeGeneration, ICLR, 2020

For Natural Generation


原 PPT 独立图片资源
For Natural Generation
Objective: Generate text similar to human (golden) output
Information:
Motivation:
Adaptively truncate unreliable tail words (see the incoherent gibberish of results of pure sampling);
Better sampling, not just maximize the probability nor pure sampling;
Algorithm:
Choose a set of expected tokens 𝑉;
Design a new distribution 𝑝 over tokens;
原 PPT 独立图片资源

Algorithms


原 PPT 独立图片资源
Algorithms
Top-𝑘 Sampling:
𝑌𝑘 contains tokens with top-𝑘 probabilities;
Top-𝑝(Nucleus) Sampling:
𝑌𝑝 contains least tokens with summarized probabilities larger than 𝑝;
Sampling with Temperature:
𝑌𝑡=𝒴, without truncation;
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Adaptability


原 PPT 独立图片资源
Adaptability
Adaptability is essential when encountering various distributions;
Top-𝑝 Sampling is most adaptable algorithm;
原 PPT 独立图片资源
Image Source: Ari Holtzman, et.al, The Curious Case of Neural Text DeGeneration, ICLR, 2020

Measure the Similarity with Human Generated Text


原 PPT 独立图片资源
Measure the Similarity with Human Generated Text
Features of Generated Texts:
Perplexity: Measure the generation probability; lower means higher probability;
Self-BLEU: Matching score between outputs; lower means higher diversity;
Zipf Coefficient: Parameter of the word frequency distribution;
Repetition: Frequency of finding repeated phrases (at least three times) in output;
HUSE: A unified score of human evaluation and statistical evaluation.
The closer the features of generated texts to those of human generated texts, the better the sampling strategy is.

Evaluation Results


原 PPT 独立图片资源
Evaluation Results
Top-𝑝 Sampling generates texts most similar to human’s;
It is the most popular decoding strategy for LLM;
Their combinations are also used;
Table and Image Source: Ari Holtzman, et.al, The Curious Case of Neural Text DeGeneration, ICLR, 2020
原 PPT 独立图片资源
原 PPT 独立图片资源

2.5 Large Language Models 2.5.5 Suggested Readings



原 PPT 独立图片资源

2.5 Large Language Models 2.5.5 Suggested Readings

原 PPT 独立图片资源

Papers


原 PPT 独立图片资源
Papers
T. B. Brown, et.al., Language Models are Few-Shot Learners, 2020
J. Kaplan, et.al., Scaling Laws for Neural Language Models, 2020
J. Wei, et.al. Emergent Abilities of Large Language Models, 2022
J. Wei, et.al, Finetuned Language Models Are Zero-Shot Learners, 2022
L. Ouyang, et.al., Training language models to follow instructions with human feedback, 2022

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源