Natural Language UnderstandingLanguage Understanding Tasks


原 PPT 独立图片资源
Natural Language
Understanding

Language Understanding Tasks
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Word Sense Disambiguation
Sentiment Analysis
Semantic Textual Similarity
Text Summarization
Information Retrieval
Question Answering
Dialogue Systems

2.4 Language Understanding Tasks 2.4.1 Word Sense Disambiguation



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.1 Word Sense Disambiguation

原 PPT 独立图片资源

Recall: Word Sense and Contextual Embedding


原 PPT 独立图片资源
Recall: Word Sense and Contextual Embedding
World has different Sense in different environment;
Bat (noun) — Animal
Bat (noun) — Sports equipment
Bat (verb) — To hit with a bat
Bat (verb) — To flutter or blink quickly
World has different contextual Embedding in different environment;
原 PPT 独立图片资源
https://www.cs.utexas.edu/~gdurrett/courses/fa2022/lectures/lec20-1pp.pdf

Recall: Word Sense and Contextual Embedding


原 PPT 独立图片资源
Recall: Word Sense and Contextual Embedding
https://web.stanford.edu/~jurafsky/slp3/11.pdf
原 PPT 独立图片资源

Word Sense Disambiguation (WSD)


原 PPT 独立图片资源
Word Sense Disambiguation (WSD)
Given:
a word in context;
a set of fixed potential word senses;
Output:
Identify the correct sense of the word in the context.
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/11.pdf

Problem Modeling


原 PPT 独立图片资源
Problem Modeling
Target: Matching a word in context to a set of sense;
What we have:
We can get contextual embedding of the word given context through a pre-trained BERT;
But we have no embedding of senses.
How to get the embedding of senses?
A sentence embedding from its definition?
Other method?

How to compute the Sense Embedding


原 PPT 独立图片资源
How to compute the Sense Embedding
Sense embedding from its definition:
Plan I: Use sentence-BERT to compute the phrase/sentence embedding.
Plan II: Average embedding of words in definition (using the same Bert of computing the contextual embedding of the target word)
Pros and Cons?

How to compute the Sense Embedding


原 PPT 独立图片资源
How to compute the Sense Embedding
If we have a sense-labeled dataset (SemCore or SenseEval):
The words in each sentence are labeled with it correct sense in the context.
When applying BERT in each sentence, we obtain the corresponding contextual embedding of that word, hence the corresponding sense.
So, for each sense, we gather a collection of contextual embeddings.
原 PPT 独立图片资源

In Inference Time


原 PPT 独立图片资源
In Inference Time
Just apply the nearest neighbor classification:
原 PPT 独立图片资源
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/11.pdf

2.4 Language Understanding Tasks 2.4.2 Sentiment Analysis



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.2 Sentiment Analysis

原 PPT 独立图片资源

Sentiment Analysis


原 PPT 独立图片资源
Sentiment Analysis
Movie reviews
Product comments
Online Twitter and Weibo
News
Financial report
unbelievably disappointing
Full of zany characters and richly applied satire, and some great plot twists
this is the greatest screwball comedy ever filmed
It was pathetic. The worst part about it was the boxing scenes.

Why Sentiment Analysis


原 PPT 独立图片资源
Why Sentiment Analysis

原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/slides/7_Sent.pdf

Why Sentiment Analysis


原 PPT 独立图片资源
Why Sentiment Analysis

https://web.stanford.edu/~jurafsky/slp3/slides/7_Sent.pdf
原 PPT 独立图片资源

Why Sentiment Analysis


原 PPT 独立图片资源
Why Sentiment Analysis

From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series. In ICWSM-2010
原 PPT 独立图片资源

Why Sentiment Analysis


原 PPT 独立图片资源
Why Sentiment Analysis
原 PPT 独立图片资源
原 PPT 独立图片资源
Bollen et al. (2011)
Twitter mood predicts the stock market, 2011

Other names of Sentiment Analysis


原 PPT 独立图片资源
Other names of Sentiment Analysis
Opinion extraction
Opinion mining
Sentiment mining
Subjectivity analysis

Sentiment Analysis Task


原 PPT 独立图片资源
Sentiment Analysis Task
Sentiment analysis is the detection of attitudes:
“enduring, affectively colored beliefs, dispositions towards objects or persons”
Holder (source) of attitude
Target (aspect) of attitude
Type of attitude
From a set of types: Like, love, hate, value, desire, etc.
Or (more commonly) simple weighted polarity: positive, negative, neutral, together with strength
Text containing the attitude: Sentence or entire document

Sentiment Analysis Task


原 PPT 独立图片资源
Sentiment Analysis Task
Simple task
Is the attitude of this text positive or negative?
Rank the attitude of this text from 1 to 5
Advanced:
Aspect based sentiment analysis

Aspect based Sentiment Analysis


原 PPT 独立图片资源
Aspect based Sentiment Analysis
Phrase 1: Aspect Term Extraction: extract explicit aspect expressions on which users express opinions in the given text.
The pizza is delicious, but the service is terrible. → “pizza”, “service”
Phrase 2: Aspect Sentiment Classification: predict the sentiment polarity for a specific aspect within a sentence
The pizza is delicious, but the service is terrible.
Aspect: pizza → Positive
Aspect: service → Negative

Algorithm: Simple Task


原 PPT 独立图片资源
Algorithm: Simple Task
Global classification problem.
Fine-tuning BERT with the right paradigm.
原 PPT 独立图片资源

Algorithm: Aspect based Sentiment Analysis


原 PPT 独立图片资源
Algorithm: Aspect based Sentiment Analysis
原 PPT 独立图片资源
原 PPT 独立图片资源
Aspect Term Extraction
Aspect Sentiment Classification
Aspect Term, left and right context

2.4 Language Understanding Tasks 2.4.3 Semantic Textual Similarity



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.3 Semantic Textual Similarity

原 PPT 独立图片资源

Semantic Textual Similarity (STS)


原 PPT 独立图片资源
Semantic Textual Similarity (STS)
Definition: STS measures the degree of semantic equivalence between two text snippets.
Input: A pair of sentences.
Output: A similarity score (typically on a scale from 0 to 5).
Example:
Sentence A: “A man is playing a guitar.”
Sentence B: “A person is performing music.”
Similarity Score: 4.5

Why STS Matters


原 PPT 独立图片资源
Why STS Matters
Core NLP task for understanding meaning.
Used in:
Duplicate question detection
Semantic search
Paraphrase identification
Dialogue understanding
Helps systems move beyond surface matching to deeper meaning comparison.

STS vs. Related Tasks


原 PPT 独立图片资源
STS vs. Related Tasks
Task
Goal
Difference from STS
Paraphrase Detection
Are the sentences semantically equivalent?
Binary decision (yes/no)
NLI / Textual Entailment
Does one sentence entail another?
Focuses on logical inference
Text Matching
Do the texts match in form/content?
Often used in retrieval
But their solutions may share the same framework.

Candidate Solution


原 PPT 独立图片资源
Candidate Solution
Pairwise sentence relation identification using BERT.
But if we want to cluster a set of sentences, we need to run 𝑁2 times of BERT.
How to do it efficiently?
原 PPT 独立图片资源

Make complex decision via embedding computation!


原 PPT 独立图片资源
Make complex decision via embedding computation!
Sentence-BERT(SBERT) : embed each sentence into a vector!
原 PPT 独立图片资源
https://arxiv.org/pdf/1908.10084

Training


原 PPT 独立图片资源
Training
Trained on 1M sentence pairs of NLI task of three classes: contradiction, eintailment, and neutral.
Finetuned with a 3-way softmax classifier objective function for one epoch.
The default pooling strategy is MEAN.

Performances


原 PPT 独立图片资源
Performances
The performances answers following questions:
Why not just average the BERT embedding?
Why use NLI dataset?
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.4 Text Summarization



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.4 Text Summarization

原 PPT 独立图片资源

Text Summarization


原 PPT 独立图片资源
Text Summarization
Definition: Automatically generating a shorter version of a text that preserves its key information.
Input: Long document or passage
Output: Concise summary
Goal: Retain the most important ideas in fewer words
Input: "Artificial Intelligence is transforming industries such as healthcare, finance, and education..."
Summary: "AI is revolutionizing major industries."

Road Map


原 PPT 独立图片资源
Road Map
Type
Description
Example
Extractive
Selects and combines important sentences from the original text
Sentence selection
Abstractive
Generates new sentences using understanding and rephrasing
Paraphrased summary

Abstractive Approach


原 PPT 独立图片资源
Abstractive Approach
We can apply Encoder-Decoder Pretrained Model + Fine tuning to do the task – BART or T5;
But we will introduce two more interesting models:
Summarization-Focused Pretrained Model: PEGASUS by Google;
Copy or Generation? CopyNet

PEGASUS


原 PPT 独立图片资源
PEGASUS
Mask and Predicting Important Sentences:
Important Sentences: Most similar with other part of the document measured by metric ROUGE;
Pretrained on large amount of data, finetuned on summarization tasks;
原 PPT 独立图片资源
https://research.google/blog/pegasus-a-state-of-the-art-model-for-abstractive-text-summarization/

PEGASUS: Performance


原 PPT 独立图片资源
PEGASUS: Performance
New state-of-the-art results as measured by automatic metrics, while using only 5% of the number of parameters of T5. 
Strong performance with 1000 finetuning samples;
原 PPT 独立图片资源
原 PPT 独立图片资源

Copy Net


原 PPT 独立图片资源
Copy Net
In ancient times:
Model cannot honestly repeat details of source document;
The algorithm still suffers from the Out-of-Vocabulary problem.
Abstractive Summarization is not reliable.
原 PPT 独立图片资源
https://arxiv.org/pdf/1603.06393

Copy Net


原 PPT 独立图片资源
Copy Net
Hybrid Generation + Copying:
At each decoding step, the model decides whether to generate a word from the vocabulary or copy a word from the input
Unified probability:
Final output probability is a combination of generation and copy probabilities:
原 PPT 独立图片资源
原 PPT 独立图片资源
https://arxiv.org/pdf/1603.06393

Copy Net


原 PPT 独立图片资源
Copy Net
Model
Copy Ability
Generate New Words
OOV Handling
Use Case Examples
Seq2Seq + Attn
❌
✅
Poor
General NLG
Pointer Network
✅ (positions)
❌
Good
Span selection, TSP
CopyNet
✅ (tokens)
✅
Excellent
Summarization, QA, NLG

Evaluation Metric


原 PPT 独立图片资源
Evaluation Metric
Text Generation Task generally measured by the n-gram overlapping ratio of the generated text to the reference/ground truth text.





BLEU: Precision, better for Machine Translation
ROUGE: Recall, better for For Summarization
Bleu score bigrams
https://clementbm.github.io/theory/2021/12/23/rouge-bleu-scores.html

2.4 Language Understanding Tasks 2.4.5 Information Retrieval



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.5 Information Retrieval

原 PPT 独立图片资源

Recall: The Bert Paradigm for QA


原 PPT 独立图片资源
Recall: The Bert Paradigm for QA
原 PPT 独立图片资源

Why Information Retrieval


原 PPT 独立图片资源
Why Information Retrieval
What if the paragraphs is not available?
T5 (and current LLM) can answer the question with their own memory.
But the memory is outdated and not reliable.
We need to retrieve the relevant paragraphs!

Information Retrieval


原 PPT 独立图片资源
Information Retrieval
Information Retrieval (IR) is the process of obtaining relevant information from a large repository based on a user’s query.
Examples:
Web search (e.g., Google, Baidu)
Searching a digital library
Product search on e-commerce platforms
Core Objective:
Find documents relevant to a user’s information need.

Key Concepts


原 PPT 独立图片资源
Key Concepts
Document Collection: The database of information (e.g., webpages, articles).
Query: A user-issued question or phrase.
Relevance: How well a document meets the user’s information need.
Indexing: Efficient organization of documents for fast retrieval.
Ranking: Ordering search results by relevance.

System Overview


原 PPT 独立图片资源
System Overview
原 PPT 独立图片资源

Retrieval vs Ranking


原 PPT 独立图片资源
Retrieval vs Ranking
Modern large scale system generally involve two stage:
Retrieval Stage: For Efficiency:
Narrow down the massive document collection to a small candidate set (e.g., top 1000 docs) that are potentially relevant.
Fast and scalable
Ranking Stage: For Quality:
Take the small set of candidates from the retrieval stage and finely rank them using more sophisticated and computationally expensive methods.
Precision over speed.

How to Retrieve


原 PPT 独立图片资源
How to Retrieve
Inverted index + BM25
Embedding Vector + ANN Search

Invert Index + BM25 Retrieve


原 PPT 独立图片资源
Invert Index + BM25 Retrieve
GitHub - Noureldin2303/Inverted-index-python: Writing a simple Inverted  Index in Python
https://github.com/Noureldin2303/Inverted-index-python
Query has a set of terms {𝑞1,𝑞2,⋯, 𝑞𝑛};
A large amount of documents can be filtered by invert index;
Compute BM25 between query and each remaining document;
Most of the component can be pre-computed and cached.

BM25 (Best Matching 25): Improved TFIDF


原 PPT 独立图片资源
BM25 (Best Matching 25): Improved TFIDF
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
→
→
Widely used in industry level retrieval system, like elastic search.

Embedding Vector + ANN Search


原 PPT 独立图片资源
Embedding Vector + ANN Search
Documents/Chunks are embedded into vectors using Embedding Language Models (like SBERT);
When a query is coming, embed it into an embedding vector;
Find the k-nearest neighbors of query-embedding from the collection of document embeddings.
How to find the 1000 nearest neighbors out of 1 billion document embeddings in 0.2 seconds?
We need Approximate Nearest Neighbor (ANN) Search

ANN Search: Local Sensitive Hash


原 PPT 独立图片资源
ANN Search: Local Sensitive Hash
Efficient Neural Networks Training through Locality Sensitive Hashing | by  LightOn | Medium
https://medium.com/@LightOnIO/efficient-neural-networks-training-through-locality-sensitive-hashing-93dc37077a89

ANN Search: KD-Tree


原 PPT 独立图片资源
ANN Search: KD-Tree

https://www.baeldung.com/cs/k-d-trees
A visual example of a simple K-D tree and its partitions.
Error In this Graph?

ANN Search: HNSW (Hierarchical Navigable Small Worlds )


原 PPT 独立图片资源
ANN Search: HNSW (Hierarchical Navigable Small Worlds )

https://www.pinecone.io/learn/series/faiss/hnsw/
The search process through the multi-layer structure of an HNSW graph.

How to Rank


原 PPT 独立图片资源
How to Rank
After initial retrieval, a dense re-ranker takes the query and candidate documents and computes a more accurate relevance score, usually with deep neural networks.
原 PPT 独立图片资源
https://medium.com/@mhammadkhan/neural-re-ranking-models-c0a67278f626

2.4 Language Understanding Tasks 2.4.6 Question Answering



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.6 Question Answering

原 PPT 独立图片资源

Question Answering Task


原 PPT 独立图片资源
Question Answering Task
Classification by Questions:
Factoid (e.g., “Who is the president of France?”)
Descriptive (e.g., “Why is the sky blue?”)
Procedural (e.g., “How to reset a router?”)
Classification by Resources:
Open-Domain QA:
No context is given; system must retrieve relevant docs then answer
Two-step pipeline: retrieval + reading
Closed-Book QA:
Model answers from its internal knowledge only

Road Maps


原 PPT 独立图片资源
Road Maps
Extractive QA:
Answer is a span in the provided context
Example datasets: SQuAD
Abstractive/Generative QA:
Answer is generated in free-form language
Example datasets: MS MARCO 

Extractive QA


原 PPT 独立图片资源
Extractive QA
原 PPT 独立图片资源
https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/Warsaw.html

Model for Extractive QA: DeBERTa


原 PPT 独立图片资源
Model for Extractive QA: DeBERTa
https://towardsdatascience.com/large-language-models-deberta-decoding-enhanced-bert-with-disentangled-attention-90016668db4b/
Embedding construction in BERT and DeBERTa. Instead of storing all the information in a single vector, DeBERTa uses two separate vectors to store word and position embeddings.
Enhanced mask decoder in DeBERTa. Image adopted by the author.

Abstractive/Generative QA


原 PPT 独立图片资源
Abstractive/Generative QA
{
"query":". what is a corporation?",
"passages":[
{"url":"http:\/\/www.wisegeek.com\/what-is-a-corporation.htm",
“passage_text”:“A company is incorporated in a specific nation, often within the bounds of a smaller subset of that nation, such as a state or province. The corporation is then governed by the laws of incorporation in that state. A corporation may issue stock, either private or public, or may be classified as a non-stock corporation. If stock is issued, the corporation will usually be governed by its shareholders, either directly or indirectly.”},
…],
"answers":["A corporation is a company or group of people authorized to act as a single entity and recognized as such in law."],
}

Model for Abstractive QA: T5


原 PPT 独立图片资源
Model for Abstractive QA: T5
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR, 2020
原 PPT 独立图片资源

Evaluation Metrics


原 PPT 独立图片资源
Evaluation Metrics
Extractive QA:
Exact Match (EM): Measures whether the prediction exactly matches any one ground truth answer.
F1 Score: Measures overlap at the token level between prediction and ground truth.
Abstractive/Generative QA:
ROUGE: Measures n-gram overlap between prediction and reference.

2.4 Language Understanding Tasks 2.4.7 Dialogue Systems



原 PPT 独立图片资源

2.4 Language Understanding Tasks 2.4.7 Dialogue Systems

原 PPT 独立图片资源

Types of Dialogue


原 PPT 独立图片资源
Types of Dialogue
Task-oriented: Help users achieve specific goals (e.g., book flights, tech support)
Open-domain (chit-chat): Free-form conversation on any topic.

Dialogue System Components


原 PPT 独立图片资源
Dialogue System Components
Requirement Understanding:
Intent detection
Slot filling
Dialogue Management:
Keeps track of dialogue state
Determines next action (e.g., via rules or policy learning)
Natural Language Generation:
Converts system actions into human-readable text

Dialogue System Components


原 PPT 独立图片资源
Dialogue System Components
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Requirement Understanding


原 PPT 独立图片资源
Requirement Understanding
Task Domains: Design ontology for each domain via a set of Frame.
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Requirement Understanding


原 PPT 独立图片资源
Requirement Understanding
Domain Classification: What is the domain of the user‘s domain?
Detect the domain;
Example: “Book me a flight” → travel
Intent/Frame Detection: What does the user want to do?
Detect the Frame type;
Example: “Book me a flight” → BookFlight
Slot Filling: Extract arguments or parameters.
Fill the Frame
Example: “from New York to London tomorrow” → ORIG CITY=NYC, DEST CITY=London, DEPA TIME=tomorrow

Requirement Understanding


原 PPT 独立图片资源
Requirement Understanding
Show me morning flights from Boston to San Francisco on Tuesday



Wake me tomorrow at 6
原 PPT 独立图片资源
原 PPT 独立图片资源

Requirement Understanding


原 PPT 独立图片资源
Requirement Understanding
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Define Dialog Acts
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Label examples of Dialog Acts
原 PPT 独立图片资源
https://web.stanford.edu/~jurafsky/slp3/15.pdf

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Dialog State Tracking: Maintain memory of the dialogue so far.
https://web.stanford.edu/~jurafsky/slp3/15.pdf
原 PPT 独立图片资源

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Dialogue Policy: Which act to generate
Decide when to answer user’s questions, when to instead ask the user a clarification question, and so on
原 PPT 独立图片资源
原 PPT 独立图片资源

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Rule based Policy
Book a hotel in Paris for 2 nights starting April 20.
IF "intent" = "book_hotel" AND "location" IS NOT filled
THEN system action = Ask for location

ELSE IF "location" IS filled AND "nights" IS NOT filled
THEN system action = Ask for number of nights

ELSE IF all slots are filled
THEN system action = Confirm booking

Dialogue Management


原 PPT 独立图片资源
Dialogue Management
Learning based Policy
Book a hotel in Paris for 2 nights starting April 20.
The policy is represented as a function: 𝜋(𝑠) = 𝑎
where 𝑠 is the current dialogue state, and a is the next system action.
Example:
𝑠 = {𝑖𝑛𝑡𝑒𝑛𝑡=𝐵𝑜𝑜𝑘𝐻𝑜𝑡𝑒𝑙, 𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛=𝑃𝑎𝑟𝑖𝑠, 𝑛𝑖𝑔h𝑡𝑠=𝑁𝑜𝑛𝑒}⇒ 𝑎 = 𝐴𝑠𝑘(𝑛𝑖𝑔h𝑡𝑠)
The agent learns this mapping by receiving rewards (e.g., +1 for task success, -1 for confusion or errors).

Natural Language Generation


原 PPT 独立图片资源
Natural Language Generation
Once a dialogue act has been chosen, we need to generate the text of the response to the user.
https://web.stanford.edu/~jurafsky/slp3/15.pdf
原 PPT 独立图片资源

Natural Language Generation


原 PPT 独立图片资源
Natural Language Generation
Generation with Placeholders
https://web.stanford.edu/~jurafsky/slp3/15.pdf
原 PPT 独立图片资源
原 PPT 独立图片资源

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源