Natural Language UnderstandingSymbolic Representing of Language


原 PPT 独立图片资源
Natural Language
Understanding

Symbolic Representing
of Language
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Symbolic Structures of Language
History of Symbolic Language Representation
Case Studies:
Noam Chomsky’s Generative Grammar
Dependency Grammar
Open Information Annotation
Coreference
Semantic Relationship Between Sentences
Obtain the Symbolic Representation

1.1 Symbolic Representation of Language 1.1.1 Symbolic Structures of Language



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.1 Symbolic Structures of Language

原 PPT 独立图片资源

Symbolic Structures of Language


原 PPT 独立图片资源
Symbolic Structures of Language

Corpus
Document
Paragraph
Sentence
Chunk
Word
Morpheme
Character

Character, Morpheme, Word


原 PPT 独立图片资源
Character, Morpheme, Word

Category
English
Chinese
Character
A letter (e.g., 'a', 'b', 'c') has no meaning by itself.
A character (e.g., '水', '火', '学') often has meaning by itself.
Morpheme
The smallest meaningful unit (e.g., 'un-', 'run', '-ing').
A character or a combination of characters that carries meaning (e.g., '学' in '学习', '化' in '现代化').
Word
A combination of morphemes that forms a complete meaning (e.g., 'running', 'unhappy').
Usually a compound word combining multiple morphemes (e.g., '电脑' from '电' + '脑').

Word, Chunk, Sentence


原 PPT 独立图片资源
Word, Chunk, Sentence
Category
Definition
English Examples
Chinese Examples
Word
A combination of morphemes forming a meaningful unit.
'computer', 'running'
'电脑' , '跑步'
Chunk
A group of words forming a meaningful phrase but not a full sentence.
'the big house', 'eating lunch'
'漂亮的房子', '吃午饭'
Sentence
A complete grammatical structure expressing a full thought.
'I am reading a book.'
'南京市长江大桥'

Sentence, Paragraph, Document, Corpus


原 PPT 独立图片资源
Sentence, Paragraph, Document, Corpus
Category
Definition
Note
Sentence
A complete grammatical structure expressing a full thought.

Paragraph
A collection of related sentences forming a coherent unit.
Sentences in it are related and not independent.
Document
A structured collection of paragraphs forming a complete text.
Paragraphs in it are related and not independent.
Corpus
A large collection of documents used for linguistic or computational analysis.
The documents are generally considered independent.

Studies on the Symbolic Representation


原 PPT 独立图片资源
Studies on the Symbolic Representation
Where the elements and structure come from.
How lower-level elements form and constitute the upper structure.
What is the relationship between elements in the same layer.

1.1 Symbolic Representation of Language 1.1.2 History of Symbolic Language Representation



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.2 History of Symbolic Language Representation

原 PPT 独立图片资源

Early Formal Representations in Linguistics—— Ferdinand de Saussure (1857–1913): Structuralism


原 PPT 独立图片资源
Early Formal Representations in Linguistics—— Ferdinand de Saussure (1857–1913): Structuralism
A sign is the basic unit of meaning in language.It consists of:
Signifier (signifiant): The sound pattern or written form (e.g., the word “tree”).
Signified (signifié): The concept or meaning associated with the signifier (the mental image of a tree).
Structural Relationships in Language
Syntagmatic relations: How words are arranged in a sequence (e.g., word order in a sentence).
Paradigmatic relations: How words can be substituted for one another in a given context (e.g., synonyms or different tenses of a verb). In “The cat sleeps,” the word “cat” can be substituted with “dog” (paradigmatic), while “sleeps” follows the subject “cat” (syntagmatic).

Formal Grammar and Symbolic Syntax——Noam Chomsky (1950s–Present): Generative Grammar


原 PPT 独立图片资源
Formal Grammar and Symbolic Syntax——
Noam Chomsky (1950s–Present): Generative Grammar
Phrase Structure Rules & Context-Free Grammar (CFG)
Chomsky introduced formal grammar rules that generate valid sentences.
Example: S → NP VP;NP → Det N;VP → V NP 
Generates sentences like “The cat chased the mouse.”
Deep Structure vs. Surface Structure
A sentence has an underlying deep structure (logical meaning) and a surface structure (actual words).
Example: “John is easy to please” vs. “John is eager to please” (different deep structures despite similar surface structures)
Transformational Grammar
Introduces transformations (e.g., passive voice) to explain structural variations.
Example: “The boy saw the girl” → “The girl was seen by the boy.”

Dependency Grammar and Semantic Structures—— Lucien Tesnière (1959): Dependency Grammar


原 PPT 独立图片资源
Dependency Grammar and Semantic Structures
—— Lucien Tesnière (1959): Dependency Grammar
Emphasize word-to-word relationships instead of hierarchical phrase structures.
Dependency Relations
Unlike phrase structure grammar, which organizes sentences into hierarchical constituents, dependency grammar represents syntax as a network of head-dependent relationships.
In a dependency structure, each word (except the root) depends on another word (its governor/head).
The relationships between words are represented as directed edges in a tree.
Structural Simplicity
Avoids unnecessary intermediate nodes like noun phrases (NP) or verb phrases (VP)
Instead, it directly links words based on their syntactic and semantic relationships.

Abstract Meaning Representation (AMR, 2010s)


原 PPT 独立图片资源
Abstract Meaning Representation (AMR, 2010s)
AMR represents the semantic structure of a sentence as a directed acyclic graph (DAG).
Nodes represent concepts (entities, events, actions).
Edges represent semantic relations.
Predicate-Argument Structure
Verbs: PropBank-style verb senses (e.g., want-01, eat-01).
Roles are labeled using numbered arguments:
ARG0 → Agent/Doer
ARG1 → Theme/Object
ARG2 → Beneficiary
It abstracts away from word order and syntactic variations, meaning different syntactic realizations of the same idea map to the same AMR structure.

(w / want-01
:ARG0 (b / boy)
:ARG1 (e / eat-01
:ARG0 b
:ARG1 (a / apple)))

Open Domain Knowledge Representations (OIA, 2022 and Lingua 2026)


原 PPT 独立图片资源
Open Domain Knowledge Representations (OIA, 2022 and Lingua 2026)
An intermediate representation of cognition elements:
Entities, facts in the world and their relationships;
Each sentence can be represented as a Predicate/Function/Argument structure.
Use original words to express the structure, without loss of information.

The Central Problem: Surface Expression and Underlying Meaning


原 PPT 独立图片资源
The Central Problem: Surface Expression and Underlying Meaning


Theory
View on Surface Expression
View on Underlying Meaning
Key Mechanism of Mapping
Example
Structuralism (Saussure, 1916)
Words and grammar are part of a structured system of signs.
Meaning comes from differences between signs, not direct reference to reality.
Signifier (word form) ↔ Signified (concept)
The word “dog” means “dog” not because of an inherent property, but because it differs from “cat”.
Generative Grammar (Chomsky, 1957)
Sentences have surface structure (what we see) and deep structure (abstract rules).
Deep structure captures core meaning, independent of syntax.
Transformational rules convert deep structure → surface structure.
“John saw Mary” vs. “Mary was seen by John” have different syntax but same deep meaning.
Dependency Grammar (Tesnière, 1959)
Words are connected by dependency relations (head-modifier structure).
Meaning is derived from syntactic relations rather than linear order.
Dependency trees capture who does what to whom.
“The cat chased the mouse” → “chased” is the head, “cat” is subject, “mouse” is object.
Abstract Meaning Representation (AMR, 2010s)
Surface syntax varies, but core meaning is a graph.
Meaning is a directed graph of concepts and relations.
Sentences are mapped into AMR graphs.
“She wants to buy a car” → (w / want-01 :arg0 (s / she) :arg1 (b / buy-01 :arg1 (c / car)))
Open Information Annotation (OIA, 2020s)
Language can be expressed as Predicate/Function/Arguemnt structure;
Meaning is relationships among entities and facts.
Sentences are mapped into OIA Graphs;
“She buy a car” → (buy) –arg1->(She) -arg2->(a car)

1.1 Symbolic Representation of Language 1.1.3 Case Studies



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.3 Case Studies

原 PPT 独立图片资源

Generative Grammar and Constituency Parsing


原 PPT 独立图片资源
Generative Grammar and Constituency Parsing

https://www.structural-learning.com/post/chomskys-theory
Noam Chomsky, cognitive behavioral theory, cognitive development theory

Generative Grammar


原 PPT 独立图片资源
Generative Grammar

https://en.wikipedia.org/wiki/Generative_grammar
undefined

Rules for Generative Grammar


原 PPT 独立图片资源
Rules for Generative Grammar

https://kevinbinz.com/2017/09/05/intro-grammar/
Syntax- Compressed vs Unique Ruleset (1)
Syntax- Simple Tree (1)

Constituency Parsing


原 PPT 独立图片资源
Constituency Parsing

https://kevinbinz.com/2017/09/05/intro-grammar/
syntax_tree_construction

Dependency Grammar and Parsing


原 PPT 独立图片资源
Dependency Grammar and Parsing

原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture04-dep-parsing.pdf

Why Dependency


原 PPT 独立图片资源
Why Dependency

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture04-dep-parsing.pdf
Dependency goes back a long way:
To Pāṇini’s grammar (c. 5th century BCE)
Basic approach of 1st millennium Arabic grammarians
Solving the ambiguity in language understanding and parsing into the generative gramma actually rely on dependency information.
I eat the food [on the table] [in the morning].
I saw the man with the telescope. ?
原 PPT 独立图片资源

Dependency Parsing: Information Source


原 PPT 独立图片资源
Dependency Parsing: Information Source

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1162/handouts/SLoSP-2014-4-dependencies.pdf
Bilexical affinities: [issues → the] is plausible
Dependency distance: mostly with nearby words
Intervening material: Dependencies rarely span intervening verbs or punctuation
Valency of heads: How many dependents on which side are usual for a head?
原 PPT 独立图片资源

Relationship between Dependency and Constituency


原 PPT 独立图片资源
Relationship between Dependency and Constituency

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture04-dep-parsing.pdf
Projective parse: There are no crossing dependency arcs when the words are laid out in their linear order, with all arcs above the words
Dependencies corresponding to a Constituency Tree must be projective
Most syntactic structure is projective, but dependency theory normally does allow non-projective structures to account for displaced constituents.
原 PPT 独立图片资源

Open Information Annotation


原 PPT 独立图片资源
Open Information Annotation

Mingming Sun, et.al . 2020. A Predicate-Function-Argument Annotation of Natural Language for Open-Domain Information eXpression. In Proceedings of EMNLP, pages 2140–2150
原 PPT 独立图片资源

Why OIA


原 PPT 独立图片资源
Why OIA

Cognition happens in intermediate level:
Higher than byte, character, and words;
Lower than high-level real-world tasks;
Cognition is about entities, facts in the world and their relationships;
The basis of Cognitive Language Modeling is:
Entities, Facts and Relationships between them.

Intermediate Representation of Knowledge


原 PPT 独立图片资源
Intermediate Representation of Knowledge

Constants: express entities (“the solar system”) and any abstract concept (“expensive”);
Functions: 𝑓(𝑎𝑟𝑔1,⋯,𝑎𝑟𝑔𝑛)⟶{𝑒} express query of entities or delegation of entities, such as “the CEO of X”, the questions and wh-clause “when Y”
Predicates: p(𝑎𝑟𝑔1,⋯,𝑎𝑟𝑔𝑛)⟶{0,1}, express relationships or logical connections among entities, predicates, and functions, such as “X buy Y”, “X says Y”, “Y, because Z”.

Lingua Graph


原 PPT 独立图片资源
Lingua Graph

原 PPT 独立图片资源

Lingua Graph


原 PPT 独立图片资源
Lingua Graph

原 PPT 独立图片资源

Lingua Graph


原 PPT 独立图片资源
Lingua Graph

原 PPT 独立图片资源

Sentence Parsing: Intermediate Level Expression


原 PPT 独立图片资源
Sentence Parsing: Intermediate Level Expression

原 PPT 独立图片资源

Sentence Parsing: Break through the Limitation of Sequence


原 PPT 独立图片资源
Sentence Parsing: Break through the Limitation of Sequence

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1162/handouts/SLoSP-2014-4-dependencies.pdf
原 PPT 独立图片资源

Coreference


原 PPT 独立图片资源
Coreference

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/slides/cs224n-2019-lecture16-coref.pdf
Identify all mentions that refer to the same real world entity.
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Types of Coreference


原 PPT 独立图片资源
Types of Coreference

Pronominal Coreference
Pronouns referring to noun phrases
Example: “Lisa is a scientist. She works at NASA.”
Nominal Coreference
Two noun phrases referring to the same entity
Example: “The president gave a speech. The leader addressed the nation.”
Bridging Coreference
Inferring the relationship between mentions
Example: “I bought a laptop. The screen is amazing.”

Coreference is Complex


原 PPT 独立图片资源
Coreference is Complex

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/slides/cs224n-2019-lecture16-coref.pdf
原 PPT 独立图片资源

Coreference is Ccrucial


原 PPT 独立图片资源
Coreference is Ccrucial

Resolving Ambiguity in Language
“Mary told Sue that she won the award.”
Maintaining Context and Coherence
“A scientist discovered a new particle. The researcher presented her findings.”
Constructing Accurate Knowledge Graphs
“Google acquired DeepMind. The AI lab specializes in reinforcement learning.”
“Apple Inc. released a new iPhone. The Cupertino-based company expects high sales.”

Coherence Relations


原 PPT 独立图片资源
Coherence Relations 

https://www.sfu.ca/~mtaboada/lot/coherence_relations.html


Logical and semantic connections between sentences or discourse segments that contribute to the overall coherence of a text.
These relations help make discourse understandable and structured.
原 PPT 独立图片资源

Theoretical Foundations of Coherence Relations


原 PPT 独立图片资源
Theoretical Foundations of Coherence Relations



Rhetorical Structure Theory (RST)
Introduced by Mann & Thompson (1988).
Views text as a hierarchical structure with different relations.
Coherence is achieved through a network of nucleus (main idea) and satellite (supporting idea).
Example RST relations:
Cause-Effect
Elaboration
Contrast
Hobbs’ Theory of Coherence (1979)
Defines coherence as an inferential process where readers fill in missing links.
Emphasizes logical reasoning and causal inferences.
Example: John fell off his bike. He broke his arm.”
Humans infer a causal connection between these events.
Kehler’s Coherence Model (2002)
Inspired by theories of event coherence.
Divides coherence into three main categories:
Resemblance: Similarities between sentences.
Cause-Effect: Causal relationships.
Contiguity:: Events occurring in a sequence.

Types of Coherence Relations


原 PPT 独立图片资源
Types of Coherence Relations



Type
Description
Example
Causal Relations (因果关系)
One sentence (cause) leads to another (effect).
“It rained heavily. The streets got flooded.” (Cause → Effect)
- Result (结果关系)
The second sentence is a direct consequence of the first.
“She studied hard. She passed the exam.”
- Reason/Explanation (原因关系)
The second sentence explains the first.
“He left early because he was feeling sick.”
Temporal Relations (时间关系)
One event happens before, after, or at the same time as another.
“She woke up. Then she made coffee.”
- Sequential (顺序关系)
One event follows another.
“He finished work. Then he went home.”
- Simultaneous (同时关系)
Two events happen at the same time.
“She was cooking while he was reading.”
Contrastive Relations (对比关系)
The two segments express opposite ideas.
“She likes tea. He prefers coffee.”
- Contrast (对照关系)
Highlights a clear opposition.
“She loves winter, but I hate it.”
- Concession (让步关系)
Acknowledges one idea but presents another contradicting it.
“It was raining, but they still went out.”
Additive Relations (添加关系)
The second segment adds more information.
“She speaks English. She also knows French.”
- Elaboration (详细说明关系)
The second sentence expands on the first.
“He bought a car. It’s a red Tesla.”
- Parallel (并列关系)
The two segments describe similar ideas.
“She enjoys hiking. He likes running.”
Attribution Relations (引述关系)
One sentence attributes an action or statement to someone.
“The professor said that the exam would be postponed.”
- Statement (陈述关系)
Reporting an assertion.
“Experts claim that AI will change the world.”
- Question (提问关系)
Reporting a question.
“She asked if I could help her.”
Condition & Hypothetical Relations (条件关系)
Expresses a condition that affects an outcome.
“If you study hard, you will pass the exam.”
- Conditional (条件关系)
Explicit “if-then” condition.
“If it rains, we will cancel the trip.”
- Hypothetical (假设关系)
A condition that is unlikely or speculative.
“If I were rich, I would travel the world.”
Purpose & Means Relations (目的与手段关系)
Expresses a goal and the method to achieve it.
“He left early to catch the train.”
- Purpose (目的关系)
The second sentence explains the goal of the first.
“She trains daily to become a professional athlete.”
- Means (手段关系)
The second sentence describes how something is achieved.
“He solved the problem by using an equation.”

Importance of Coherence Relations


原 PPT 独立图片资源
Importance of Coherence Relations



Coherence relations play a crucial role in:
Text Understanding – Helps in resolving ambiguities in meaning.
Text Generation & Summarization – Ensures well-structured, logical output.
Discourse Parsing – Used in NLP tasks for organizing large texts.
Question Answering – Helps infer implicit knowledge.
Dialogue Systems – Maintains logical conversational flow

1.1 Symbolic Representation of Language 1.1.4 Obtaining Symbolic Representations



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.4 Obtaining Symbolic Representations

原 PPT 独立图片资源

Rule-Based and Heuristic Methods (Pre-1990s)


原 PPT 独立图片资源
Rule-Based and Heuristic Methods (Pre-1990s)



Expert Hand-Crafted Grammar Rules
In English, a verb (V) typically governs a noun phrase (NP) as its subject or object.
Rule: V → NP (subject), V → NP (object)
Limitations:
✔ High precision in well-defined cases
❌ Poor scalability, hard to adapt to new languages and ambiguous cases

Statistical Parsing (1990s–2000s)


原 PPT 独立图片资源
Statistical Parsing (1990s–2000s)



Probabilistic extension of Generative Gramma:
Each rule has an associated probability.
Data-Driven Statistical Parsing
Data Sets: Penn Treebank, Universal Dependencies.
Lexicalized Models: Assigned probabilities to dependencies based on word co-occurrence. The probability of a dependency relation 𝑑𝑒𝑝(ℎ𝑒𝑎𝑑, 𝑐ℎ𝑖𝑙𝑑) was often calculated using: 𝑃(𝑐ℎ𝑖𝑙𝑑∣ ℎ𝑒𝑎𝑑, 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛)
Example: If “eats” frequently governs “apple” in a treebank, then the parser learns that “eats → apple” is a probable dependency.
Limitations:
✔ More flexible than rule-based methods
❌ Struggled with long-range dependencies and ambiguous structures
P(S → NP VP) = 0.9
P(VP → V NP) = 0.7

Classical Machine Learning (2000s–2015)


原 PPT 独立图片资源
Classical Machine Learning (2000s–2015)



Transition-Based Dependency Parsing
Key Idea: Parse sentences by making a sequence of shift-reduce decisions.
Uses a classifier (SVM, perceptron, or neural network) to decide each parsing step.
Graph-Based Dependency Parsing
Key Idea: Treat parsing as a graph optimization problem.
Define a scoring function to determine the best dependency tree.
Limitations:
✔ Much Powerful than Statistical Parser;
❌ Need Feature Engineering; Long-term dependencies not well modeled.

Neural Network-Based Methods (2014–2017)


原 PPT 独立图片资源
Neural Network-Based Methods (2014–2017)



Machine Learning goes to Deep Learning:
Feature Engineering → Neural Embedding Representation
Shallow Machine → BiLSTM + Shallow Machine
Limitations:
✔ Long-term dependencies well modeled.
❌ Inherent pattern of language not used.

Pretrained Model (2018–2022)


原 PPT 独立图片资源
Pretrained Model (2018–2022)



Pretrained Model + Fine-tuning:
BiLSTM → Transformer
Random Initialization → Pretrained /Bert Weights
Limitations:
✔ Inherent pattern of language used.
❌ Information across tasks not used.

AGI (2022–)


原 PPT 独立图片资源
AGI (2022–)



Specific Task → AGI:
Hidden Task Info → Explicit Task Instructions
Output Labels → Natural Language Generation
Training on one task → Training on every task
Limitations:
✔ Information across tasks not used.
❌ Language instructions may not be accurate. Subtle task requirement may need finetuning.

Tree-Bank Datasets


原 PPT 独立图片资源
Tree-Bank Datasets



Tree Bank
Tree: Linguistic structures, particularly syntactic structures, are naturally represented as trees in formal grammar.
Bank: The dataset stores (or “banks”) a large number of syntactically annotated sentences.
Instances:
Penn Tree-Bank
Universal Dependencies
OntoNote

Penn Tree Bank (PTB)


原 PPT 独立图片资源
Penn Tree Bank (PTB)



One of the most influential linguistic corpora;
Developed at the University of Pennsylvania ;
Language: English
Annotation Type: Constituency-based phrase structure trees, POS tags, and some dysfluency annotations.
Source Data:
Wall Street Journal (WSJ) articles (about 1 million words).
Brown Corpus (a balanced dataset of American English).
Switchboard (spoken conversations).
ATIS-3 material
Annotation Scheme:
Each word is labeled with a POS tag.
Phrases are marked with constituency labels (e.g., NP, VP, PP).

原 PPT 独立图片资源

Universal Dependencies (UD)


原 PPT 独立图片资源
Universal Dependencies (UD)



Cross-linguistic dependency annotation that standardizes the annotation of syntactic structure and morphology across many languages.  https://universaldependencies.org/
Language: More than 100 languages;
Annotation Type: Cross-Linguistic Consistency; Universal POS Tags; Morphological Features
Annotation Scheme: CoNLL-U format

1 She she PRON _ Number=Sing|Person=3 2 nsubj _ _
2 said say VERB _ Tense=Past 0 root _ _
3 that that SCONJ _ _ 5 mark _ _
4 he he PRON _ Number=Sing|Person=3 5 nsubj _ _
5 likes like VERB _ Tense=Pres|Number=Sing 2 ccomp _ _
6 apples apple NOUN _ Number=Plur 5 obj _ _
7 . . PUNCT _ _ 2 punct _ _

OntoNote


原 PPT 独立图片资源
OntoNote



Large-scale, multi-layered corpus offering syntactic, semantic, and discourse-level annotations
Created as part of the DARPA GALE program;
Language: English, Chinese, Arabic
Annotation Type: POS tags, Constituency Trees, Word Sense Disambiguation, Named Entity Recognition, Coreference Resolution, PropBank Semantic Role Labeling (SRL).
Source Data:
Newswire (Wall Street Journal, Xinhua News)
Broadcast News (CNN, MSNBC)
Broadcast Conversations (Transcribed interviews)
Web Texts (Blogs, discussion forums)
Telephone Conversations (Switchboard corpus)
Annotation Scheme: CoNLL-U format and others

Comparison


原 PPT 独立图片资源
Comparison



Feature
Penn Treebank (PTB)
Universal Dependencies (UD)
OntoNotes
POS Tagging
✅ Yes
✅ Yes
✅ Yes
Constituency Trees
✅ Yes
❌ No
✅ Yes
Dependency Parsing
❌ No
✅ Yes
❌ No
Named Entities (NER)
❌ No
❌ No
✅ Yes
Coreference
❌ No
❌ No
✅ Yes
Semantic Role Labeling
❌ No
❌ No
✅ Yes
Word Sense Disambiguation (WSD)
❌ No
❌ No
✅ Yes
Multilingual
❌ No
✅ Yes
✅ Yes (Eng, Chi, Ara)

Types of Tasks


原 PPT 独立图片资源
Types of Tasks



Segmentation
Element Identification
Element Attribute Prediction
Pairwise Relationship Identification

Recall: Neural Learning Paradigms


原 PPT 独立图片资源
Recall: Neural Learning Paradigms
原 PPT 独立图片资源

Recall: Applications: Sequence Classification


原 PPT 独立图片资源
Recall: Applications: Sequence Classification
Feed the last hidden-state to the classifier;
Widely used in NLP task such as text classification;
Text classification model based on recurrent neural network.

Recall: Applications: Sequence Labeling


原 PPT 独立图片资源
Recall: Applications: Sequence Labeling
Predict a label for each item of input sequence.
Widely used in NLP task like NER;
bilstm

Recall: Applications: Sequence Prediction


原 PPT 独立图片资源
Recall: Applications: Sequence Prediction
Modeling the problem as a sequence prediction task;
Typical application: Machine Translation, AGI LLM;
原 PPT 独立图片资源
原 PPT 独立图片资源

Segmentation —— Task Modeling


原 PPT 独立图片资源
Segmentation —— Task Modeling



Modeling as a Sequence Labeling Task



Modeling as a Sequence Prediction Task
https://medium.com/towards-data-science/joint-khmer-word-segmentation-and-pos-tagging-cad650e78d30
原 PPT 独立图片资源
STA
STA
STA
STA

Segmentation —— Network Architecture


原 PPT 独立图片资源
Segmentation —— Network Architecture



原 PPT 独立图片资源
https://medium.com/towards-data-science/joint-khmer-word-segmentation-and-pos-tagging-cad650e78d30

Segmentation —— Algorithm


原 PPT 独立图片资源
Segmentation —— Algorithm



Loss:
Typical Classification Loss: Cross-Entropy
Optimization:
Gradient Descent: Adam

Segmentation —— Evaluation


原 PPT 独立图片资源
Segmentation —— Evaluation



Model Level Evaluation:
Binary Classification Error: Precision, Recall, F1
Task Level Evaluation:
Words Prediction Performance: Precision, Recall, F1
Boundary Prediction Performance: Precision, Recall, F1

Element Identification——Task Modeling


原 PPT 独立图片资源
Element Identification——Task Modeling



Same with Segmentation, just change the target label system.
Typical Task: Noun Phrase Identification, Entity Detection
https://medium.com/@muskaan.maurya06/name-entity-recognition-and-various-tagging-schemes-533f2ac99f52
手机屏幕的截图

AI 生成的内容可能不正确。

Element Attribute Prediction——Task Modeling


原 PPT 独立图片资源
Element Attribute Prediction——Task Modeling



Same with Segmentation, just change the target label system.
Typical Task: POSTag Prediction, Named Entity Recognition
Can merge with Element Identification Task by merging the label system.
https://www.researchgate.net/publication/330424080_Adverse_Drug_Events_Detection_in_Clinical_Notes_by_Jointly_Modeling_Entities_and_Relations_Using_Neural_Networks
BIO Tagging for the entity extraction.

Pairwise Relationship Identification——Task Modeling


原 PPT 独立图片资源
Pairwise Relationship Identification——Task Modeling



Elementwise Representation + Bi-Affine transformation → Pairwise Relationship
Bi-Affine transformation: 𝐵𝑖𝐴𝑓𝑓𝑖𝑛𝑒(𝑥1,𝑥2) is a vector with i-th element equals to:
原 PPT 独立图片资源

Pairwise Relationship Identification——Task Modeling


原 PPT 独立图片资源
Pairwise Relationship Identification——Task Modeling



Task Constraint:
Dependency Parsing: Tree structure;
Coreference: Multiple Fully Connected Graph;
OIA Graph: DAG;
原 PPT 独立图片资源
https://www.cs.princeton.edu/courses/archive/fall19/cos484/lectures/lec11.pdf

Pairwise Relationship Identification——Network Architecture


原 PPT 独立图片资源
Pairwise Relationship Identification——Network Architecture



原 PPT 独立图片资源
https://arxiv.org/pdf/1611.01734

Parsing Complex Graphs: Word-OIA


原 PPT 独立图片资源
Parsing Complex Graphs: Word-OIA

原 PPT 独立图片资源
Xin Wang, et. al. 2022. OIE@OIA: an Adaptable and Efficient Open Information Extraction Framework. In Proceedings of ACL, pages 6213–6226, Dublin, Ireland.

1.1 Symbolic Representation of Language 1.1.5 Suggested Readings



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.5 Suggested Readings

原 PPT 独立图片资源

Books


原 PPT 独立图片资源
Books
Speech and Language Processing, Dan Jurafsky and James H. Martin. https://web.stanford.edu/~jurafsky/slp3/
Neural Network Methods in Natural Language Processing, Yoav Goldberg

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源

Natural Language UnderstandingSymbolic Representation of Knowledge


原 PPT 独立图片资源
Natural Language
Understanding

Symbolic Representation of
Knowledge
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
History of Symbolic Knowledge Representation
Knowledge Based System
Knowledge Graph Overview
Knowledge Graph Construction
Knowledge Graph Refinement

1.2 Symbolic Representation of Knowledge 1.2.1 History of Symbolic Knowledge Representation



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.1 History of Symbolic Knowledge Representation

原 PPT 独立图片资源

Logical and Semantic Representations——Richard Montague, 1970s: Montague Grammar


原 PPT 独立图片资源
Logical and Semantic Representations——Richard Montague, 1970s: Montague Grammar
Language can be given a formal, logical structure, much like the languages of mathematics and logic. (English as a Formal Language, 1970, and Universal Grammar, 1973)

原 PPT 独立图片资源
原 PPT 独立图片资源
https://en.wikipedia.org/wiki/Montague_grammar

Knowledge-Based Representations——Semantic Networks (1960s–1980s)


原 PPT 独立图片资源
Knowledge-Based Representations——Semantic Networks (1960s–1980s)
A semantic network is a directed graph :
Nodes represent concepts (entities, words, or ideas).
Edges represent relationships between concepts.
Instances: WordNet, ConceptNet

2: A semantic network for knowledge representation. | Download Scientific  Diagram
https://www.researchgate.net/publication/2701045_On_Generating_Semantic_Dispositions_in_a_Given_Subject_Domain

Types of Semantic Relations in Semantic Networks


原 PPT 独立图片资源
Types of Semantic Relations in Semantic Networks

Category
Relation Type
Description
Example
Hierarchical Relations (Taxonomy)
is-a (Hypernymy/Hyponymy)
X is a subtype of Y (generalization/specialization)
Dog is-a Mammal
part-of (Meronymy/Holonymy)
X is a part of Y
Wheel part-of Car
has-a (Possession)
Y possesses X
Bird has-a Beak
Lexical Relations
Synonymy
X and Y have similar meanings
Big ↔ Large
Antonymy
X and Y have opposite meanings
Hot ↔ Cold
Homonymy
X and Y have the same form but different meanings
Bank (financial) vs. Bank (riverbank)
Polysemy
X has multiple related meanings
Mouth (of a river, of a person)
Associative Relations
Causality (Cause-Effect)
X causes Y
Fire causes Smoke
Entailment
If X is true, then Y must be true
Kill entails Die
Related-to
General associative relation
Doctor related-to Hospital
Thematic Roles (Case Roles)
Agent
The doer of an action
John (agent) eats an apple
Patient
The receiver of an action
John eats an Apple (patient)
Instrument
The means used for an action
He wrote with a Pen (instrument)
Experiencer
The entity experiencing a state
She felt Happy (experiencer)
Event-Based Relations
Precondition
X must occur before Y
Study is required for Passing
Temporal (Before/After)
X occurs before/after Y
Breakfast before Work
Subevent
X is a part of Y as an event
Buying includes Paying
Linguistic Relations
Coreference
X and Y refer to the same entity
John said he is happy. (he = John)
Anaphora
X refers back to Y in discourse
Mary saw a dog. It was barking.
Ellipsis
Omitted elements understood from context
I like coffee, and she does too. (ellipsis of “likes coffee”)

Knowledge-Based Representations——Frame Semantics (Charles Fillmore, 1970s)


原 PPT 独立图片资源
Knowledge-Based Representations——Frame Semantics (Charles Fillmore, 1970s)
Structured representation of knowledge that provides a context for understanding words or concepts. It consists of:
Participants (Entities in the frame)
Roles (How participants interact)
Relations (Connections between roles and entities)
With the BUYING frame, when someone says: “Alice purchased a book from the store for $10.”
Instance: FrameNet

Role
Example
Buyer
Alice
Seller
Store
Goods
A book
Money
$10

The Central Problem: Surface Expression and Underlying Meaning


原 PPT 独立图片资源
The Central Problem:
Surface Expression and Underlying Meaning


Theory
View on Surface Expression
View on Underlying Meaning
Key Mechanism of Mapping
Example
Montague Semantics (Montague, 1970s)
Language is a formal system like logic.
Meaning is truth conditions in a formal logical system.
Lambda calculus and first-order logic assign formal meaning to expressions.
“Every student passed” = ∀x (Student(x) → Passed(x)) in logic.
Frame Semantics (Fillmore, 1970s)
Words activate semantic frames (structured scenarios).
Meaning is a network of roles (agent, patient, action).
Words are linked to real-world experiences.
“Buy” involves a buyer, seller, goods, and money.
Knowledge-Based Representations (WordNet, ConceptNet, 1990s–present)
Words are nodes in a network of meanings.
Meaning is defined by relations to other concepts.
Hierarchical & associative links between concepts.
“Dog” → “is-a” → “Mammal”.

1.2 Symbolic Representation of Knowledge 1.2.2 Knowledge-Based Systems



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.2 Knowledge-Based Systems

原 PPT 独立图片资源

Components of Knowledge Based System


原 PPT 独立图片资源
Components of Knowledge Based System
原 PPT 独立图片资源
https://www.techtarget.com/searchcio/definition/knowledge-based-systems-KBS

Example of Schema/Ontology


原 PPT 独立图片资源
Example of Schema/Ontology
A person owns a dog:
(isa Person Human)
(isa Dog Animal)
(owns ?person ?dog)# A person owns a dog.
A person is available:
(available ?person) # A person is available.
The weather is good:
(isa Weather Condition)
(goodWeather ?weather) # The weather is good.
Taking a dog for a walk involves the action of walking the dog:
(takesForWalk ?person ?dog) # This indicates the specific action of a person taking their dog for a walk.

Example of Knowledge


原 PPT 独立图片资源
Example of Knowledge
Person availability:
(isa John Person)
(available John) #(John is available)
Ownership of the dog:
(owns John Rover) #(John owns a dog named Rover)
Weather condition:
(goodWeather Sunny) #(The weather is sunny, so it’s good)

Example of Reasoning Rules


原 PPT 独立图片资源
Example of Reasoning Rules
If a person owns a dog, is available, and the weather is good, they are likely to take the dog for a walk:
(=> (and (owns ?person ?dog) (isa ?person Person) (goodWeather ?weather)) (canTakeForWalk ?person ?dog))
When apply this rule to the previous given knowledge, the following knowledge can be obtained:
(canTakeForWalk John Rover) #(John is likely to take Rover for a walk).

1.2 Symbolic Representation of Knowledge 1.2.3 Successes of Knowledge-Based Systems



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.3 Successes of Knowledge-Based Systems

原 PPT 独立图片资源

Early Breakthroughs (1950s-1970s)Logic Theorist (1956)


原 PPT 独立图片资源
Early Breakthroughs (1950s-1970s)
Logic Theorist (1956)
Developed by Newell and Simon
Contribution: the first program capable of automatically proving mathematical theorems. It successfully proved 38 theorems from Principia Mathematica and even proposed more concise proof methods.
Significance: This was the first demonstration of how symbolic systems can simulate human reasoning, marking the birth of the symbolic school.
原 PPT 独立图片资源
https://guillaumelauzier.com/1st-ai-projects/

Early Breakthroughs (1950s-1970s)General Problem Solver (GPS, 1957)


原 PPT 独立图片资源
Early Breakthroughs (1950s-1970s)
General Problem Solver (GPS, 1957)
Developed by Newell and Simon
Principle: The GPS used “Means-Ends Analysis” to decompose problems into sub-goals, solving them step by step.
Application: It was applied to classic problems like the Tower of Hanoi and cryptarithmetic, demonstrating that symbolic systems can handle structured problems.
原 PPT 独立图片资源
https://guillaumelauzier.com/1st-ai-projects/

Early Breakthroughs (1950s-1970s)LISP Programming Language (1958)


原 PPT 独立图片资源
Early Breakthroughs (1950s-1970s)
LISP Programming Language (1958)
Developer: John McCarthy.
Features: Specifically designed for symbolic processing, it supports recursion, list manipulation, and dynamic memory management, becoming a core tool for symbolic AI.
Impact: LISP influenced almost all programming language.
The Klisp repl
https://dotink.co/posts/klisp/

Early Breakthroughs (1950s-1970s)ELIZA: Pioneering the Conversation with Machines (1966)


原 PPT 独立图片资源
Early Breakthroughs (1950s-1970s)
ELIZA: Pioneering the Conversation with Machines (1966)
‌Developed in the mid-1960s by Joseph Weizenbaum at MIT
Contribution: It was one of the first AI programs to be developed and was designed to mimic human conversation by using a set of rules and patterns to generate responses to user input.‌
‌Foundational Impact:‌ It inspired future developments in natural language processing (NLP) and chatbots, paving the way for modern AI-driven systems.
原 PPT 独立图片资源
https://guillaumelauzier.com/1st-ai-projects/

Golden Age of Expert Systems (1970s-1980s)Edward Feigenbaum: Pioneer in Expert Systems and AI


原 PPT 独立图片资源
Golden Age of Expert Systems (1970s-1980s)
Edward Feigenbaum: Pioneer in Expert Systems and AI
Background:
Born 1936, Ph.D. in Computer Science from Stanford University.
Renowned for his work in symbolic AI and knowledge-based systems.
Key Contributions: DENDRAL (1960s-70s) and MYCIN (1970s)
Core Beliefs:
Advocated for symbolic AI using formal logic and rule-based reasoning to represent knowledge and simulate human intelligence.
Emphasized the importance of knowledge representation in solving complex problems.
Legacy:
Turing Award (1994) for his pioneering work in AI.
His work laid the foundation for modern AI applications in medicine, science, and more.
https://www.britannica.com/biography/Edward-Albert-Feigenbaum
Edward Albert Feigenbaum | AI Pioneer, Computer Scientist ...

Golden Age of Expert Systems (1970s-1980s)DENDRAL System (1965-1970s)


原 PPT 独立图片资源
Golden Age of Expert Systems (1970s-1980s)
DENDRAL System (1965-1970s)
Domain: Chemical mass spectrometry analysis.
Function: Used rule-based reasoning to identify the molecular structure of unknown organic compounds.
Reasons for Success:
Combined a chemical expert knowledge base with heuristic search algorithms.
Achieved or even surpassed the accuracy of human experts in experiments.
https://expertsystem101.weebly.com/dendral.html
Picture

Golden Age of Expert Systems (1970s-1980s)MYCIN System (1976)


原 PPT 独立图片资源
Golden Age of Expert Systems (1970s-1980s)
MYCIN System (1976)
Domain: Medical diagnosis (blood infections).
Technical Highlights:
Used “production rules” (If-Then rule bases) to represent medical knowledge.
Introduced uncertainty reasoning (confidence factors).
Outcome:
Achieved diagnostic accuracy comparable to top doctors, becoming a milestone in medical AI.
https://medium.com/version-1/an-overview-of-the-rise-and-fall-of-expert-systems-14e26005e70e
原 PPT 独立图片资源

Golden Age of Expert Systems (1970s-1980s)XCON (R1) System (1980s)


原 PPT 独立图片资源
Golden Age of Expert Systems (1970s-1980s)
XCON (R1) System (1980s)
Domain: Computer hardware configuration (DEC Corporation).
Value: Saved DEC $25 million annually and reduced error rates from 30% to 2%.
Significance:
The first large-scale commercial expert system, demonstrating the practical economic value of symbolic AI.

Deepening Knowledge Engineering and Knowledge RepresentationParadigms of Knowledge Representation


原 PPT 独立图片资源
Deepening Knowledge Engineering and Knowledge Representation
Paradigms of Knowledge Representation
Frame Theory (1974):
Proposed by Marvin Minsky, it uses “frames” to represent the structured knowledge of objects.
Semantic Networks:
Represent relationships between concepts using nodes and edges, laying the foundation for knowledge graphs.
undefined
https://en.wikipedia.org/wiki/Semantic_network#/media/File:Semantic_Net.svg

Deepening Knowledge Engineering and Knowledge RepresentationCyc Project (1984)


原 PPT 独立图片资源
Deepening Knowledge Engineering and Knowledge Representation
Cyc Project (1984)
Goal: To build a massive knowledge base of human common sense.
Method: Manually encoding millions of logical assertions (e.g., “water is a liquid”).
Controversy and Insights: Although the goal was not fully achieved, it highlighted the challenges of symbolic systems in handling common sense and advanced knowledge representation research.
https://www.researchgate.net/figure/The-Upper-Cyc-Ontology_fig2_228368116
The Upper Cyc Ontology. 

Deepening Knowledge Engineering and Knowledge RepresentationProlog Language (1972)


原 PPT 独立图片资源
Deepening Knowledge Engineering and Knowledge Representation
Prolog Language (1972)
Developer: Alain Colmerauer.
Features: A declarative programming language based on first-order logic, supporting automatic reasoning.
Application: Core tool in Japan’s “Fifth Generation Computer Project,” promoting the international development of symbolic AI.
https://medium.com/@neerajsharma95/understanding-logical-programming-paradigm-with-prolog-49b738a293ca
原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.4 Limitations of Knowledge-Based Systems



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.4 Limitations of Knowledge-Based Systems

原 PPT 独立图片资源

Technical Bottlenecks and Classic FailuresDecline of Expert Systems (1980s-1990s)


原 PPT 独立图片资源
Technical Bottlenecks and Classic Failures
Decline of Expert Systems (1980s-1990s)
Case: XCON System (DEC Corporation)
Early Success:
In the 1980s, the XCON system saved DEC tens of millions of dollars annually.
Reasons for Failure:
High Knowledge Maintenance Costs: Hardware configuration rules frequently updated with product iterations required continuous manual intervention.
Poor Generalization: The system could not be adapted to other domains (e.g., software design).
Outcome:
By the 1990s, DEC went bankrupt, and XCON was discontinued.
Industry-wide Collapse:
Companies found that the cost of developing expert systems far exceeded the benefits, and the commercial value bubble burst.
In 1993, the U.S. AI industry market shrank by 50%, leading to what is known as the “AI Winter.”

Technical Bottlenecks and Classic FailuresCyc Project: The Utopian Failure of Common Sense Engineering (1984-Present)


原 PPT 独立图片资源
Technical Bottlenecks and Classic Failures
Cyc Project: The Utopian Failure of Common Sense Engineering (1984-Present)
Goal:
Build a knowledge base covering human common sense (e.g., “birds can fly,” “glass cups are fragile”).
Failures:
Knowledge Encoding Disaster: After 40 years and over 25 million manually entered logical assertions, the complexity of common sense was still far from being covered.
Rigid Reasoning: Could not handle exceptions (e.g., “penguins are birds but cannot fly”) and context-dependent situations (e.g., “water is drinkable” vs. “floods are dangerous”).
Lesson:
Common sense cannot be realized through exhaustive rule systems as a “closed system.”

Technical Bottlenecks and Classic FailuresJapan’s Fifth Generation Computer Project (1982-1992)


原 PPT 独立图片资源
Technical Bottlenecks and Classic Failures
Japan’s Fifth Generation Computer Project (1982-1992)
Ambition:
Build an “intelligent computer” based on Prolog and symbolic logic to lead the global AI field.
Reasons for Failure:
Single Technology Path: Over-reliance on symbolic reasoning, neglecting other capabilities such as perception and learning.
Mismatch between Hardware and Algorithms: Parallel reasoning machines were difficult to implement, and Prolog’s efficiency was inadequate.
Outcome:
With a decade-long investment of $850 million, the results were almost negligible, severely damaging the international reputation of symbolic AI.

Theoretical Defects of the Symbolic SchoolKnowledge Acquisition Bottleneck


原 PPT 独立图片资源
Theoretical Defects of the Symbolic School
Knowledge Acquisition Bottleneck
Problem:
Knowledge depends on manual encoding, making it difficult to automatically learn from data.
Case: MYCIN Medical System
The system required doctors to input thousands of diagnostic rules, which took years.
When faced with new diseases (e.g., AIDS), the system could not autonomously update its knowledge base.

Theoretical Defects of the Symbolic SchoolInability to Handle Uncertainty


原 PPT 独立图片资源
Theoretical Defects of the Symbolic School
Inability to Handle Uncertainty
Rigidity of Symbolic Logic:
Symbolic systems only support binary logic (True/False), making it impossible to handle probability, vagueness, or partial truths.
Counterexample:
In autonomous driving, real-time decisions like “a pedestrian suddenly appears” require probabilistic reasoning, which symbolic systems cannot address.

Theoretical Defects of the Symbolic SchoolCombinatorial Explosion


原 PPT 独立图片资源
Theoretical Defects of the Symbolic School
Combinatorial Explosion
Problem:
The search space of symbolic reasoning grows exponentially with the size of the problem.
Case: Automated Theorem Provers
These systems work efficiently for logical propositions with up to 10 steps but become uncontrollable in computation time when the number of steps exceeds 100.
In 1996, IBM’s Deep Blue defeated the world chess champion, relying on brute-force search rather than symbolic reasoning.

Theoretical Defects of the Symbolic SchoolLack of Common Sense and Contextual Understanding


原 PPT 独立图片资源
Theoretical Defects of the Symbolic School
Lack of Common Sense and Contextual Understanding
Vulnerability of Symbolic Systems:
Symbolic systems cannot understand “implicit knowledge” (e.g., “a cup overturned will spill water” requires physical common sense).
Classic Failures:
Early chatbots (e.g., ELIZA) simply matched keywords, leading to responses that were often absurd or nonsensical.

External Shocks and Paradigm ShiftRise of Connectionism (1980s-2000s)


原 PPT 独立图片资源
External Shocks and Paradigm Shift
Rise of Connectionism (1980s-2000s)
Technological Comparison:
Symbolism: Rule-driven (logic designed by humans), high interpretability, poor generalization.
Connectionism: Data-driven (automatically learning features), low interpretability, strong generalization.
Turning Point:
In 2012, AlexNet outperformed traditional symbolic methods in the ImageNet image recognition competition, and deep learning became mainstream.

External Shocks and Paradigm ShiftBig Data and Computational Power Revolution


原 PPT 独立图片资源
External Shocks and Paradigm Shift
Big Data and Computational Power Revolution
Critical Weakness of Symbolism:
Symbolic AI relies on small-scale manually annotated knowledge and cannot take advantage of the massive amounts of unstructured data available in the internet era (e.g., text, images).
Case Comparison:
Symbolic School: The Cyc project spent 40 years building 25 million rules.
Connectionism: GPT-3 learned 175 billion parameters from 450 TB of text data through automatic learning.

External Shocks and Paradigm ShiftShift in Industry Demands


原 PPT 独立图片资源
External Shocks and Paradigm Shift
Shift in Industry Demands
Reasons for Enterprises Abandoning Symbolic AI:
Expert systems had long development cycles and high costs, whereas deep learning models can iterate quickly (e.g., recommendation systems, speech recognition).
Symbolic systems struggled to be embedded into real-time applications (e.g., autonomous driving requires millisecond-level responses).

Simplified Version: Knowledge Graph


原 PPT 独立图片资源
Simplified Version: Knowledge Graph
Focus on Specific Domains/Tasks/Products
Simplified Knowledge Expression
Natural Incorporate Machine Learning Techniques for Acquiring and Maintaining Knowledge
Able to Handling Ambiguity and Uncertainty
Simplified Reasoning Methods

1.2 Symbolic Representation of Knowledge 1.2.5 Introduction to Knowledge Graphs



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.5 Introduction to Knowledge Graphs

原 PPT 独立图片资源

Representing Knowledge via Graphs


原 PPT 独立图片资源
Representing Knowledge via Graphs
Why not Tables?
Example: Table Event(name, venue, type, start, end)
Problems: multiple names, multiple types, unknown fields
New representing method:
EventName(id,name);
EventStart(id,start);
EventEnd(id,end);
EventVenue(id,venue);
EventType(id,type);
原 PPT 独立图片资源

Schema


原 PPT 独立图片资源
Schema
Schema prescribes a high-level structure and/or semantics that the graph follows or should follow.
原 PPT 独立图片资源
Aidan Hogan, et.al, Knowledge Graphs, 2021
原 PPT 独立图片资源

Identity


原 PPT 独立图片资源
Identity
Globally-unique identifiers to help avoid such ambiguity.
原 PPT 独立图片资源
Aidan Hogan, et.al, Knowledge Graphs, 2021

Context


原 PPT 独立图片资源
Context

Many (arguably all) facts can be considered true with respect to a certain context.
原 PPT 独立图片资源
Aidan Hogan, et.al, Knowledge Graphs, 2021

Representing Knowledge as Graphs


原 PPT 独立图片资源
Representing Knowledge as Graphs
原 PPT 独立图片资源
Taxonomy
Ontologies

More Examples: Scientific Publications


原 PPT 独立图片资源
More Examples: Scientific Publications
原 PPT 独立图片资源

More Examples: E-Commerce


原 PPT 独立图片资源
More Examples: E-Commerce
原 PPT 独立图片资源

More Examples: Social Networks


原 PPT 独立图片资源
More Examples: Social Networks
原 PPT 独立图片资源

More Examples: Geopolitical Events


原 PPT 独立图片资源
More Examples: Geopolitical Events
原 PPT 独立图片资源

Query/Edit the Graphs


原 PPT 独立图片资源
Query/Edit the Graphs
MATCH (startContent:Content)-[relationship:IS_RELATED_TO]->(endContent:Content) WHERE endContent.source = 'user' OPTIONAL MATCH (endContent)-[r]-() DELETE relationship, endContent
Cypher for Neo4J

Framework: Resource Description Framework


原 PPT 独立图片资源
Framework: Resource Description Framework

原 PPT 独立图片资源

Framework: Wikidata


原 PPT 独立图片资源
Framework: Wikidata

原 PPT 独立图片资源

Framework: Semantic Web


原 PPT 独立图片资源
Framework: Semantic Web

原 PPT 独立图片资源

The Value of Knowledge Graph in Web Search


原 PPT 独立图片资源
The Value of Knowledge Graph in Web Search
原 PPT 独立图片资源
原 PPT 独立图片资源

The Value of Knowledge Graph in LLM Ages


原 PPT 独立图片资源
The Value of Knowledge Graph in LLM Ages
Memory System for LLM
Reducing Hallucinations with Reliable Knowledge Source (vs LLM Knowledge Editing)
Multi-Hop reasoning and efficient calculation over Graphs
Entity Linking and Disambiguation
Cross-Document Information Gathering.

1.2 Symbolic Representation of Knowledge 1.2.6 Knowledge Graph Construction



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.6 Knowledge Graph Construction

原 PPT 独立图片资源

Whole Picture


原 PPT 独立图片资源
Whole Picture
原 PPT 独立图片资源

Knowledge extraction: Schema or Not?


原 PPT 独立图片资源
Knowledge extraction: Schema or Not?
Dimension
Schema-based
Schema-free
Prior knowledge
Strong (know what you need)
Weak (discover what the text contains)
Quality profile
Precise, consistent, alignable
Broad, noisy, needs post-processing
Evolution
Schema changes are costly
Easy to absorb new relations
Typical pipeline
Define schema → extract → ingest
Open extraction → induce/normalize → ingest

1.2 Symbolic Representation of Knowledge 1.2.7 Schema-Based Knowledge Extraction



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.7 Schema-Based Knowledge Extraction

原 PPT 独立图片资源

Expert Interview and Schema Design


原 PPT 独立图片资源
Expert Interview and Schema Design
Expert Interviews: To capture domain expertise and tacit knowledge, identify key entities, relationships, and attributes within the domain.
Schema Design: To define the structure of the Knowledge Graph, ensuring it can accurately represent the domain knowledge.
Output: A well-defined schema that captures domain knowledge efficiently and serves as the blueprint for building the Knowledge Graph.

Seed Sample Mining or Labeling


原 PPT 独立图片资源
Seed Sample Mining or Labeling
Seed Sample Mining: To identify a small set of initial entities and relationships that can serve as the foundation of the Knowledge Graph under the guidance of the schema;
Labeling Existing Data: To manually label data or use semi-automated techniques to identify relevant entities and relationships in existing datasets.
Outcome: This stage produces a set of labeled or mined seed samples that will form the basis of the Knowledge Graph.

Training Knowledge Acquisition Components


原 PPT 独立图片资源
Training Knowledge Acquisition Components
Named Entity Recognition (NER): Identify and classify entities (into the schema’s typing system) from unstructured data.
Relation Extraction: Automatically detect and extract relationships between entities.
Entity Linking: Connect identified entities to their corresponding entries the knowledge graph;

Acquire Knowledges and Evaluation


原 PPT 独立图片资源
Acquire Knowledges and Evaluation
Run the trained acquisition components at scale on unstructured text to populate the Knowledge Graph;
Or acquire knowledge from other sources (table, other KG, LLM);
Systematically evaluating the quality and coverage of the acquired knowledge.
Outcome: This stage delivers a populated, structured, and evaluated Knowledge Graph ready for deployment.

1.2 Symbolic Representation of Knowledge 1.2.8 Named Entity Recognition



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.8 Named Entity Recognition

原 PPT 独立图片资源

Named Entity Recognition For Specific Domain/Task


原 PPT 独立图片资源
Named Entity Recognition For Specific Domain/Task
Generally, tasks want entities belonging to several given types and neglect other entities.
Joint learning detection and typing.
https://www.researchgate.net/publication/330424080_Adverse_Drug_Events_Detection_in_Clinical_Notes_by_Jointly_Modeling_Entities_and_Relations_Using_Neural_Networks
BIO Tagging for the entity extraction.

Existing Models and Tools: Spacy


原 PPT 独立图片资源
Existing Models and Tools: Spacy
原 PPT 独立图片资源
原 PPT 独立图片资源
https://spacy.io/usage/linguistic-features#named-entities

Challenge: Nested Entities


原 PPT 独立图片资源
Challenge: Nested Entities


An example of a sentence annotated with nested named entity tags ...
Sławomir Dadas, Bidirectional Iterative Algorithm for Nested Named Entity Recognition, 2020
Sławomir Dadas at National Information Processing Institute

Challenge: Different Criterion


原 PPT 独立图片资源
Challenge: Different Criterion


原 PPT 独立图片资源
The Great wall
The Great wall, Beijing
Yuming Yang, et. al, Beyond Boundaries: Learning a Universal Entity Taxonomy across Datasets and Languages for Open Named Entity Recognition, 2024

1.2 Symbolic Representation of Knowledge 1.2.9 Relation Extraction



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.9 Relation Extraction

原 PPT 独立图片资源

Relation Extraction Problem


原 PPT 独立图片资源
Relation Extraction Problem
Relation extraction. The example is chosen from the MADE corpus, where blue, yellow, red, and green circles denote Duration, Drug, ADE, and Severity entities, respectively; duration and adverse, severity type denote different types of relations.
Multiple Features for Clinical Relation Extraction: a Machine Learning Approach, Journal of Biomedical Informatics 
When a pair of Entities/Interested Phrases are detected, one can identify whether there exists a relation between them, and the type of the relation if exists.

Relation Extraction Methods


原 PPT 独立图片资源
Relation Extraction Methods

原 PPT 独立图片资源
https://arxiv.org/pdf/2306.02051v3

Beyond the Sentence Level


原 PPT 独立图片资源
Beyond the Sentence Level

原 PPT 独立图片资源
https://aclanthology.org/D19-3029.pdf

Existing Models and Tools


原 PPT 独立图片资源
Existing Models and Tools

OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction
原 PPT 独立图片资源
https://aclanthology.org/D19-3029.pdf

1.2 Symbolic Representation of Knowledge 1.2.10 Schema-Free Knowledge Extraction



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.10 Schema-Free Knowledge Extraction

原 PPT 独立图片资源

Discover Knowledge System from Plain Text


原 PPT 独立图片资源
Discover Knowledge System from Plain Text
Knowledge = Facts and relation between facts;
All these can be naturally expressed by human language;
But how
language express complex knowledge?
knowledge be organized in human brain for reasoning ability and efficiency?
Towards universal knowledge schema.

1.2 Symbolic Representation of Knowledge 1.2.11 Open Information Extraction



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.11 Open Information Extraction

原 PPT 独立图片资源

Open Information Extraction


原 PPT 独立图片资源
Open Information Extraction
Definition: extract structured information (e.g., relationships, facts) from unstructured text without relying on predefined schema.
Goal: To discover triples (subject, relation, object) directly from text.
𝐿𝑖𝑠𝑎 𝑖𝑠 𝑡𝑎𝑘𝑖𝑛𝑔 𝑐𝑎𝑟𝑒 𝑜𝑓 𝑡ℎ𝑒 𝑏𝑎𝑏𝑦 → < 𝐿𝑖𝑠𝑎, 𝑖𝑠 𝑡𝑎𝑘𝑖𝑛𝑔 𝑐𝑎𝑟𝑒 𝑜𝑓, 𝑡ℎ𝑒 𝑏𝑎𝑏𝑦 >
𝑇𝑜𝑚 𝑔𝑖𝑣𝑒 𝑚𝑒 𝑎 𝑏𝑜𝑜𝑘→ <Tom,𝑔𝑖𝑣𝑒, 𝑚𝑒, 𝑎 𝑏𝑜𝑜𝑘>

IE vs OIE


原 PPT 独立图片资源
IE vs OIE

Feature
Information Extraction (IE)
Open Information Extraction (OpenIE)
Predefined Schemas
Requires predefined schemas.
Does not require predefined schemas.
Relation Expression
Predefined Symbols
Natural phrase from original text
Scope
Typically domain-specific (e.g., medical, legal, financial).
Domain-independent; works on any text.
Input Type
Structured or semi-structured text, sometimes annotated data.
Raw unstructured text (e.g., news articles, blogs).
Extraction
Focuses on extracting specific entities or relations using rules.
Extracts general subject-relation-object triples without prior knowledge.
Supervision
Often supervised (requiring labeled data for training).
Primarily unsupervised, does not need labeled training data.
Flexibility
Less flexible; works best with a fixed domain and predefined structure.
Highly flexible; can be applied to a variety of domains.

OIE Methods: Rule Based Method


原 PPT 独立图片资源
OIE Methods: Rule Based Method

原 PPT 独立图片资源
https://nlp.stanford.edu/software/openie.html

OIE Methods: Neural Based Method


原 PPT 独立图片资源
OIE Methods: Neural Based Method

https://arxiv.org/pdf/2009.08128
原 PPT 独立图片资源

Challenges


原 PPT 独立图片资源
Challenges

Different Definition of Facts



Cannot express Implicit Relations
原 PPT 独立图片资源
https://aclanthology.org/2022.acl-long.430.pdf
𝐿𝑖𝑠𝑎 𝑐𝑎𝑙𝑙𝑒𝑑 𝐵𝑜𝑏 𝑑𝑎𝑑.  → IE → < 𝐿𝑖𝑠𝑎, 𝐷𝑎𝑢𝑔ℎ𝑡𝑒𝑟_𝑂𝑓, 𝐵𝑜𝑏 >
𝐿𝑖𝑠𝑎 𝑐𝑎𝑙𝑙𝑒𝑑 𝐵𝑜𝑏 𝑑𝑎𝑑.  → OIE → < 𝐿𝑖𝑠𝑎, 𝑐𝑎𝑙𝑙𝑒𝑑, 𝐵𝑜𝑏, 𝑑𝑎𝑑>

OIE@OIA: Parsing Based OIE


原 PPT 独立图片资源
OIE@OIA: Parsing Based OIE

原 PPT 独立图片资源
https://aclanthology.org/2022.acl-long.430.pdf
原 PPT 独立图片资源

OIE@OIA: Comparison


原 PPT 独立图片资源
OIE@OIA: Comparison

原 PPT 独立图片资源

OIE@OIA: Results


原 PPT 独立图片资源
OIE@OIA: Results

原 PPT 独立图片资源

Lingua Graph


原 PPT 独立图片资源
Lingua Graph

原 PPT 独立图片资源

Lingua Graph


原 PPT 独立图片资源
Lingua Graph

原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.12 Taxonomy and Ontology Learning



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.12 Taxonomy and Ontology Learning

原 PPT 独立图片资源

Steps of Learning the Taxonomy/Ontology


原 PPT 独立图片资源
Steps of Learning the Taxonomy/Ontology
General steps of the domain ontology learning process.  
Learning Medical Ontologies from the Web, January 2007

Taxonomy Discovery: Probase


原 PPT 独立图片资源
Taxonomy Discovery: Probase
Noun Phrases are taken as entities;
Typing is discovered by patterns like “such as”.
Wu et,al - Probase A Probabilistic Taxonomy for Text Understanding
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Ontology Learning


原 PPT 独立图片资源
Ontology Learning
Starting with triplets extracted by Open IE: {<𝐸𝑀𝑖,𝑅𝑀𝑘,𝐸𝑀𝑗>};
Map the Entity Mentions to their Types: {<𝑇𝑖,𝑅𝑀𝑘,𝑇𝑗>};
Find the significant Relation Mentions between Types.
原 PPT 独立图片资源

Ontology Learning


原 PPT 独立图片资源
Ontology Learning

原 PPT 独立图片资源
原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.13 Entity Linking



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.13 Entity Linking

原 PPT 独立图片资源

Entity Linking Problem


原 PPT 独立图片资源
Entity Linking Problem
原 PPT 独立图片资源
Aidan Hogan, et.al, Knowledge Graphs, 2021
When querying or completing a Knowledge Graph with queries/texts;
When a name of an entity is mentioned, we should know the exact entity it refers to.
Otherwise, the entity information will be mistakenly mixed.

Entity Linking System


原 PPT 独立图片资源
Entity Linking System
https://medium.com/nerd-for-tech/building-bi-encoder-based-entity-linking-system-with-transformer-6c111d86500
Based on Representation Learning;
Match the Entity Mention Representation to Candidate Entity Representation
原 PPT 独立图片资源

The Mention - Entity Matching Model


原 PPT 独立图片资源
The Mention - Entity Matching Model

https://medium.com/nerd-for-tech/building-bi-encoder-based-entity-linking-system-with-transformer-6c111d86500
原 PPT 独立图片资源

Challenges


原 PPT 独立图片资源
Challenges
Similarity != Identity: Limited Resolution of Embedding;
Limited Context Information;
Incomplete Knowledge Graph: Empty Linking.
原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.14 Knowledge from Other Sources



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.14 Knowledge from Other Sources

原 PPT 独立图片资源

Tables


原 PPT 独立图片资源
Tables

原 PPT 独立图片资源
原 PPT 独立图片资源

Challenges


原 PPT 独立图片资源
Challenges

The table information can be complete only if we consider the title of the table and surrounding texts.
It would be better to use Table2Text to convert it into text chunk and then extract information from the text chunk.

Other KGs


原 PPT 独立图片资源
Other KGs

原 PPT 独立图片资源
https://arxiv.org/pdf/2307.11772
If another KG is available, you can merge it’s knowledge into your KG by align the entities and the relations between these two KGs.

Distant Supervision


原 PPT 独立图片资源
Distant Supervision

Or you can make training samples using another KG through distant supervision and train your own knowledge acquisition models.
Applsci 12 08821 g001 550
https://www.mdpi.com/2076-3417/12/17/8821

LLMs


原 PPT 独立图片资源
LLMs

LLMs have exhibited strong ability in:
Extracting entities/relations/facts from texts;
Directly answer factoid questions.
So it is a strong tools for the initial knowledge graph construction.

1.2 Symbolic Representation of Knowledge 1.2.15 Knowledge Graph Refinement



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.15 Knowledge Graph Refinement

原 PPT 独立图片资源

Background


原 PPT 独立图片资源
Background
After Knowledge Graph Construction, we obtained a KG with knowledge extracted from multiple external resources.
But there are mistakes, conflicts, missing facts in KG;
Refinements is vital to eliminate the mistakes, solve conflicts and completing the missing facts.
These are very difficult tasks.

Methods


原 PPT 独立图片资源
Methods
Logical refinements:
Confliction Detection
Rules-based Constraints

Statistical refinements:
Neural Representation Learning
Statistical Relational Learning

1.2 Symbolic Representation of Knowledge 1.2.16 Knowledge Graph Construction: Case Study



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.16 Knowledge Graph Construction: Case Study

原 PPT 独立图片资源

DBpedia


原 PPT 独立图片资源
DBpedia
DBpedia is a collaborative project that extracts structured information from Wikipedia, initiated in 2007 by several German Universities.
原 PPT 独立图片资源
https://en.wikipedia.org/wiki/DBpedia

DBpedia


原 PPT 独立图片资源
DBpedia
DBpedia “Largest Diamond” Graph:
~2.2 billion RDF triples and over 220 million entities from DBpedia, Geonames, DNB, MusicBrainz, and more.

External Links:
Over 130 million links to 179 external datasets (e.g., Wikidata, VIAF, YAGO), enabling linked data integration.
Entity Type
Count
Persons
~1,792,308
Places
~748,372
Creative Works
~610,589
Organizations
~345,523
Species
~1,933,436
Diseases
~10,591

DBpedia


原 PPT 独立图片资源
DBpedia
If one were interested in the Japanese shōjo manga series Tokyo Mew Mew:
原 PPT 独立图片资源
https://en.wikipedia.org/wiki/DBpedia

DBpedia Construction


原 PPT 独立图片资源
DBpedia Construction

原 PPT 独立图片资源
https://svn.aksw.org/papers/2013/SWJ_DBpedia/public.pdf

1.2 Symbolic Representation of Knowledge 1.2.17 Suggested Readings



原 PPT 独立图片资源

1.2 Symbolic Representation of Knowledge 1.2.17 Suggested Readings

原 PPT 独立图片资源

Books


原 PPT 独立图片资源
Books
Aidan Hogan, et.al, Knowledge Graphs, SYNTHESIS LECTURES ON DATA, SEMANTICS, AND KNOWLEDGE, 2022
Mayank Kejriwal, et.al, Knowledge Graphs Fundamentals, Techniques, and Applications, the MIT Press, 2021

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源

Natural Language UnderstandingStatistical Representation of Language and Knowledge


原 PPT 独立图片资源
Natural Language
Understanding

Statistical Representation
of Language and Knowledge
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Why Statistical Representation
Vector Semantics for Word and Document
Topic Modeling
Neural Representations for Language
Neural Representations for Knowledge

1.3 Statistical Representation of Language and Knowledge 1.3.1 Why Statistical Representation



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.1 Why Statistical Representation

原 PPT 独立图片资源

Statistical Nature: Evolutionary Perspective


原 PPT 独立图片资源
Statistical Nature: Evolutionary Perspective
As Natural Outcome of Language Emergence
Language is invented for maximizing information transition efficiency and reducing ambiguity.
So according to information theory, the human language naturally forms a certain distribution:
Zipf’s Law: The frequency of word usage in language follows a power-law distribution (a few words are used very frequently, while many words are used rarely), indicating a statistical regularity.
Menzerath’s Law: Longer linguistic units (e.g., sentences) tend to be composed of shorter components (e.g., words), which is also a statistical phenomenon.

Statistical Nature: Cognitive Perspective


原 PPT 独立图片资源
Statistical Nature: Cognitive Perspective
Cognitive Perspective: Language Learning and Statistical Patterns
Child Language Acquisition: Studies show that infants use statistical learning abilities to identify phonetic patterns (such as word boundaries), indicating a strong link between language acquisition and statistical mechanisms.
Neuroscience: The brain utilizes probabilistic inference to predict the next word or sentence when processing language, a mechanism similar to modern neural language models (e.g., GPT).

Statistical Nature


原 PPT 独立图片资源
Statistical Nature
The statistical nature of language is not an innate, a prior chacteristic but a natural property that gradually emerges during language evolution.
This statistical nature likely results from a combination of cognitive constraints, information transmission optimization, and biological evolution.

Statistical Representation vs Symbolic Representation


原 PPT 独立图片资源
Statistical Representation vs Symbolic Representation
Symbolic Representation: Map Surface Form into Inner Symbolic Form with connection to real-world entities, facts, logics.
Statistical Representation: Find statistical connection between elements in the Surface Form, and connection to real-world may emerge.

1.3 Statistical Representation of Language and Knowledge 1.3.2 Vector Semantics for Words



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.2 Vector Semantics for Words

原 PPT 独立图片资源

What is Word: Recall the Structuralism


原 PPT 独立图片资源
What is Word: Recall the Structuralism

A sign is the basic unit of meaning in language. It consists of:
Signifier (signifiant): The sound pattern or written form (e.g., the word “tree”).
Signified (signifié): The concept or meaning associated with the signifier (the mental image of a tree).
In N-Gram model, word is just surface strings (Signifier), and we do not directly care about its meaning (Signified).
We also talked about the logic representation of language, where the meaning of words is entity or relations.
Lets consider the meaning of word (Signified) more thoroughly.

Lemmas and Senses


原 PPT 独立图片资源
Lemmas and Senses

Lemma is Signifier, and Sense can be considered some kind of Signified.
sense
mouse (N)
1. any of numerous small rodents...
2. a hand-operated device that controls a cursor...
lemma
Modified from the online thesaurus WordNet

Relations: Synonymy


原 PPT 独立图片资源
Relations: Synonymy

Synonyms have the same meaning in some or all contexts.
filbert / hazelnut
couch / sofa
big / large
automobile / car
vomit / throw up
water / H20

Relations: No Perfect Synonymy


原 PPT 独立图片资源
Relations: No Perfect Synonymy

Note that there are probably no examples of perfect synonymy.
Even if many aspects of meaning are identical
Still may differ based on politeness, slang, register, genre, etc.
Examples:
water/H20: "H20" in a surfing guide?
big/large: my big sister != my large sister

The Linguistic Principle of Contrast


原 PPT 独立图片资源
The Linguistic Principle of Contrast

Difference in form → difference in meaning
原 PPT 独立图片资源
I do not believe that there is a synonymous word in any language
原 PPT 独立图片资源

Relations: Similarity


原 PPT 独立图片资源
Relations: Similarity

Not synonyms, but sharing some element of meaning
word1
word2
similarity
vanish
disappear
9.8
behave
obey
7.3
belief
impression
5.95
muscle
bone
3.65
modest
flexible
0.98
hole
agreement
0.3
SimLex-999 dataset (Hill et al., 2015)

Relations: Antonymy


原 PPT 独立图片资源
Relations: Antonymy

Senses that are opposites with respect to only one specific semantic feature, while they remain similar in other aspects.
For example:
hot vs. cold → Opposites in temperature, but both describe a degree of heat.
buy vs. sell → Opposites in perspective (buyer vs. seller), but both refer to the same transaction.
teacher vs. student → Opposites in role, but both exist within the context of education.

Relations: Relatedness


原 PPT 独立图片资源
Relations: Relatedness

Words can be related in any way:
<coffee, tea>: similar; <coffee, cup>: related.
Semantic Field: Words that
cover a particular semantic domain
bear structured relations with each other.
Hospitals
surgeon, scalpel, nurse, hospital
Restaurants
waiter, menu, plate, food
Semantic Frame: Words that:
denote perspectives or participants in a particular type of event.
Buy :
buy, sell, goods
Travel
traveler, source, destination

Relations: Connotation(Sentiment)


原 PPT 独立图片资源
Relations: Connotation(Sentiment)

Words have affective meanings
Positive connotations (happy, great, love)
Negative connotations (sad, terrible, hate)
Connotations can be subtle:
Positive connotation: copy, replica, reproduction
Negative connotation: fake, knockoff, forgery

Relations: Connotation(Sentiment)


原 PPT 独立图片资源
Relations: Connotation(Sentiment)

Words seem to vary along 3 affective dimensions(Osgood et al. (1957)):
valence: the pleasantness of the stimulus
arousal: the intensity of emotion provoked by the stimulus
dominance: the degree of control exerted by the stimulus
原 PPT 独立图片资源

Summary: Senses and Relations


原 PPT 独立图片资源
Summary: Senses and Relations

Senses (symbolic Signified)
Have a complex many-to-many association with words (Signifier)
Have relations with each other
Synonymy
Antonymy
Similarity
Relatedness
Connotation
The relation cannot be obtained from the sense through calculation.

The meaning of a word is its use in the language (Ludwig Wittgenstein)


原 PPT 独立图片资源
The meaning of a word is its use in the language (Ludwig Wittgenstein)

Words are defined by their environments (the words around them)

If A and B have almost identical environments we say that they are synonyms. -- Zellig Harris (1954)

What does ongchoi mean?


原 PPT 独立图片资源
What does ongchoi mean?

Suppose you see these sentences:
Ong choi is delicious sautéed with garlic.
Ong choi is superb over rice
Ong choi leaves with salty sauces
And you've also seen these:
…spinach sautéed with garlic over rice
Chard stems and leaves are delicious
Collard greens and other salty leafy greens
Conclusion:
Ongchoi is a leafy green like spinach, chard, or collard greens
Based on words like "leaves" and "delicious" and "sauteed"

ongchoi


原 PPT 独立图片资源
ongchoi

原 PPT 独立图片资源
Yamaguchi, Wikimedia Commons, public domain
空心菜
kangkong
rau muống
…

Basic Ideas


原 PPT 独立图片资源
Basic Ideas

Idea 1: Defining meaning by linguistic distribution
Let's define the meaning of a word by its distribution in language use, meaning its neighboring words or grammatical environments.
Idea 2: Meaning as a point in multidimensional space
原 PPT 独立图片资源

Defining meaning as a point in space based on distribution


原 PPT 独立图片资源
Defining meaning as a point in space based on distribution

Each word = a vector (embedding) (not just "good" or "w45")
Similar words are "nearby in semantic space"
We build this space automatically by seeing which words are nearby in text
原 PPT 独立图片资源

Vector Representation vs Symbolic Representation


原 PPT 独立图片资源
Vector Representation vs Symbolic Representation


With words, a feature is a word identity
Feature 5: 'The previous word was "terrible"'
requires exact same word to be in training and test
With embeddings:
Feature is a word vector
'The previous word was vector [35,22,17…]
Now in the test set we might see a similar vector [34,21,14]
We can generalize to similar but unseen words!!!

Two kinds of embeddings


原 PPT 独立图片资源
Two kinds of embeddings

Sparse Embedding
Representation calculated by explicit statistics of environments;
Dimension has explicit meaning;
Dense Embedding
Representation obtained by neural learning from environments;
Dimension generally has no explicit meaning.

From now on: Computing with meaning representationsinstead of string representations


原 PPT 独立图片资源
From now on: Computing with meaning representations
instead of string representations

原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.3 Sparse Embeddings


原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.3 Sparse Embeddings


The environment of words are documents;
Vector or distributional models of meaning are generally based on how words co-occur in documents;
We’ll look at two model methods: the word-document matrix and the word-word matrix.

Term-Document Matrix


原 PPT 独立图片资源
Term-Document Matrix

Each document is represented by a vector of words



Each word can be represented by a vector of documents
原 PPT 独立图片资源
原 PPT 独立图片资源

Visualizing document vectors


原 PPT 独立图片资源
Visualizing document vectors

原 PPT 独立图片资源

Word-word Matrix


原 PPT 独立图片资源
Word-word Matrix

The count of word A co-occur with word B in a context range:
The whole document;
The limited window;
原 PPT 独立图片资源
原 PPT 独立图片资源

Visualizing word-word vectors


原 PPT 独立图片资源
Visualizing word-word vectors

原 PPT 独立图片资源

Raw frequency is a bad representation


原 PPT 独立图片资源
Raw frequency is a bad representation

The co-occur frequency is the source of valuable information;
But overly frequent words like the, it, or they are not very informative about the context
It's a paradox! How can we balance these two conflicting constraints?

TF-IDF for Word-Document Representation


原 PPT 独立图片资源
TF-IDF for Word-Document Representation

TF: Frequence of Term t in Document d

IDF: Inverse Document Frequence of Term t in Corpus
measure the importance of a word in the entire document collection D

TF-IDF:
If a word appears frequently in a document (high TF) but is relatively rare in the entire corpus (high IDF), then its importance in that document is greater.
Conversely, if a word is common across all documents (such as “the” or “is”), its IDF is low, resulting in a lower TF-IDF value.
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

PMI for Word-Word Representation


原 PPT 独立图片资源
PMI for Word-Word Representation

PMI: (Pointwise mutual information)
It is a measure of how often two events x and y occur, compared with what we would expect if they were independent:

𝑃𝑀𝐼(𝑤1,𝑤2)=𝑙𝑜𝑔𝑝(𝑤1,𝑤2)𝑝(𝑤1)𝑝(𝑤2)

1.3 Statistical Representation of Language and Knowledge 1.3.4 Vector Semantics for Documents



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.4 Vector Semantics for Documents

原 PPT 独立图片资源

Let’s focus on Document


原 PPT 独立图片资源
Let’s focus on Document

Could be a play or a Wikipedia article
But documents can be anything; even a paragraph can be considered a document.
However, we generally consider document to have complete and explicit meaning.

Bag of Words


原 PPT 独立图片资源
Bag of Words

Recall the Word-Document Matrix:




Bag of Words (BOW): each document is represented by a vector of size |𝑉| with element of count/frequency.
原 PPT 独立图片资源

TF-IDF Model


原 PPT 独立图片资源
TF-IDF Model

TF-IDF: each document is represented by a vector of size |𝑉| with element of tf-idf value.
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Applications of Document Embedding


原 PPT 独立图片资源
Applications of Document Embedding

Text Classification
Used to train machine learning models (e.g., Naïve Bayes, SVM) for spam detection, sentiment analysis, and topic categorization.
Example: Detecting whether an email is spam based on the presence of certain words.
Text Clustering
Helps group similar documents together (e.g., k-means clustering).
Example: Categorizing news articles into topics like politics, sports, and entertainment.
Search Engine Ranking
Used to prioritize documents based on how uniquely relevant a term is.
Example: Google ranking a webpage higher if it contains unique words related to the search query.
Recommendation Systems
Measures similarity between documents using cosine similarity on TF-IDF vectors.
Example: Suggesting similar news articles based on the words in a given article.

Go deeper into Document Representation


原 PPT 独立图片资源
Go deeper into Document Representation

One most fundamental attribute is Topics.
Recall the Synonymy and Semantic Field of word relatedness, words may be different but talking about the same topic;
Topic Similarity is an essential similarity of documents.
But BOW and TF-IDF do not represent the topic information.
How to model it?

Topics and Words


原 PPT 独立图片资源
Topics and Words

Topic Modeling with LSA, PLSA, LDA & lda2Vec
https://nanonets.com/blog/topic-modeling-with-lsa-plsa-lda-lda2vec/
Words are related by Semantic Field

Topic Modeling


原 PPT 独立图片资源
Topic Modeling

Key assumption:
each document consists of a mixture of topics, and
each topic consists of a collection of words.
The semantics of the document are governed by some “topics”;
We cannot observing the topics directly; It is hidden or “latent”;
The goal of topic modeling is to uncover these latent variables — topics — that shape the meaning of our document and corpus;

Latent Semantic Analysis (LSA)


原 PPT 独立图片资源
Latent Semantic Analysis (LSA)

What we have: Documents and Terms (Words) Matrix (TF-IDF)
The matrix is very sparse, very noisy, and very redundant across its many dimensions;
Topics:
are hidden inner intermediate structures between the documents and words;
are with limited number, generally much smaller than that of doc and words.
How to discover the structures from the documents-words matrix?
Matrix Decomposition!

Latent Semantic Analysis (LSA)


原 PPT 独立图片资源
Latent Semantic Analysis (LSA)


原 PPT 独立图片资源
原 PPT 独立图片资源
U ∈ ℝ^(m ⨉ t) emerges as our document-topic matrix,
V ∈ ℝ^(n ⨉ t) becomes our word-topic matrix.
S ∈ ℝ^(t ⨉ t) represents importance of a latent topic.
https://nanonets.com/blog/topic-modeling-with-lsa-plsa-lda-lda2vec/

Application of Topic related Embeddings


原 PPT 独立图片资源
Application of Topic related Embeddings


With these document vectors and word vectors, we can now easily apply measures such as cosine similarity to evaluate:
the similarity of different documents
the similarity of different words
the similarity of terms (or “queries”) and documents (which becomes useful in information retrieval, when we want to retrieve passages most relevant to our search query).

Limitation of LSA


原 PPT 独立图片资源
Limitation of LSA


Computationally Expensive
need for really large set of documents and vocabulary to get accurate results, where SVD is expensive for large-scale datasets;
Sensitivity to Noise and Parameter Selection
The choice of k (number of singular values retained) significantly impacts performance.
Lacks of Interpretation
We don’t know what the topics are (no modeling about topics).

Statistical Topic Modeling


原 PPT 独立图片资源
Statistical Topic Modeling


Statistical View Point:
Document as a mixture of topics
Topic as a distribution over words;
Generative model for documents:
document features as being generated by sampling from the conditional probability of topic and words.
The goal of statistical topic modeling is to recover the distribution that best fit our observations -- document-term matrix.

Statistical Topic Modeling


原 PPT 独立图片资源
Statistical Topic Modeling


Training on (large) corpus to learn:
Per-topic word distributions
Per-document topic distributions
原 PPT 独立图片资源

PLSA: Document Generation Procedure


原 PPT 独立图片资源
PLSA: Document Generation Procedure

Document Generation Procedure:
Choose a document d from the document collection.
Select a topic z according to 𝑃(𝑧 | 𝑑) (each doc has a topic distribution, the parameter of PLSA).
Choose a word w according to 𝑃(𝑤 | 𝑧) (each topic has a word distribution, the parameter of PLSA).
Repeat steps 2 and 3 for each word in the document.

PLSA: Document, Topic, Words


原 PPT 独立图片资源
PLSA: Document, Topic, Words

原 PPT 独立图片资源
https://www.machinelearningplus.com/nlp/topic-modeling-visualization-how-to-present-results-lda-models/

PLSA


原 PPT 独立图片资源
PLSA


原 PPT 独立图片资源
https://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/AV1011/oneata.pdf

PLSA


原 PPT 独立图片资源
PLSA


We want the model P(D,W) such that for any document d and word w, P(d,w) corresponds to that entry in the document-term matrix.
Probabilistic Latent Semantic Analysis

PLSA Training


原 PPT 独立图片资源
PLSA Training

Joint probability model:


Likelihood
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

PLSA Training


原 PPT 独立图片资源
PLSA Training


Training with EM:
Initialization of the per-topic word distributions and per-document topic distributions
E-step:



M-step:
Schermata 2014-10-29 a 14.07.23.png
Schermata 2014-10-29 a 14.08.21.png

PLSA and LSA


原 PPT 独立图片资源
PLSA and LSA


原 PPT 独立图片资源
https://nanonets.com/blog/topic-modeling-with-lsa-plsa-lda-lda2vec/

Limitations of PLSA


原 PPT 独立图片资源
Limitations of PLSA


No Generative Capability:
PLSA does not model how new, unseen documents are generated—it just assigns topics to existing documents.
Prone to Overfitting:
The number of parameters for PLSA is 𝑁 ∗ 𝐾 + 𝐾 ∗ 𝑉;
Grows linearly with the number of documents.
Rarely used in practice.

Prior Distributions


原 PPT 独立图片资源
Prior Distributions


If we have priors for the document-topic and word-topic distributions;
We can:
generate new document by sampling from the prior document-topic distribution;
reduced the number of parameter of the model to 𝐾 + 𝑉;
Lending itself to better generalization.
The document-topic and word-topic distribution is multi-nominal;
What is the distribution of multi-nominal distribution?

Dirichlet Distribution


原 PPT 独立图片资源
Dirichlet Distribution


The Dirichlet distribution is a probability distribution over the space of multinomial distributions:
to generate data X from a Dirichlet distribution with parameters 𝛼;
you first draw 𝜃∼𝐷𝑖𝑟(𝛼).
and then draw 𝑋∼𝑀𝑢𝑙𝑡𝑖(𝜃).
原 PPT 独立图片资源

Explanation


原 PPT 独立图片资源
Explanation


The overall distribution shape depends on the relative magnitudes of all 𝛼𝑖 ’s.
Relative magnitude determines the mean proportions: 𝛼𝑖/𝛼0. A larger 𝛼𝑖 implies a higher expected proportion.
Absolute magnitude determines the degree of uncertainty: a larger 𝛼0 leads to a more concentrated distribution, while a smaller 𝛼0 leads to a more dispersed or sparse distribution
If all 𝛼𝑖 are small (<1) → The distribution is sparse, meaning most categories have near-zero probabilities, and only a few dominate.
If all 𝛼𝑖 are large (>1) → The distribution is dense, meaning all categories tend to have moderate values.
If 𝛼𝑖 values are imbalanced → Some categories (with larger values) are more dominant than others.

Example of Dirichlet Distribution


原 PPT 独立图片资源
Example of Dirichlet Distribution


File:Dirichlet.pdf
https://en.wikipedia.org/wiki/Dirichlet_distribution
undefined

Latent Dirichlet Allocation


原 PPT 独立图片资源
Latent Dirichlet Allocation


https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf
原 PPT 独立图片资源

Latent Dirichlet Allocation


原 PPT 独立图片资源
Latent Dirichlet Allocation

Schermata 2014-10-02 a 19.37.22.png
For each doc in a collection of N docs
For each word position in a doc of length M
Topic assignment to a word at position i in doc dj
Word token at position i in doc dj
Per-document topic distribution
Schermata 2014-10-29 a 14.49.28.png
Schermata 2014-10-29 a 14.49.52.png
Per-topic word distribution
[Moens and Vulic, Tutorial @WSDM 2014]

Latent Dirichlet Allocation


原 PPT 独立图片资源
Latent Dirichlet Allocation


原 PPT 独立图片资源

Parameter Estimation


原 PPT 独立图片资源
Parameter Estimation


Given a corpus of documents 𝐷={𝒘1,⋯, 𝒘𝑀}, we are to find parameter 𝛼 and 𝜂 that maximize the log likelihood:

𝐿(𝛼,𝜂)=∑log⁡𝑝(𝒘𝑑|𝛼, 𝜂)

Can be solved by variational EM algorithm.

Inference


原 PPT 独立图片资源
Inference


Given a document 𝐰, we are to find the 𝜃 and 𝒛.
𝑝(𝜃,𝒛|𝒘,𝛼,𝜂)= 𝑝(𝜃,𝒛,𝒘|𝛼, 𝜂)𝑝(𝒘|𝛼, 𝜂)
Variational EM
Gibbs sampling

Case Analysis: LDA on News Data


原 PPT 独立图片资源
Case Analysis: LDA on News Data

原 PPT 独立图片资源
https://www.machinelearningplus.com/nlp/topic-modeling-visualization-how-to-present-results-lda-models/

Case Analysis: learned LDA Topics by Gensim


原 PPT 独立图片资源
Case Analysis: learned LDA Topics by Gensim

https://www.machinelearningplus.com/nlp/topic-modeling-visualization-how-to-present-results-lda-models/
0, 0.017*"write" + 0.015*"people" + 0.014*"organization" + 0.014*"article" +
#> 0.013*"time" + 0.008*"give" + 0.008*"first" + 0.007*"tell" + 0.007*"new" +
#> 0.007*"question
1, 0.008*"christian" + 0.008*"believe" + 0.007*"god" + 0.007*"law" +
#> 0.006*"state" + 0.006*"israel" + 0.006*"israeli" + 0.005*"exist" +
#> 0.005*"way" + 0.004*"bible"
2, 0.024*"armenian" + 0.012*"bike" + 0.006*"kill" + 0.006*"work" +
#> 0.005*"well" + 0.005*"year" + 0.005*"sumgait" + 0.005*"soldier" +
#> 0.004*"way" + 0.004*"ride"
3 0.019*"team" + 0.019*"game" + 0.013*"hockey" + 0.010*"player" +
#> 0.009*"play" + 0.009*"win" + 0.009*"nhl" + 0.009*"year" + 0.009*"hawk" +
#> 0.009*"season"

Case Analysis: Top-N words cloud in Each Topic


原 PPT 独立图片资源
Case Analysis: Top-N words cloud in Each Topic


原 PPT 独立图片资源

Case Analysis: Dominant topic and its percentage contribution in each document


原 PPT 独立图片资源
Case Analysis: Dominant topic and its percentage contribution in each document


原 PPT 独立图片资源

Case Analysis: Dominant topic and its percentage contribution in each document


原 PPT 独立图片资源
Case Analysis: Dominant topic and its percentage contribution in each document


原 PPT 独立图片资源

Case Analysis: t-SNE Visualization of Documents


原 PPT 独立图片资源
Case Analysis: t-SNE Visualization of Documents


原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.5 Neural Embeddings



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.5 Neural Embeddings

原 PPT 独立图片资源

Relax the Constraint of Parametric Model


原 PPT 独立图片资源
Relax the Constraint of Parametric Model


Parametric Models
A parametric model assumes a fixed number of parameters and a specific functional form (structure). The learning process involves estimating these parameters based on the given data.
Sparse Embeddings are Parametric Models:
TF-IDF, PMI, PLSA, LDA
Focus on objective, relax the capacity of models, let the data drive the direction of the model.

Learning Representation


原 PPT 独立图片资源
Learning Representation


Recall: The meaning is determined by its environment, that is, neighborhood;
Words cooccur means their embeddings has certain relations.
How about predicting the neighborhood from the embedding?
Predicting each other: Word2Vec
Predicting the co-occurrence: GloVe
Word embeddings = the weights of learned machines.
Big idea: self-supervision by co-occurrence information without human label;

1.3 Statistical Representation of Language and Knowledge 1.3.6 Word2Vec



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.6 Word2Vec

原 PPT 独立图片资源

Word2Vec Overview


原 PPT 独立图片资源
Word2Vec Overview


We have a large corpus (“body”) of text: a long list of words
Every word in a fixed vocabulary is represented by a vector
Go through each position t in the text, which has a center word c and context (“outside”) words o
Use the similarity of the word vectors for c and o to calculate the probability of o given c (or vice versa)
Keep adjusting the word vectors to maximize this probability
原 PPT 独立图片资源
(Mikolov et al. 2013)

Skip-Gram: The context window


原 PPT 独立图片资源
Skip-Gram: The context window


Assume a +/- 2 word window, given training sentence:
…lemon, a [tablespoon of apricot jam, a] pinch…
c1 c2 [target] c3 c4


Skip-Gram: Positive Samples


原 PPT 独立图片资源
Skip-Gram: Positive Samples


Assume a +/- 2 word window, given training sentence:
…lemon, a [tablespoon of apricot jam, a] pinch…
c1 c2 [target] c3 c4


原 PPT 独立图片资源

Skip-Gram: Negative Samples


原 PPT 独立图片资源
Skip-Gram: Negative Samples


Assume a +/- 2 word window, given training sentence:
…lemon, a [tablespoon of apricot jam, a] pinch…
c1 c2 [target] c3 c4


原 PPT 独立图片资源
For each positive example we'll grab k negative examples, sampling by frequency

Skip-Gram: Negative Samples


原 PPT 独立图片资源
Skip-Gram: Negative Samples


Assume a +/- 2 word window, given training sentence:
…lemon, a [tablespoon of apricot jam, a] pinch…
c1 c2 [target] c3 c4


原 PPT 独立图片资源
原 PPT 独立图片资源

Skip-Gram: Probabilities


原 PPT 独立图片资源
Skip-Gram: Probabilities


Assume a +/- 2 word window, given training sentence:



Given a candidate (word, context) pair :
(apricot, jam)
(apricot, aardvark)
…
We want to compute the probability using their embeddings
𝑃(+|𝑤, 𝑐) 
𝑃(−|𝑤, 𝑐) = 1 − 𝑃(+|𝑤, 𝑐) 
…lemon, a [tablespoon of apricot jam, a] pinch…
c1 c2 [target] c3 c4


How to Compute 𝑃(+|𝑤, 𝑐)


原 PPT 独立图片资源
How to Compute 𝑃(+|𝑤, 𝑐)


The probability of prediction should be definitely related to relatedness or similarity.
Two vectors are similar if they have a high dot product
Cosine is just a normalized dot product
So 𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝑤,𝑐)  ∝ 𝑤 ∙ 𝑐
𝑃(+|𝑤, 𝑐) can be obtained by normalize it to a probability.

For one context word: 𝑃(+|𝑤, 𝑐)


原 PPT 独立图片资源
For one context word: 𝑃(+|𝑤, 𝑐)


原 PPT 独立图片资源
原 PPT 独立图片资源

For the Full Context: 𝑃(+|𝑤, 𝑐1:𝐿)


原 PPT 独立图片资源
For the Full Context: 𝑃(+|𝑤, 𝑐[1:𝐿])  


We have lots of context words.
We'll assume independence and just multiply them:
原 PPT 独立图片资源

The Parameters – Embeddings


原 PPT 独立图片资源
The Parameters -- Embeddings


原 PPT 独立图片资源

Learning Procedure


原 PPT 独立图片资源
Learning Procedure


Given the set of positive and negative training instances, and an initial set of embedding vectors
The goal of learning is to adjust those word vectors such that we:
Maximize the similarity of the target word, context word pairs (𝑤 , 𝑐𝑝𝑜𝑠) drawn from the positive data
Minimize the similarity of the (𝑤 , 𝑐𝑛𝑒𝑔) pairs drawn from the negative data.

Loss Function


原 PPT 独立图片资源
Loss Function


Maximize the similarity of the target with the actual context words, and minimize the similarity of the target with the k negative sampled non-neighbor words.
原 PPT 独立图片资源

Optimization: Gradient Descent


原 PPT 独立图片资源
Optimization: Gradient Descent


原 PPT 独立图片资源

Gradients


原 PPT 独立图片资源
Gradients


原 PPT 独立图片资源
Start with randomly initialized C and W matrices, then incrementally do updates

Two representations?


原 PPT 独立图片资源
Two representations?


Skip-Gram with Negative Sampling learns two sets of embeddings
Target embeddings matrix W
Context embedding matrix C
It's common to just add them together, representing word i as the vector 𝑤𝑖 + 𝑐𝑖

Learning Procedure Summary


原 PPT 独立图片资源
Learning Procedure Summary


Start with random d-dimensional vectors as initial embeddings
Train a classifier based on embedding similarity
Take a corpus and take pairs of words that co-occur as positive examples
Take pairs of words that don't co-occur as negative examples
Train the classifier to distinguish these by slowly adjusting all the embeddings to improve the classifier performance
Throw away the classifier code and keep the embeddings.

Trained Word2Vec Embeddings


原 PPT 独立图片资源
Trained Word2Vec Embeddings


Google Word2Vec: includes pre-trained vectors trained on a portion of the Google News dataset (consisting of about 100 billion words). The model contains 300-dimensional vectors for about 3 million words and phrases. 

1.3 Statistical Representation of Language and Knowledge 1.3.7 GloVe



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.7 GloVe

原 PPT 独立图片资源

Glove: Basic Idea


原 PPT 独立图片资源
Glove: Basic Idea


Word co-occurrence is the most important statistical information available for the model to ‘learn’. the word representation.
Glove: Use a Log-Bilinear (LBL) Regression Model to approximate the co-occurrence information:


the name of “Global”: Focus on the global co-occurrence information over the entire corpus.
原 PPT 独立图片资源

Why the Log-Bilinear Form?


原 PPT 独立图片资源
Why the Log-Bilinear Form?

Let’s check the co-occurrence probability carefully.






Ratio of co-occur probabilities is more informative than the raw prob.
原 PPT 独立图片资源
https://jonathan-hui.medium.com/nlp-word-embedding-glove-5e7f523999f6

Why the Log-Bilinear Form?


原 PPT 独立图片资源
Why the Log-Bilinear Form?


let us consider a function ‘F’ that models the ratio of probabilities relationship.
原 PPT 独立图片资源
原 PPT 独立图片资源

Add Linear Structure


原 PPT 独立图片资源
Add Linear Structure


We want to given some desirable behavior for the embedding vector 
Linearity
原 PPT 独立图片资源

Add Symmetry between Words


原 PPT 独立图片资源
Add Symmetry between Words


F be a homomorphism between the groups (R,+) and (R>0, ×),




Make it symmetry:
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Loss: Weighting the co-occurrence


原 PPT 独立图片资源
Loss: Weighting the co-occurrence


Large amount of rare cooccurrences are noisy and carry less information than the more frequent ones
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
https://nlp.stanford.edu/pubs/glove.pdf

Trained Glove Embeddings


原 PPT 独立图片资源
Trained Glove Embeddings


Wikipedia 2014 + Gigaword 5 (6B tokens, 400K vocab, uncased, 50d, 100d, 200d, & 300d vectors, 822 MB)
Common Crawl (42B tokens, 1.9M vocab, uncased, 300d vectors, 1.75 GB)
Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB)
Twitter (2B tweets, 27B tokens, 1.2M vocab, uncased, 25d, 50d, 100d, & 200d vectors, 1.42 GB)
Available at: https://nlp.stanford.edu/projects/glove/

1.3 Statistical Representation of Language and Knowledge 1.3.8 Embedding Properties



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.8 Embedding Properties

原 PPT 独立图片资源

Relations


原 PPT 独立图片资源
Relations


Small windows (C= +/- 2) : nearest words are syntactically similar words in same taxonomy
Hogwarts nearest neighbors are other fictional schools:
Sunnydale, Evernight, Blandings
Large windows (C= +/- 5) : nearest words are related words in same semantic field
Hogwarts nearest neighbors are Harry Potter world:
Dumbledore, half-blood, Malfoy

Example of Similarities


原 PPT 独立图片资源
Example of Similarities


The neighbor of “Frog”
1.frogs 2. toad 3. litoria 4. leptodactylidae 5.rana 6. lizard 7.eleutherodactylus
原 PPT 独立图片资源
https://nlp.stanford.edu/projects/glove/

Analogical relations


原 PPT 独立图片资源
Analogical relations


The classic parallelogram model of analogical reasoning (Rumelhart and Abrahamson 1973)
To solve: "apple is to tree as grape is to _____"
Add tree – apple to grape to get vine
原 PPT 独立图片资源

Analogical relations


原 PPT 独立图片资源
Analogical relations


The parallelogram method can solve analogies with both sparse and dense embeddings (Turney and Littman 2005, Mikolov et al. 2013b)
𝑘𝑖𝑛𝑔⃑ –𝑚𝑎𝑛⃑ +𝑤𝑜𝑚𝑎𝑛⃑ is close to 𝑞𝑢𝑒𝑒𝑛⃑
𝑃𝑎𝑟𝑖𝑠⃑ –𝐹𝑟𝑎𝑛𝑐𝑒⃑ +𝐼𝑡𝑎𝑙𝑦⃑ is close to 𝑅𝑜𝑚𝑒⃑
For a problem 𝑎:𝑎∗::𝑏:𝑏∗, the parallelogram method is:
原 PPT 独立图片资源

Structure in GloVE Embedding space


原 PPT 独立图片资源
Structure in GloVE Embedding space

原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
https://nlp.stanford.edu/projects/glove/

More about the Parallelogram Method


原 PPT 独立图片资源
More about the Parallelogram Method


Parallelorgram method has been greatly developed and widely applied in the field of Knowledge Graph Embeddings (KGE).
It only seems to work for frequent words, small distances and certain relations (relating countries to capitals, or parts of speech), but not others. (Linzen 2016, Gladkova et al. 2016, Ethayarajh et al. 2019a)
Understanding analogy is an open area of research (Peterson et al. 2020)

Embeddings: a window onto Historical Semantics


原 PPT 独立图片资源
Embeddings: a window onto Historical Semantics


Embeddings on different decades of historical text reveals meanings shift
原 PPT 独立图片资源
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. Proceedings of ACL.

Embeddings reflect cultural bias!


原 PPT 独立图片资源
Embeddings reflect cultural bias!


Bolukbasi, Tolga, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. "Man is to computer programmer as woman is to homemaker? debiasing word embeddings." In NeurIPS, pp. 4349-4357. 2016.
Ask “Paris : France :: Tokyo : x”
x = Japan
Ask “father : doctor :: mother : x”
x = nurse
Ask “man : computer programmer :: woman : x”
x = homemaker
Algorithms that use embeddings as part of e.g., hiring searches for programmers, might lead to bias in hiring

Historical embedding as a tool to study cultural biases


原 PPT 独立图片资源
Historical embedding as a tool to study cultural biases


Garg, N., Schiebinger, L., Jurafsky, D., and Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences 115(16), E3635–E3644.
Compute a gender or ethnic bias for each adjective: e.g., how much closer the adjective is to "woman" synonyms than "man" synonyms, or names of particular ethnicities
Embeddings for competence adjective (smart, wise, brilliant, resourceful, thoughtful, logical) are biased toward men, a bias slowly decreasing 1960-1990
Embeddings for dehumanizing adjectives (barbaric, monstrous, bizarre) were biased toward Asians in the 1930s, bias decreasing over the 20th century.
These match the results of old surveys done in the 1930s

1.3 Statistical Representation of Language and Knowledge 1.3.9 Knowledge Graph Embedding



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.9 Knowledge Graph Embedding

原 PPT 独立图片资源

Recall: Representation Learning


原 PPT 独立图片资源
Recall: Representation Learning
Representation learning:
maps real-world objects into vectors (embeddings) in a vector space;
models their attributes, relationships, and interactions through the mathematical structure of these vectors;
thereby providing a computable and generalizable cognitive foundation for intelligent systems.
Can the elements (like relations) and operations (like reasoning) in KG be represented by embeddings?

Recall: Entity Linking


原 PPT 独立图片资源
Recall: Entity Linking
We’ve learnt the entity embedding from its texture attributes.
原 PPT 独立图片资源
https://medium.com/nerd-for-tech/building-bi-encoder-based-entity-linking-system-with-transformer-6c111d86500

Node Embeddings


原 PPT 独立图片资源
Node Embeddings
The meaning of nodes are defined by there environment.
To find embedding for nodes, just construct environment, and apply the Skip-Gram Algorithm.
DeepWalk

原 PPT 独立图片资源
https://www.researchgate.net/figure/The-framework-of-DeepWalk-and-Node2Vec_fig2_343892685

Knowledge Graph Embeddings


原 PPT 独立图片资源
Knowledge Graph Embeddings
Embeddings for KG elements:
Entity, Relation
How to obtain the embedding from the environment of KG?
What is the environments?
What is the objective of the embedding in this environment? That is, what is the expected relationship between the embedding of these elements?

Knowledge Graph Embeddings


原 PPT 独立图片资源
Knowledge Graph Embeddings
Given a knowledge in KG <𝐸ℎ, 𝑅, 𝐸𝑡>, the entities and relations are logically connected;
Entity’s environments: associated relations and target entities;
Relation’s environments: associated head entities and tail entities.
We are expecting there exists certain relationship between their embeddings 𝒉, 𝒓, 𝒕, that is 𝑓(𝒉, 𝒓, 𝒕)=0.

Knowledge Graph Embedding


原 PPT 独立图片资源
Knowledge Graph Embedding

原 PPT 独立图片资源
https://arxiv.org/pdf/2410.14733

Recall: Analogical relations


原 PPT 独立图片资源
Recall: Analogical relations
In word embedding:
𝑘𝑖𝑛𝑔⃑ –𝑚𝑎𝑛⃑ +𝑤𝑜𝑚𝑎𝑛⃑ is close to 𝑞𝑢𝑒𝑒𝑛⃑
𝑃𝑎𝑟𝑖𝑠⃑ –𝐹𝑟𝑎𝑛𝑐𝑒⃑ +𝐼𝑡𝑎𝑙𝑦⃑ is close to 𝑅𝑜𝑚𝑒⃑
For a problem 𝑎:𝑎∗::𝑏:𝑏∗, the parallelogram method is:


𝒃 − 𝒂: a hidden relation representation?

原 PPT 独立图片资源

Recall: Analogical relations


原 PPT 独立图片资源
Recall: Analogical relations



原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
https://nlp.stanford.edu/projects/glove/

Translation Based Embeddings


原 PPT 独立图片资源
Translation Based Embeddings

原 PPT 独立图片资源
https://arxiv.org/pdf/2210.14905
TransE:
𝑓(𝒉, 𝒓, 𝒕)=||𝒉+𝒓−𝒕||2
For positive facts, 𝑓(𝒉, 𝒓, 𝒕) should be small;
For negative facts, 𝑓(𝒉, 𝒓, 𝒕) should be large;

Training the Embedding Model


原 PPT 独立图片资源
Training the Embedding Model

Use margin-based ranking loss:

For each positive triple (h, r, t), generate corrupted triple (ℎ’, 𝑟, 𝑡) or (ℎ, 𝑟, 𝑡’) by replace:
Head entity: ℎ’∈ ℰ, ℎ’≠ ℎ
Tail entity: 𝑡’∈ℰ, 𝑡’≠ 𝑡
原 PPT 独立图片资源

Model Variants


原 PPT 独立图片资源
Model Variants

原 PPT 独立图片资源
https://arxiv.org/pdf/2410.14733

Model Variants


原 PPT 独立图片资源
Model Variants

https://arxiv.org/pdf/2410.14733
原 PPT 独立图片资源

Why so many variants?


原 PPT 独立图片资源
Why so many variants?

https://arxiv.org/pdf/2410.14733
原 PPT 独立图片资源

Why so many variants?


原 PPT 独立图片资源
Why so many variants?

原 PPT 独立图片资源
MQuadE
|| 𝐻𝑅 − ˆ𝑅𝑇 ||𝐹2

Why so many variants?


原 PPT 独立图片资源
Why so many variants?

MQuinE
∥𝐻𝑅ℎ−𝑅𝑡𝑇+𝐻𝑅𝑐𝑇∥𝐹2
原 PPT 独立图片资源

Applications of KGE


原 PPT 独立图片资源
Applications of KGE

Link Prediction
(?, 𝑟, 𝑡)  or (ℎ, 𝑟, ?)
Triplet Classification
(ℎ, 𝑟, 𝑡) is true or not
Entity Typing
(ℎ, 𝐼𝑠𝐴, ?)
Entity Matching
(𝑥, 𝑠𝑎𝑚𝑒𝐴𝑠, 𝑦)?
Symmetry 13 00485 g001
https://www.mdpi.com/2073-8994/13/3/485

1.3 Statistical Representation of Language and Knowledge 1.3.10 Suggested Readings



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.10 Suggested Readings

原 PPT 独立图片资源

Papers


原 PPT 独立图片资源
Papers
Efficient Estimation of Word Representations in Vector Space https://arxiv.org/abs/1301.3781
GloVe: Global Vectors for Word Representation https://nlp.stanford.edu/pubs/glove.pdf
Enriching Word Vectors with Subword Information https://arxiv.org/abs/1607.04606

1.3 Statistical Representation of Language and Knowledge 1.3.11 Assignment



原 PPT 独立图片资源

1.3 Statistical Representation of Language and Knowledge 1.3.11 Assignment

原 PPT 独立图片资源

Try and play with the ModelScope Platform


原 PPT 独立图片资源
Try and play with the ModelScope Platform



Register and Login: ModelScope: https://www.modelscope.cn/
Start the Jupyter Lab with CPU and GPU;
Make virtual environment;
Play with them, make sure the GPU is working as expected;
Remember to save data in the persistent path;
Remember to close the Jupyter Lab instance (Limited Time GPU)

The Report for Practical Project


原 PPT 独立图片资源
The Report for Practical Project



Written in IPython notebook
A good experimental report: Introduction, Task description, Mathematical Model, Learning System Design, Experimental Setting, Model Selection, Performance Analysis and Case Study, Conclusion
Learning System Design: Write your math equation and implementation code and explanations
Experimental Setting: Describe your dataset, and hyper parameter candidates;
Model Selection: Perform cross-validation, choose the proper hyper-parameters and illustrate the underfit-overfit phenomenon;
Performance Analysis and Case Study: Evaluate the performance of the chosen model, and then analyze several good cases and bad cases.
Example: https://github.com/Sohaib1424/Support-Vector-Machine-from-scratch/blob/main/SVM.ipynb

Questions?



原 PPT 独立图片资源
Questions?
原 PPT 独立图片资源