Natural Language UnderstandingSymbolic Representing of Language


原 PPT 独立图片资源
Natural Language
Understanding

Symbolic Representing
of Language
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Symbolic Structures of Language
History of Symbolic Language Representation
Case Studies:
Noam Chomsky’s Generative Grammar
Dependency Grammar
Open Information Annotation
Coreference
Semantic Relationship Between Sentences
Obtain the Symbolic Representation

1.1 Symbolic Representation of Language 1.1.1 Symbolic Structures of Language



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.1 Symbolic Structures of Language

原 PPT 独立图片资源

Symbolic Structures of Language


原 PPT 独立图片资源
Symbolic Structures of Language

Corpus
Document
Paragraph
Sentence
Chunk
Word
Morpheme
Character

Character, Morpheme, Word


原 PPT 独立图片资源
Character, Morpheme, Word

Category
English
Chinese
Character
A letter (e.g., 'a', 'b', 'c') has no meaning by itself.
A character (e.g., '水', '火', '学') often has meaning by itself.
Morpheme
The smallest meaningful unit (e.g., 'un-', 'run', '-ing').
A character or a combination of characters that carries meaning (e.g., '学' in '学习', '化' in '现代化').
Word
A combination of morphemes that forms a complete meaning (e.g., 'running', 'unhappy').
Usually a compound word combining multiple morphemes (e.g., '电脑' from '电' + '脑').

Word, Chunk, Sentence


原 PPT 独立图片资源
Word, Chunk, Sentence
Category
Definition
English Examples
Chinese Examples
Word
A combination of morphemes forming a meaningful unit.
'computer', 'running'
'电脑' , '跑步'
Chunk
A group of words forming a meaningful phrase but not a full sentence.
'the big house', 'eating lunch'
'漂亮的房子', '吃午饭'
Sentence
A complete grammatical structure expressing a full thought.
'I am reading a book.'
'南京市长江大桥'

Sentence, Paragraph, Document, Corpus


原 PPT 独立图片资源
Sentence, Paragraph, Document, Corpus
Category
Definition
Note
Sentence
A complete grammatical structure expressing a full thought.

Paragraph
A collection of related sentences forming a coherent unit.
Sentences in it are related and not independent.
Document
A structured collection of paragraphs forming a complete text.
Paragraphs in it are related and not independent.
Corpus
A large collection of documents used for linguistic or computational analysis.
The documents are generally considered independent.

Studies on the Symbolic Representation


原 PPT 独立图片资源
Studies on the Symbolic Representation
Where the elements and structure come from.
How lower-level elements form and constitute the upper structure.
What is the relationship between elements in the same layer.

1.1 Symbolic Representation of Language 1.1.2 History of Symbolic Language Representation



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.2 History of Symbolic Language Representation

原 PPT 独立图片资源

Early Formal Representations in Linguistics—— Ferdinand de Saussure (1857–1913): Structuralism


原 PPT 独立图片资源
Early Formal Representations in Linguistics—— Ferdinand de Saussure (1857–1913): Structuralism
A sign is the basic unit of meaning in language.It consists of:
Signifier (signifiant): The sound pattern or written form (e.g., the word “tree”).
Signified (signifié): The concept or meaning associated with the signifier (the mental image of a tree).
Structural Relationships in Language
Syntagmatic relations: How words are arranged in a sequence (e.g., word order in a sentence).
Paradigmatic relations: How words can be substituted for one another in a given context (e.g., synonyms or different tenses of a verb). In “The cat sleeps,” the word “cat” can be substituted with “dog” (paradigmatic), while “sleeps” follows the subject “cat” (syntagmatic).

Formal Grammar and Symbolic Syntax——Noam Chomsky (1950s–Present): Generative Grammar


原 PPT 独立图片资源
Formal Grammar and Symbolic Syntax——
Noam Chomsky (1950s–Present): Generative Grammar
Phrase Structure Rules & Context-Free Grammar (CFG)
Chomsky introduced formal grammar rules that generate valid sentences.
Example: S → NP VP;NP → Det N;VP → V NP 
Generates sentences like “The cat chased the mouse.”
Deep Structure vs. Surface Structure
A sentence has an underlying deep structure (logical meaning) and a surface structure (actual words).
Example: “John is easy to please” vs. “John is eager to please” (different deep structures despite similar surface structures)
Transformational Grammar
Introduces transformations (e.g., passive voice) to explain structural variations.
Example: “The boy saw the girl” → “The girl was seen by the boy.”

Dependency Grammar and Semantic Structures—— Lucien Tesnière (1959): Dependency Grammar


原 PPT 独立图片资源
Dependency Grammar and Semantic Structures
—— Lucien Tesnière (1959): Dependency Grammar
Emphasize word-to-word relationships instead of hierarchical phrase structures.
Dependency Relations
Unlike phrase structure grammar, which organizes sentences into hierarchical constituents, dependency grammar represents syntax as a network of head-dependent relationships.
In a dependency structure, each word (except the root) depends on another word (its governor/head).
The relationships between words are represented as directed edges in a tree.
Structural Simplicity
Avoids unnecessary intermediate nodes like noun phrases (NP) or verb phrases (VP)
Instead, it directly links words based on their syntactic and semantic relationships.

Abstract Meaning Representation (AMR, 2010s)


原 PPT 独立图片资源
Abstract Meaning Representation (AMR, 2010s)
AMR represents the semantic structure of a sentence as a directed acyclic graph (DAG).
Nodes represent concepts (entities, events, actions).
Edges represent semantic relations.
Predicate-Argument Structure
Verbs: PropBank-style verb senses (e.g., want-01, eat-01).
Roles are labeled using numbered arguments:
ARG0 → Agent/Doer
ARG1 → Theme/Object
ARG2 → Beneficiary
It abstracts away from word order and syntactic variations, meaning different syntactic realizations of the same idea map to the same AMR structure.

(w / want-01
:ARG0 (b / boy)
:ARG1 (e / eat-01
:ARG0 b
:ARG1 (a / apple)))

Open Domain Knowledge Representations (OIA, 2022 and Lingua 2026)


原 PPT 独立图片资源
Open Domain Knowledge Representations (OIA, 2022 and Lingua 2026)
An intermediate representation of cognition elements:
Entities, facts in the world and their relationships;
Each sentence can be represented as a Predicate/Function/Argument structure.
Use original words to express the structure, without loss of information.

The Central Problem: Surface Expression and Underlying Meaning


原 PPT 独立图片资源
The Central Problem: Surface Expression and Underlying Meaning


Theory
View on Surface Expression
View on Underlying Meaning
Key Mechanism of Mapping
Example
Structuralism (Saussure, 1916)
Words and grammar are part of a structured system of signs.
Meaning comes from differences between signs, not direct reference to reality.
Signifier (word form) ↔ Signified (concept)
The word “dog” means “dog” not because of an inherent property, but because it differs from “cat”.
Generative Grammar (Chomsky, 1957)
Sentences have surface structure (what we see) and deep structure (abstract rules).
Deep structure captures core meaning, independent of syntax.
Transformational rules convert deep structure → surface structure.
“John saw Mary” vs. “Mary was seen by John” have different syntax but same deep meaning.
Dependency Grammar (Tesnière, 1959)
Words are connected by dependency relations (head-modifier structure).
Meaning is derived from syntactic relations rather than linear order.
Dependency trees capture who does what to whom.
“The cat chased the mouse” → “chased” is the head, “cat” is subject, “mouse” is object.
Abstract Meaning Representation (AMR, 2010s)
Surface syntax varies, but core meaning is a graph.
Meaning is a directed graph of concepts and relations.
Sentences are mapped into AMR graphs.
“She wants to buy a car” → (w / want-01 :arg0 (s / she) :arg1 (b / buy-01 :arg1 (c / car)))
Open Information Annotation (OIA, 2020s)
Language can be expressed as Predicate/Function/Arguemnt structure;
Meaning is relationships among entities and facts.
Sentences are mapped into OIA Graphs;
“She buy a car” → (buy) –arg1->(She) -arg2->(a car)

1.1 Symbolic Representation of Language 1.1.3 Case Studies



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.3 Case Studies

原 PPT 独立图片资源

Generative Grammar and Constituency Parsing


原 PPT 独立图片资源
Generative Grammar and Constituency Parsing

https://www.structural-learning.com/post/chomskys-theory
Noam Chomsky, cognitive behavioral theory, cognitive development theory

Generative Grammar


原 PPT 独立图片资源
Generative Grammar

https://en.wikipedia.org/wiki/Generative_grammar
undefined

Rules for Generative Grammar


原 PPT 独立图片资源
Rules for Generative Grammar

https://kevinbinz.com/2017/09/05/intro-grammar/
Syntax- Compressed vs Unique Ruleset (1)
Syntax- Simple Tree (1)

Constituency Parsing


原 PPT 独立图片资源
Constituency Parsing

https://kevinbinz.com/2017/09/05/intro-grammar/
syntax_tree_construction

Dependency Grammar and Parsing


原 PPT 独立图片资源
Dependency Grammar and Parsing

原 PPT 独立图片资源
https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture04-dep-parsing.pdf

Why Dependency


原 PPT 独立图片资源
Why Dependency

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture04-dep-parsing.pdf
Dependency goes back a long way:
To Pāṇini’s grammar (c. 5th century BCE)
Basic approach of 1st millennium Arabic grammarians
Solving the ambiguity in language understanding and parsing into the generative gramma actually rely on dependency information.
I eat the food [on the table] [in the morning].
I saw the man with the telescope. ?
原 PPT 独立图片资源

Dependency Parsing: Information Source


原 PPT 独立图片资源
Dependency Parsing: Information Source

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1162/handouts/SLoSP-2014-4-dependencies.pdf
Bilexical affinities: [issues → the] is plausible
Dependency distance: mostly with nearby words
Intervening material: Dependencies rarely span intervening verbs or punctuation
Valency of heads: How many dependents on which side are usual for a head?
原 PPT 独立图片资源

Relationship between Dependency and Constituency


原 PPT 独立图片资源
Relationship between Dependency and Constituency

https://web.stanford.edu/class/cs224n/slides_w25/cs224n-2025-lecture04-dep-parsing.pdf
Projective parse: There are no crossing dependency arcs when the words are laid out in their linear order, with all arcs above the words
Dependencies corresponding to a Constituency Tree must be projective
Most syntactic structure is projective, but dependency theory normally does allow non-projective structures to account for displaced constituents.
原 PPT 独立图片资源

Open Information Annotation


原 PPT 独立图片资源
Open Information Annotation

Mingming Sun, et.al . 2020. A Predicate-Function-Argument Annotation of Natural Language for Open-Domain Information eXpression. In Proceedings of EMNLP, pages 2140–2150
原 PPT 独立图片资源

Why OIA


原 PPT 独立图片资源
Why OIA

Cognition happens in intermediate level:
Higher than byte, character, and words;
Lower than high-level real-world tasks;
Cognition is about entities, facts in the world and their relationships;
The basis of Cognitive Language Modeling is:
Entities, Facts and Relationships between them.

Intermediate Representation of Knowledge


原 PPT 独立图片资源
Intermediate Representation of Knowledge

Constants: express entities (“the solar system”) and any abstract concept (“expensive”);
Functions: 𝑓(𝑎𝑟𝑔1,⋯,𝑎𝑟𝑔𝑛)⟶{𝑒} express query of entities or delegation of entities, such as “the CEO of X”, the questions and wh-clause “when Y”
Predicates: p(𝑎𝑟𝑔1,⋯,𝑎𝑟𝑔𝑛)⟶{0,1}, express relationships or logical connections among entities, predicates, and functions, such as “X buy Y”, “X says Y”, “Y, because Z”.

Lingua Graph


原 PPT 独立图片资源
Lingua Graph

原 PPT 独立图片资源

Lingua Graph


原 PPT 独立图片资源
Lingua Graph

原 PPT 独立图片资源

Lingua Graph


原 PPT 独立图片资源
Lingua Graph

原 PPT 独立图片资源

Sentence Parsing: Intermediate Level Expression


原 PPT 独立图片资源
Sentence Parsing: Intermediate Level Expression

原 PPT 独立图片资源

Sentence Parsing: Break through the Limitation of Sequence


原 PPT 独立图片资源
Sentence Parsing: Break through the Limitation of Sequence

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1162/handouts/SLoSP-2014-4-dependencies.pdf
原 PPT 独立图片资源

Coreference


原 PPT 独立图片资源
Coreference

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/slides/cs224n-2019-lecture16-coref.pdf
Identify all mentions that refer to the same real world entity.
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Types of Coreference


原 PPT 独立图片资源
Types of Coreference

Pronominal Coreference
Pronouns referring to noun phrases
Example: “Lisa is a scientist. She works at NASA.”
Nominal Coreference
Two noun phrases referring to the same entity
Example: “The president gave a speech. The leader addressed the nation.”
Bridging Coreference
Inferring the relationship between mentions
Example: “I bought a laptop. The screen is amazing.”

Coreference is Complex


原 PPT 独立图片资源
Coreference is Complex

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/slides/cs224n-2019-lecture16-coref.pdf
原 PPT 独立图片资源

Coreference is Ccrucial


原 PPT 独立图片资源
Coreference is Ccrucial

Resolving Ambiguity in Language
“Mary told Sue that she won the award.”
Maintaining Context and Coherence
“A scientist discovered a new particle. The researcher presented her findings.”
Constructing Accurate Knowledge Graphs
“Google acquired DeepMind. The AI lab specializes in reinforcement learning.”
“Apple Inc. released a new iPhone. The Cupertino-based company expects high sales.”

Coherence Relations


原 PPT 独立图片资源
Coherence Relations 

https://www.sfu.ca/~mtaboada/lot/coherence_relations.html


Logical and semantic connections between sentences or discourse segments that contribute to the overall coherence of a text.
These relations help make discourse understandable and structured.
原 PPT 独立图片资源

Theoretical Foundations of Coherence Relations


原 PPT 独立图片资源
Theoretical Foundations of Coherence Relations



Rhetorical Structure Theory (RST)
Introduced by Mann & Thompson (1988).
Views text as a hierarchical structure with different relations.
Coherence is achieved through a network of nucleus (main idea) and satellite (supporting idea).
Example RST relations:
Cause-Effect
Elaboration
Contrast
Hobbs’ Theory of Coherence (1979)
Defines coherence as an inferential process where readers fill in missing links.
Emphasizes logical reasoning and causal inferences.
Example: John fell off his bike. He broke his arm.”
Humans infer a causal connection between these events.
Kehler’s Coherence Model (2002)
Inspired by theories of event coherence.
Divides coherence into three main categories:
Resemblance: Similarities between sentences.
Cause-Effect: Causal relationships.
Contiguity:: Events occurring in a sequence.

Types of Coherence Relations


原 PPT 独立图片资源
Types of Coherence Relations



Type
Description
Example
Causal Relations (因果关系)
One sentence (cause) leads to another (effect).
“It rained heavily. The streets got flooded.” (Cause → Effect)
- Result (结果关系)
The second sentence is a direct consequence of the first.
“She studied hard. She passed the exam.”
- Reason/Explanation (原因关系)
The second sentence explains the first.
“He left early because he was feeling sick.”
Temporal Relations (时间关系)
One event happens before, after, or at the same time as another.
“She woke up. Then she made coffee.”
- Sequential (顺序关系)
One event follows another.
“He finished work. Then he went home.”
- Simultaneous (同时关系)
Two events happen at the same time.
“She was cooking while he was reading.”
Contrastive Relations (对比关系)
The two segments express opposite ideas.
“She likes tea. He prefers coffee.”
- Contrast (对照关系)
Highlights a clear opposition.
“She loves winter, but I hate it.”
- Concession (让步关系)
Acknowledges one idea but presents another contradicting it.
“It was raining, but they still went out.”
Additive Relations (添加关系)
The second segment adds more information.
“She speaks English. She also knows French.”
- Elaboration (详细说明关系)
The second sentence expands on the first.
“He bought a car. It’s a red Tesla.”
- Parallel (并列关系)
The two segments describe similar ideas.
“She enjoys hiking. He likes running.”
Attribution Relations (引述关系)
One sentence attributes an action or statement to someone.
“The professor said that the exam would be postponed.”
- Statement (陈述关系)
Reporting an assertion.
“Experts claim that AI will change the world.”
- Question (提问关系)
Reporting a question.
“She asked if I could help her.”
Condition & Hypothetical Relations (条件关系)
Expresses a condition that affects an outcome.
“If you study hard, you will pass the exam.”
- Conditional (条件关系)
Explicit “if-then” condition.
“If it rains, we will cancel the trip.”
- Hypothetical (假设关系)
A condition that is unlikely or speculative.
“If I were rich, I would travel the world.”
Purpose & Means Relations (目的与手段关系)
Expresses a goal and the method to achieve it.
“He left early to catch the train.”
- Purpose (目的关系)
The second sentence explains the goal of the first.
“She trains daily to become a professional athlete.”
- Means (手段关系)
The second sentence describes how something is achieved.
“He solved the problem by using an equation.”

Importance of Coherence Relations


原 PPT 独立图片资源
Importance of Coherence Relations



Coherence relations play a crucial role in:
Text Understanding – Helps in resolving ambiguities in meaning.
Text Generation & Summarization – Ensures well-structured, logical output.
Discourse Parsing – Used in NLP tasks for organizing large texts.
Question Answering – Helps infer implicit knowledge.
Dialogue Systems – Maintains logical conversational flow

1.1 Symbolic Representation of Language 1.1.4 Obtaining Symbolic Representations



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.4 Obtaining Symbolic Representations

原 PPT 独立图片资源

Rule-Based and Heuristic Methods (Pre-1990s)


原 PPT 独立图片资源
Rule-Based and Heuristic Methods (Pre-1990s)



Expert Hand-Crafted Grammar Rules
In English, a verb (V) typically governs a noun phrase (NP) as its subject or object.
Rule: V → NP (subject), V → NP (object)
Limitations:
✔ High precision in well-defined cases
❌ Poor scalability, hard to adapt to new languages and ambiguous cases

Statistical Parsing (1990s–2000s)


原 PPT 独立图片资源
Statistical Parsing (1990s–2000s)



Probabilistic extension of Generative Gramma:
Each rule has an associated probability.
Data-Driven Statistical Parsing
Data Sets: Penn Treebank, Universal Dependencies.
Lexicalized Models: Assigned probabilities to dependencies based on word co-occurrence. The probability of a dependency relation 𝑑𝑒𝑝(ℎ𝑒𝑎𝑑, 𝑐ℎ𝑖𝑙𝑑) was often calculated using: 𝑃(𝑐ℎ𝑖𝑙𝑑∣ ℎ𝑒𝑎𝑑, 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛)
Example: If “eats” frequently governs “apple” in a treebank, then the parser learns that “eats → apple” is a probable dependency.
Limitations:
✔ More flexible than rule-based methods
❌ Struggled with long-range dependencies and ambiguous structures
P(S → NP VP) = 0.9
P(VP → V NP) = 0.7

Classical Machine Learning (2000s–2015)


原 PPT 独立图片资源
Classical Machine Learning (2000s–2015)



Transition-Based Dependency Parsing
Key Idea: Parse sentences by making a sequence of shift-reduce decisions.
Uses a classifier (SVM, perceptron, or neural network) to decide each parsing step.
Graph-Based Dependency Parsing
Key Idea: Treat parsing as a graph optimization problem.
Define a scoring function to determine the best dependency tree.
Limitations:
✔ Much Powerful than Statistical Parser;
❌ Need Feature Engineering; Long-term dependencies not well modeled.

Neural Network-Based Methods (2014–2017)


原 PPT 独立图片资源
Neural Network-Based Methods (2014–2017)



Machine Learning goes to Deep Learning:
Feature Engineering → Neural Embedding Representation
Shallow Machine → BiLSTM + Shallow Machine
Limitations:
✔ Long-term dependencies well modeled.
❌ Inherent pattern of language not used.

Pretrained Model (2018–2022)


原 PPT 独立图片资源
Pretrained Model (2018–2022)



Pretrained Model + Fine-tuning:
BiLSTM → Transformer
Random Initialization → Pretrained /Bert Weights
Limitations:
✔ Inherent pattern of language used.
❌ Information across tasks not used.

AGI (2022–)


原 PPT 独立图片资源
AGI (2022–)



Specific Task → AGI:
Hidden Task Info → Explicit Task Instructions
Output Labels → Natural Language Generation
Training on one task → Training on every task
Limitations:
✔ Information across tasks not used.
❌ Language instructions may not be accurate. Subtle task requirement may need finetuning.

Tree-Bank Datasets


原 PPT 独立图片资源
Tree-Bank Datasets



Tree Bank
Tree: Linguistic structures, particularly syntactic structures, are naturally represented as trees in formal grammar.
Bank: The dataset stores (or “banks”) a large number of syntactically annotated sentences.
Instances:
Penn Tree-Bank
Universal Dependencies
OntoNote

Penn Tree Bank (PTB)


原 PPT 独立图片资源
Penn Tree Bank (PTB)



One of the most influential linguistic corpora;
Developed at the University of Pennsylvania ;
Language: English
Annotation Type: Constituency-based phrase structure trees, POS tags, and some dysfluency annotations.
Source Data:
Wall Street Journal (WSJ) articles (about 1 million words).
Brown Corpus (a balanced dataset of American English).
Switchboard (spoken conversations).
ATIS-3 material
Annotation Scheme:
Each word is labeled with a POS tag.
Phrases are marked with constituency labels (e.g., NP, VP, PP).

原 PPT 独立图片资源

Universal Dependencies (UD)


原 PPT 独立图片资源
Universal Dependencies (UD)



Cross-linguistic dependency annotation that standardizes the annotation of syntactic structure and morphology across many languages.  https://universaldependencies.org/
Language: More than 100 languages;
Annotation Type: Cross-Linguistic Consistency; Universal POS Tags; Morphological Features
Annotation Scheme: CoNLL-U format

1 She she PRON _ Number=Sing|Person=3 2 nsubj _ _
2 said say VERB _ Tense=Past 0 root _ _
3 that that SCONJ _ _ 5 mark _ _
4 he he PRON _ Number=Sing|Person=3 5 nsubj _ _
5 likes like VERB _ Tense=Pres|Number=Sing 2 ccomp _ _
6 apples apple NOUN _ Number=Plur 5 obj _ _
7 . . PUNCT _ _ 2 punct _ _

OntoNote


原 PPT 独立图片资源
OntoNote



Large-scale, multi-layered corpus offering syntactic, semantic, and discourse-level annotations
Created as part of the DARPA GALE program;
Language: English, Chinese, Arabic
Annotation Type: POS tags, Constituency Trees, Word Sense Disambiguation, Named Entity Recognition, Coreference Resolution, PropBank Semantic Role Labeling (SRL).
Source Data:
Newswire (Wall Street Journal, Xinhua News)
Broadcast News (CNN, MSNBC)
Broadcast Conversations (Transcribed interviews)
Web Texts (Blogs, discussion forums)
Telephone Conversations (Switchboard corpus)
Annotation Scheme: CoNLL-U format and others

Comparison


原 PPT 独立图片资源
Comparison



Feature
Penn Treebank (PTB)
Universal Dependencies (UD)
OntoNotes
POS Tagging
✅ Yes
✅ Yes
✅ Yes
Constituency Trees
✅ Yes
❌ No
✅ Yes
Dependency Parsing
❌ No
✅ Yes
❌ No
Named Entities (NER)
❌ No
❌ No
✅ Yes
Coreference
❌ No
❌ No
✅ Yes
Semantic Role Labeling
❌ No
❌ No
✅ Yes
Word Sense Disambiguation (WSD)
❌ No
❌ No
✅ Yes
Multilingual
❌ No
✅ Yes
✅ Yes (Eng, Chi, Ara)

Types of Tasks


原 PPT 独立图片资源
Types of Tasks



Segmentation
Element Identification
Element Attribute Prediction
Pairwise Relationship Identification

Recall: Neural Learning Paradigms


原 PPT 独立图片资源
Recall: Neural Learning Paradigms
原 PPT 独立图片资源

Recall: Applications: Sequence Classification


原 PPT 独立图片资源
Recall: Applications: Sequence Classification
Feed the last hidden-state to the classifier;
Widely used in NLP task such as text classification;
Text classification model based on recurrent neural network.

Recall: Applications: Sequence Labeling


原 PPT 独立图片资源
Recall: Applications: Sequence Labeling
Predict a label for each item of input sequence.
Widely used in NLP task like NER;
bilstm

Recall: Applications: Sequence Prediction


原 PPT 独立图片资源
Recall: Applications: Sequence Prediction
Modeling the problem as a sequence prediction task;
Typical application: Machine Translation, AGI LLM;
原 PPT 独立图片资源
原 PPT 独立图片资源

Segmentation —— Task Modeling


原 PPT 独立图片资源
Segmentation —— Task Modeling



Modeling as a Sequence Labeling Task



Modeling as a Sequence Prediction Task
https://medium.com/towards-data-science/joint-khmer-word-segmentation-and-pos-tagging-cad650e78d30
原 PPT 独立图片资源
STA
STA
STA
STA

Segmentation —— Network Architecture


原 PPT 独立图片资源
Segmentation —— Network Architecture



原 PPT 独立图片资源
https://medium.com/towards-data-science/joint-khmer-word-segmentation-and-pos-tagging-cad650e78d30

Segmentation —— Algorithm


原 PPT 独立图片资源
Segmentation —— Algorithm



Loss:
Typical Classification Loss: Cross-Entropy
Optimization:
Gradient Descent: Adam

Segmentation —— Evaluation


原 PPT 独立图片资源
Segmentation —— Evaluation



Model Level Evaluation:
Binary Classification Error: Precision, Recall, F1
Task Level Evaluation:
Words Prediction Performance: Precision, Recall, F1
Boundary Prediction Performance: Precision, Recall, F1

Element Identification——Task Modeling


原 PPT 独立图片资源
Element Identification——Task Modeling



Same with Segmentation, just change the target label system.
Typical Task: Noun Phrase Identification, Entity Detection
https://medium.com/@muskaan.maurya06/name-entity-recognition-and-various-tagging-schemes-533f2ac99f52
手机屏幕的截图

AI 生成的内容可能不正确。

Element Attribute Prediction——Task Modeling


原 PPT 独立图片资源
Element Attribute Prediction——Task Modeling



Same with Segmentation, just change the target label system.
Typical Task: POSTag Prediction, Named Entity Recognition
Can merge with Element Identification Task by merging the label system.
https://www.researchgate.net/publication/330424080_Adverse_Drug_Events_Detection_in_Clinical_Notes_by_Jointly_Modeling_Entities_and_Relations_Using_Neural_Networks
BIO Tagging for the entity extraction.

Pairwise Relationship Identification——Task Modeling


原 PPT 独立图片资源
Pairwise Relationship Identification——Task Modeling



Elementwise Representation + Bi-Affine transformation → Pairwise Relationship
Bi-Affine transformation: 𝐵𝑖𝐴𝑓𝑓𝑖𝑛𝑒(𝑥1,𝑥2) is a vector with i-th element equals to:
原 PPT 独立图片资源

Pairwise Relationship Identification——Task Modeling


原 PPT 独立图片资源
Pairwise Relationship Identification——Task Modeling



Task Constraint:
Dependency Parsing: Tree structure;
Coreference: Multiple Fully Connected Graph;
OIA Graph: DAG;
原 PPT 独立图片资源
https://www.cs.princeton.edu/courses/archive/fall19/cos484/lectures/lec11.pdf

Pairwise Relationship Identification——Network Architecture


原 PPT 独立图片资源
Pairwise Relationship Identification——Network Architecture



原 PPT 独立图片资源
https://arxiv.org/pdf/1611.01734

Parsing Complex Graphs: Word-OIA


原 PPT 独立图片资源
Parsing Complex Graphs: Word-OIA

原 PPT 独立图片资源
Xin Wang, et. al. 2022. OIE@OIA: an Adaptable and Efficient Open Information Extraction Framework. In Proceedings of ACL, pages 6213–6226, Dublin, Ireland.

1.1 Symbolic Representation of Language 1.1.5 Suggested Readings



原 PPT 独立图片资源

1.1 Symbolic Representation of Language 1.1.5 Suggested Readings

原 PPT 独立图片资源

Books


原 PPT 独立图片资源
Books
Speech and Language Processing, Dan Jurafsky and James H. Martin. https://web.stanford.edu/~jurafsky/slp3/
Neural Network Methods in Natural Language Processing, Yoav Goldberg

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源