Natural Language UnderstandingCourse Introduction


原 PPT 独立图片资源
Natural Language
Understanding

Course Introduction
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Teaching Staff
Content and Goal;
Prerequisites;
Reference Texts;
Assignments and Project.

0.1 Course Introduction 0.1.1 Teaching Staff



原 PPT 独立图片资源

0.1 Course Introduction 0.1.1 Teaching Staff

原 PPT 独立图片资源

Teacher: Mingming Sun


原 PPT 独立图片资源
Teacher: Mingming Sun
Professor at Beijing Institute of Mathematical Sciences and Applications (BIMSA);
Research Area: Cognitive Computing, World Model, Multi-Agent System;

Teaching Assistant: Haitao Su


原 PPT 独立图片资源
Teaching Assistant: Haitao Su
Ph.D. student at Qiuzhen College
Research Area: Safety of Deep Learning and Large Language models.
Email: sht23@mails.tsinghua.edu.cn

0.1 Course Introduction 0.1.2 Course Content and Goals



原 PPT 独立图片资源

0.1 Course Introduction 0.1.2 Course Content and Goals

原 PPT 独立图片资源

Processing vs. Understanding


原 PPT 独立图片资源
Processing vs. Understanding
Natural Language Processing = Processing the Natural Language
Natural Language Understanding = Understanding the Natural Language
Understanding = Processing + ?
John Searle’s Chinese Room
Let’s keep thinking what the “?” might represent.

Content


原 PPT 独立图片资源
Content
Representation
Language:
Knowledge

Symbolic
Distributed
Language Model
Probabilistic LM
Neural LM
LLM
RLHF
Reasoning
Logical Reasoning
Language Reasoning
Agent and System
Tool Use
Planning
Memory

Goal (I)


原 PPT 独立图片资源
Goal (I)
Language intelligence is the core of human intelligence;
The most significant distinction between human intelligence and that of other animals.
Learning natural language understanding is about:
understanding how human comprehend language intelligence
implement language understanding system.

Goal (II)


原 PPT 独立图片资源
Goal (II)
Basic Research Perspective: Understand how human intelligence works.
Applied Research Perspective: Learn how to develop appropriate methods to solve real-world problems in various language scenarios.

For Researcher from Math Background (I)


原 PPT 独立图片资源
For Researcher from Math Background (I)
Strengths
Your mathematical background gives you rigor, abstraction, and pursuit of elegance.
In AI, the main challenge is not programming skills, but shifting ways of thinking.
Embracing Complexity
AI often deals with messy, incomplete, and heuristic phenomena.
Like physicists studying nature, you must observe, experiment, and model before theory.

For Researcher from Math Background (II)


原 PPT 独立图片资源
For Researcher from Math Background (II)
Mutual Enrichment
Heuristics and phenomenological models are stepping stones.
Mathematical depth is essential later to unify these into systematic, elegant theories.
Long-Term Goal
Build your own cognitive picture of intelligence.
Develop a methodological system that bridges mathematical rigor with real-world complexity.

The Path to Goal : Learning from History


原 PPT 独立图片资源
The Path to Goal : Learning from History
The history of AI is composed of continuously discovering and solving problems.
There is no ultimate solution: every scenario has its own specific needs.
We learn from history to build a toolbox and a methodology for solving problems.
So we concern the history of NLU, not just fashion technologies.

Three Key Aspects


原 PPT 独立图片资源
Three Key Aspects
Modeling: How to build mathematic models from linguistic elements to task objectives ?
Evaluation: How to assess the quality of a language system?
Algorithms: How to optimize the model to maximize the evaluation performance?

0.1 Course Introduction 0.1.3 Prerequisites



原 PPT 独立图片资源

0.1 Course Introduction 0.1.3 Prerequisites

原 PPT 独立图片资源

Calculus and Linear Algebra


原 PPT 独立图片资源
Calculus and Linear Algebra 
Understand matrix and vector operations.
Understand matrix factorization techniques, including Principal Component Analysis (PCA) and Singular Value Decomposition (SVD).
Understand gradient computation, including the chain rule and gradients with respect to matrices, among other concepts.

Probability and Statistics


原 PPT 独立图片资源
Probability and Statistics 
Probability Foundations: Understand key concepts: probability space, conditional probability, Bayes’ Theorem, and event dependencies.
Statistical Inference & Estimation: Learn methods for parameter estimation, hypothesis testing, and p-value interpretation.
Sampling Methods: Monte Carlo, Importance Sampling, and MCMC for probabilistic inference.
Information Theory: Apply entropy, cross-entropy, and KL divergence to measure uncertainty and error.

Python Programming


原 PPT 独立图片资源
Python Programming
Proficient in writing Python scripts, functions, and classes.
Familiar with using libraries and packages such as NumPy, scikit-learn, PyTorch, etc.
Capable of developing custom modules and packages.
Skilled in visualizing experimental results.
Experienced in creating command-line interfaces and GUI applications using Streamlit.

Foundations of Machine Learning


原 PPT 独立图片资源
Foundations of Machine Learning 
Key Concepts: including model capacity, overfitting, cross-validation;
Typical Models: linear model, logistic regression, SVM, MLP, deep neural network;
Learning Methods: supervised learning, unsupervised learning, reinforcement learning;
Optimization: gradient descent, back-propagation, etc.
Evaluation Metrics: precision, recall, F1, AUC, ROC.

0.1 Course Introduction 0.1.4 Reference Texts



原 PPT 独立图片资源

0.1 Course Introduction 0.1.4 Reference Texts

原 PPT 独立图片资源

Reference Texts


原 PPT 独立图片资源
Reference Texts
Speech and Language Processing, Dan Jurafsky and James H. Martin. https://web.stanford.edu/~jurafsky/slp3/
Neural Network Methods in Natural Language Processing, Yoav Goldberg
Natural Language Processing, Jacob Eisenstein
Knowledge Graphs: Fundamentals, Techniques, and Applications, Mayank Kejriwal, Craig A. Knoblock, Pedro Szekely

0.1 Course Introduction 0.1.5 Assignments and Final Project



原 PPT 独立图片资源

0.1 Course Introduction 0.1.5 Assignments and Final Project

原 PPT 独立图片资源

Assignments


原 PPT 独立图片资源
Assignments
48% of total score;
4 Assignments, each focusing on a major topic;
Each assignment may involve algorithm design, analysis, and implementation;
Duration: 1-2 weeks for completion.

Final Project


原 PPT 独立图片资源
Final Project
52% of total score;
Default project: Implementing a Mini-LLM;
Custom project: Choose a topic of interests;
Team Size: Up to 3 members
Duration: 4 weeks for completion.

Academic Integrity (I)


原 PPT 独立图片资源
Academic Integrity (I)
Independent Work (Assignments): All assignments must be completed individually. Any form of plagiarism, ghostwriting, or unauthorized collaboration is strictly prohibited.
Teamwork (Final Project): The final project allows collaboration in teams of up to 3 members. However, all members must contribute equally, and the work must remain original.

Academic Integrity (II)


原 PPT 独立图片资源
Academic Integrity (II)
Originality: All code, reports, and project materials must be original. Proper citations must be given when referencing external work.
Academic Integrity: Adhere to academic integrity principles. Any cheating or dishonest behavior will be dealt with severely.
Code and Resource Usage: Only course-approved external resources may be used. Any external libraries or tools used must be properly cited in the report.

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源