Natural Language UnderstandingCourse Introduction


原 PPT 独立图片资源
Natural Language
Understanding

Course Introduction
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Teaching Staff
Content and Goal;
Prerequisites;
Reference Texts;
Assignments and Project.

0.1 Course Introduction 0.1.1 Teaching Staff



原 PPT 独立图片资源

0.1 Course Introduction 0.1.1 Teaching Staff

原 PPT 独立图片资源

Teacher: Mingming Sun


原 PPT 独立图片资源
Teacher: Mingming Sun
Professor at Beijing Institute of Mathematical Sciences and Applications (BIMSA);
Research Area: Cognitive Computing, World Model, Multi-Agent System;

Teaching Assistant: Haitao Su


原 PPT 独立图片资源
Teaching Assistant: Haitao Su
Ph.D. student at Qiuzhen College
Research Area: Safety of Deep Learning and Large Language models.
Email: sht23@mails.tsinghua.edu.cn

0.1 Course Introduction 0.1.2 Course Content and Goals



原 PPT 独立图片资源

0.1 Course Introduction 0.1.2 Course Content and Goals

原 PPT 独立图片资源

Processing vs. Understanding


原 PPT 独立图片资源
Processing vs. Understanding
Natural Language Processing = Processing the Natural Language
Natural Language Understanding = Understanding the Natural Language
Understanding = Processing + ?
John Searle’s Chinese Room
Let’s keep thinking what the “?” might represent.

Content


原 PPT 独立图片资源
Content
Representation
Language:
Knowledge

Symbolic
Distributed
Language Model
Probabilistic LM
Neural LM
LLM
RLHF
Reasoning
Logical Reasoning
Language Reasoning
Agent and System
Tool Use
Planning
Memory

Goal (I)


原 PPT 独立图片资源
Goal (I)
Language intelligence is the core of human intelligence;
The most significant distinction between human intelligence and that of other animals.
Learning natural language understanding is about:
understanding how human comprehend language intelligence
implement language understanding system.

Goal (II)


原 PPT 独立图片资源
Goal (II)
Basic Research Perspective: Understand how human intelligence works.
Applied Research Perspective: Learn how to develop appropriate methods to solve real-world problems in various language scenarios.

For Researcher from Math Background (I)


原 PPT 独立图片资源
For Researcher from Math Background (I)
Strengths
Your mathematical background gives you rigor, abstraction, and pursuit of elegance.
In AI, the main challenge is not programming skills, but shifting ways of thinking.
Embracing Complexity
AI often deals with messy, incomplete, and heuristic phenomena.
Like physicists studying nature, you must observe, experiment, and model before theory.

For Researcher from Math Background (II)


原 PPT 独立图片资源
For Researcher from Math Background (II)
Mutual Enrichment
Heuristics and phenomenological models are stepping stones.
Mathematical depth is essential later to unify these into systematic, elegant theories.
Long-Term Goal
Build your own cognitive picture of intelligence.
Develop a methodological system that bridges mathematical rigor with real-world complexity.

The Path to Goal : Learning from History


原 PPT 独立图片资源
The Path to Goal : Learning from History
The history of AI is composed of continuously discovering and solving problems.
There is no ultimate solution: every scenario has its own specific needs.
We learn from history to build a toolbox and a methodology for solving problems.
So we concern the history of NLU, not just fashion technologies.

Three Key Aspects


原 PPT 独立图片资源
Three Key Aspects
Modeling: How to build mathematic models from linguistic elements to task objectives ?
Evaluation: How to assess the quality of a language system?
Algorithms: How to optimize the model to maximize the evaluation performance?

0.1 Course Introduction 0.1.3 Prerequisites



原 PPT 独立图片资源

0.1 Course Introduction 0.1.3 Prerequisites

原 PPT 独立图片资源

Calculus and Linear Algebra


原 PPT 独立图片资源
Calculus and Linear Algebra 
Understand matrix and vector operations.
Understand matrix factorization techniques, including Principal Component Analysis (PCA) and Singular Value Decomposition (SVD).
Understand gradient computation, including the chain rule and gradients with respect to matrices, among other concepts.

Probability and Statistics


原 PPT 独立图片资源
Probability and Statistics 
Probability Foundations: Understand key concepts: probability space, conditional probability, Bayes’ Theorem, and event dependencies.
Statistical Inference & Estimation: Learn methods for parameter estimation, hypothesis testing, and p-value interpretation.
Sampling Methods: Monte Carlo, Importance Sampling, and MCMC for probabilistic inference.
Information Theory: Apply entropy, cross-entropy, and KL divergence to measure uncertainty and error.

Python Programming


原 PPT 独立图片资源
Python Programming
Proficient in writing Python scripts, functions, and classes.
Familiar with using libraries and packages such as NumPy, scikit-learn, PyTorch, etc.
Capable of developing custom modules and packages.
Skilled in visualizing experimental results.
Experienced in creating command-line interfaces and GUI applications using Streamlit.

Foundations of Machine Learning


原 PPT 独立图片资源
Foundations of Machine Learning 
Key Concepts: including model capacity, overfitting, cross-validation;
Typical Models: linear model, logistic regression, SVM, MLP, deep neural network;
Learning Methods: supervised learning, unsupervised learning, reinforcement learning;
Optimization: gradient descent, back-propagation, etc.
Evaluation Metrics: precision, recall, F1, AUC, ROC.

0.1 Course Introduction 0.1.4 Reference Texts



原 PPT 独立图片资源

0.1 Course Introduction 0.1.4 Reference Texts

原 PPT 独立图片资源

Reference Texts


原 PPT 独立图片资源
Reference Texts
Speech and Language Processing, Dan Jurafsky and James H. Martin. https://web.stanford.edu/~jurafsky/slp3/
Neural Network Methods in Natural Language Processing, Yoav Goldberg
Natural Language Processing, Jacob Eisenstein
Knowledge Graphs: Fundamentals, Techniques, and Applications, Mayank Kejriwal, Craig A. Knoblock, Pedro Szekely

0.1 Course Introduction 0.1.5 Assignments and Final Project



原 PPT 独立图片资源

0.1 Course Introduction 0.1.5 Assignments and Final Project

原 PPT 独立图片资源

Assignments


原 PPT 独立图片资源
Assignments
48% of total score;
4 Assignments, each focusing on a major topic;
Each assignment may involve algorithm design, analysis, and implementation;
Duration: 1-2 weeks for completion.

Final Project


原 PPT 独立图片资源
Final Project
52% of total score;
Default project: Implementing a Mini-LLM;
Custom project: Choose a topic of interests;
Team Size: Up to 3 members
Duration: 4 weeks for completion.

Academic Integrity (I)


原 PPT 独立图片资源
Academic Integrity (I)
Independent Work (Assignments): All assignments must be completed individually. Any form of plagiarism, ghostwriting, or unauthorized collaboration is strictly prohibited.
Teamwork (Final Project): The final project allows collaboration in teams of up to 3 members. However, all members must contribute equally, and the work must remain original.

Academic Integrity (II)


原 PPT 独立图片资源
Academic Integrity (II)
Originality: All code, reports, and project materials must be original. Proper citations must be given when referencing external work.
Academic Integrity: Adhere to academic integrity principles. Any cheating or dishonest behavior will be dealt with severely.
Code and Resource Usage: Only course-approved external resources may be used. Any external libraries or tools used must be properly cited in the report.

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源

Natural Language UnderstandingProgramming and Environment


原 PPT 独立图片资源
Natural Language
Understanding

Programming and Environment
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
Why Python
Programming Techniques
Soft Development Skills
Experimental Environment
Suggested Readings

0.2 Python Programming 0.2.1 Why Python



原 PPT 独立图片资源

0.2 Python Programming 0.2.1 Why Python

原 PPT 独立图片资源

Why Python?


原 PPT 独立图片资源
Why Python?

Most Popular Programming Languages Survey
Programming languages popularity in 2024: insights from the TIOBE index
https://www.researchgate.net/publication/387584210_Popularity_of_programming_languages

Efficient for Human


原 PPT 独立图片资源
Efficient for Human
Easy to learn, read, write, and run;
Interactive environment:
Quick experiment;
Easy to debug;
Great IDE:
Jupyter lab and notebook
Easy to merge into current system:
IPython, JPython

Compiler VS Interpreter VS Assembler 1
Compiler VS Interpreter VS Assembler 2

Efficient for Machine


原 PPT 独立图片资源
Efficient for Machine
Why python can be used in high performance area, such AI?
What is the Pareto Principle? | IxDF
原 PPT 独立图片资源
http://blog.behnel.de/posts/cython-pybind11-cffi-which-tool-to-choose.html

Efficient for Machine: Cython


原 PPT 独立图片资源
Efficient for Machine: Cython
Montecarlo
https://inlocrobotics.com/en/optimizing-python-with-cython/

Efficient for Machine: Pybind11


原 PPT 独立图片资源
Efficient for Machine: Pybind11
原 PPT 独立图片资源

Great Ecosystem and Community


原 PPT 独立图片资源
Great Ecosystem and Community
Open-source & Accessible: Free to use, with a vast, growing ecosystem.
Rich Libraries & Frameworks:
Data Science: pandas, NumPy
Machine Learning: Scikit-learn, PyTorch
Web Development: Django, Flask
Thriving Community:
Extensive documentation, forums, and tutorials.
Active contributors and collaborative development

0.2 Python Programming 0.2.2 Programming Techniques



原 PPT 独立图片资源

0.2 Python Programming 0.2.2 Programming Techniques

原 PPT 独立图片资源

Data Processing and Analysis (I)


原 PPT 独立图片资源
Data Processing and Analysis (I)
Regex
A powerful tool for pattern matching and text processing.
Helps with tasks like searching, replacing, or validating strings.

Json (JavaScript Object Notation)
A lightweight data format.
Commonly used for serialize data with multiple attributes, for example, sample with feature and labels.

原 PPT 独立图片资源
原 PPT 独立图片资源

Data Processing and Analysis (II)


原 PPT 独立图片资源
Data Processing and Analysis (II)
Pandas: Table Data Processing

Example of Pandas DataFrame
Force-directed network visualization of the Quaker data, created in Gephi
NetworkX: Graph Data Processing
https://programminghistorian.org/en/lessons/exploring-and-analyzing-network-data-with-python
https://codingsight.com/an-overview-of-dataframes-in-python/

Numerical Computing and Machine Learning


原 PPT 独立图片资源
Numerical Computing and Machine Learning
Numpy: Powerful Numerical Computations

原 PPT 独立图片资源
https://www.techprofree.com/most-important-features-of-scikit-learn/
numpy
Scikit-Learn: Machine Learning Made Simple
https://decodingdatascience.com/numpy-the-foundation-of-data-science-and-ai/

Deep Learning


原 PPT 独立图片资源
Deep Learning

Pytorch: Deep Learning Framework
Huggingface: Platform for Deep Learning
PyTorch modules
A table describing hugging face datasets, models, and spaces
https://www.scalablepath.com/machine-learning/hugging-face
https://www.altexsoft.com/blog/pytorch-library/

Visualization


原 PPT 独立图片资源
Visualization
Matplotlib:
Visualization library for data science, machine learning, and scientific research for data exploration and presentation.

原 PPT 独立图片资源
https://medium.com/@aakash_7/data-visualization-with-matplotlib-d7967ee42fce

GUI


原 PPT 独立图片资源
GUI
Streamlit
An open-source Python library that makes it easy to create interactive web applications for data science and machine learning projects.
Allows you to turn Python scripts into web apps with minimal code.
Ideal for quickly sharing results, visualizations, and models in a user-friendly interface.

图形用户界面, 网站

AI 生成的内容可能不正确。
https://blog.streamlit.io/introducing-streamlit-cloud/

0.2 Python Programming 0.2.3 Software Development Skills



原 PPT 独立图片资源

0.2 Python Programming 0.2.3 Software Development Skills

原 PPT 独立图片资源

Virtual Environment and Package Management


原 PPT 独立图片资源
Virtual Environment and Package Management
A virtual environment is an isolated environment that:
Manage dependencies for different projects separately.
Prevents conflicts between different projects.
Enables reproducibility by isolating project-specific libraries.
Package Management:
It allows you to install, upgrade, and remove packages from your environment.
Recommendation:
Micromamba + Pip

virtual environments
https://www.whiteboxml.com/blog/the-definitive-guide-to-python-virtual-environments-with-conda

Source Code Version Control


原 PPT 独立图片资源
Source Code Version Control
Version control is a system:
Track changes: Easily see what changes have been made to the code, who made them, and why.
Collaboration: Multiple developers can work on the same project without overwriting each other’s work.
History: Revert back to previous versions of the code if needed.
Recommendation: Github

Frame 1 (1)
https://www.legitsecurity.com/blog/github-privilege-escalation-vulnerability

0.2 Python Programming 0.2.4 Experimental Environment



原 PPT 独立图片资源

0.2 Python Programming 0.2.4 Experimental Environment

原 PPT 独立图片资源

Jupyter Lab and Notebook


原 PPT 独立图片资源
Jupyter Lab and Notebook
Lab:
Web-based interactive development environment 
Notebook
An interactive web-based environment for running code, visualizing data, and documenting results.
Ideal for data analysis, machine learning, and educational purposes.

原 PPT 独立图片资源
https://docs.jupyter.org/en/latest/

Experimental Platform


原 PPT 独立图片资源
Experimental Platform
Provide unlimited CPU resource and 100 hours GPU;
With jupyter notebook support;
Student should finish programming assignment by:
Writing document and code in the notebook,
Test the code in the provided computer,
Visualize the result;
Submit the final notebook.

0.2 Python Programming 0.2.5 Suggested Readings



原 PPT 独立图片资源

0.2 Python Programming 0.2.5 Suggested Readings

原 PPT 独立图片资源

Books


原 PPT 独立图片资源
Books
Automated the Boring Stuff with Python, by Al Sweigart, https://automatetheboringstuff.com/
Python for Data Analysis, by Wes McKinney https://wesmckinney.com/book/

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源

Natural Language UnderstandingMachine Learning Basic


原 PPT 独立图片资源
Natural Language
Understanding

Machine Learning Basic
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
What is Machine Learning
Building A Machine Learning System
Machine Learning Models
Optimization Methods
Generalization and Model Selection
Suggested Readings

0.3 Machine Learning Basics 0.3.1 What Is Machine Learning



原 PPT 独立图片资源

0.3 Machine Learning Basics 0.3.1 What Is Machine Learning

原 PPT 独立图片资源

Domains of AI


原 PPT 独立图片资源
Domains of AI

Domains of Artificial Intelligence (AI)
https://www.instagram.com/dataprocomputersofficial/p/C11GBTPRn-F/

What is Machine Learning


原 PPT 独立图片资源
What is Machine Learning
Learning is any process by which a system improves performance from experience. —— Herbert Simon
Machine Learning is the field of study that gives computers the ability to learn without being explicitly programmed. —— Arthur Samuel, 1959
Machine Learning is the science (and art) of programming computers so they can learn from data (experience).

A Formal Definition by Tom Mitchell (1998)


原 PPT 独立图片资源
A Formal Definition by Tom Mitchell (1998)
Machine Learning is the study of algorithms that
improve their performance P
at some task T
with experience E.
A well-defined learning task is given by <P, T, E>.

Why Machine Learning (I)


原 PPT 独立图片资源
Why Machine Learning (I)
The traditional approach:
You must read and understand the experience yourself;
Write a long list of complex rules pretty hard to maintain.

原 PPT 独立图片资源
https://www.slideshare.net/slideshow/the-fundamentals-of-machine-learning/238653218

Why Machine Learning (II)


原 PPT 独立图片资源
Why Machine Learning (II)
The Machine Learning approach:
Machine learns from experience;
Faster to generate a program (model);
The performance is generally better.

原 PPT 独立图片资源
https://www.slideshare.net/slideshow/the-fundamentals-of-machine-learning/238653218

Why Machine Learning (III)


原 PPT 独立图片资源
Why Machine Learning (III)
Machine Learning helps human learning, think and discover.

原 PPT 独立图片资源
https://www.slideshare.net/slideshow/the-fundamentals-of-machine-learning/238653218

Why Machine Learning (IV)


原 PPT 独立图片资源
Why Machine Learning (IV)
Machine Learning is great for:
Problems for which existing solutions require a lot of hand-tuning or long lists of rules: one Machine Learning algorithm can often simplify code and perform better.
Complex problems for which there is no good solution at all using a traditional approach: the best Machine Learning techniques can find a solution.

https://www.slideshare.net/slideshow/the-fundamentals-of-machine-learning/238653218

0.3 Machine Learning Basics 0.3.2 Building Machine Learning Systems



原 PPT 独立图片资源

0.3 Machine Learning Basics 0.3.2 Building Machine Learning Systems

原 PPT 独立图片资源

Building A Machine Learning System (I)


原 PPT 独立图片资源
Building A Machine Learning System (I)

https://insights.blackcoffer.com/the-workflow-of-a-machine-learning-artificial-intelligence-project/
原 PPT 独立图片资源

Building A Machine Learning System (II)


原 PPT 独立图片资源
Building A Machine Learning System (II)

Yae Won Park, et.al, Radiomics features of hippocampal regions in magnetic resonance imaging can differentiate medial temporal lobe epilepsy patients from healthy controls, Scientific Reports 2020
Workflow of image processing, radiomics feature extraction, and machine learning.


Preprocessing


原 PPT 独立图片资源
Preprocessing



Preprocessing step receives the raw data, and prepare the samples for feature extraction using following steps:
Data selection: Filter relevant part of data;
Clean data: Removes low quality, inconsistent data;
Handles missing values: Ensures no critical information is lost.
Normalizes data: Brings data to a common scale for better convergence;
…

Feature Extraction


原 PPT 独立图片资源
Feature Extraction



Feature Extraction bridges the raw input data of the task to the learning model;
It generally convert text, sounds, images, videos into vector of features;
Methods:
Text: word index in vocabulary, word pos-tag catetory, TF-IDF vector of document;
Image: SIFT, Gabor;
Speech: Wavelet
…

Choose the Model Family


原 PPT 独立图片资源
Choose the Model Family



Model Families:
Linear Model, Decision Tree, Support Vector Machines, Neural Network
Key Considerations
Problem Type: Classification vs Regression.
Data: Size and complexity.
Resources: Simpler models for less computation.
Key Takeaway
Start with simple models (linear models, decision trees) and move to complex models (SVM, neural networks) if needed.

Model Training and Evaluation


原 PPT 独立图片资源
Model Training and Evaluation



Model Training
Training Data: Labeled data used to teach the model.
Loss Function: Measures model performance (e.g., MSE, Cross-Entropy).
Optimization: Adjusts parameters to minimize error (e.g., gradient descent).
Model Evaluation
Test Set: Used to evaluate the final model’s real-world performance.
Evaluation Metrics: Accuracy, Precision, Recall, F1 Score; RMSE.
Common Challenges
Overfitting: Model captures noise, mitigated by regularization.
Underfitting: Model too simple, mitigated by using more complex models.

0.3 Machine Learning Basics 0.3.3 Machine Learning Models



原 PPT 独立图片资源

0.3 Machine Learning Basics 0.3.3 Machine Learning Models

原 PPT 独立图片资源

Types of Machine Learning Systems


原 PPT 独立图片资源
Types of Machine Learning Systems

Supervised Learning
Mapping samples to given label system.
Unsupervised Learning
Explore and find inner-structure of data.
Reinforcement Learning
Learning from feedbacks from the environment.

Supervised Learning


原 PPT 独立图片资源
Supervised Learning
In supervised learning, the training data feed to the algorithm includes the desired output (labels).


Regression: when y is real.
Classification: when y is discrete.

原 PPT 独立图片资源

Supervised Learning


原 PPT 独立图片资源
Supervised Learning
原 PPT 独立图片资源

https://www.slideshare.net/slideshow/the-fundamentals-of-machine-learning/238653218

Supervised Learning


原 PPT 独立图片资源
Supervised Learning

k-Nearest Neighbors
Linear Regression
Logistic Regression
Support Vector Machines (SVMs)
Decision Trees and Random Forests
Feed-Forward Neural networks

Logistic Regression


原 PPT 独立图片资源
Logistic Regression

Decision Surface for Logistic Regression on a Binary Classification Task
https://machinelearningmastery.com/plot-a-decision-surface-for-machine-learning/
Binary classification by predicting the probability of 0 or 1.
Components:
Model:   𝑧=∑𝑤𝑖𝑥𝑖+𝑏;
Prediction: 𝑃(𝑦=1|𝑋) = 11+𝑒−𝑧;
Loss function: Cross-Entropy
Optimization: Gradient Descent

Unsupervised Learning


原 PPT 独立图片资源
Unsupervised Learning
In unsupervised learning, the training data is unlabeled. The system tries to learn without a teacher.


No labels are given to the learning algorithm, leaving it on its own to explore or find structure in the data.

原 PPT 独立图片资源

Unsupervised Learning


原 PPT 独立图片资源
Unsupervised Learning
原 PPT 独立图片资源

https://www.slideshare.net/slideshow/the-fundamentals-of-machine-learning/238653218

Unsupervised Learning


原 PPT 独立图片资源
Unsupervised Learning

Distribution Estimation:
MLE, VAE, Language Model;
Clustering:
K-Means, GMM, Affinity Propagation;
Dimensionality Reduction
PCA, ICA
Visualization
t-SNE

Gaussian Mixture Model


原 PPT 独立图片资源
Gaussian Mixture Model

原 PPT 独立图片资源
https://pomegranate.readthedocs.io/en/latest/tutorials/B_Model_Tutorial_2_General_Mixture_Models.html
Estimate the density of data by a mixture of several Gaussian distributions
Components:
Model: p(x)=∑𝑝𝑘𝒩(𝑥|𝜇𝑘,Σ𝑘);
Loss: negative log-likelihood 
Optimization: EM algorithm;
Application: Clustering, Anomaly Detection

Reinforcement Learning


原 PPT 独立图片资源
Reinforcement Learning
Agent learns to make decisions by interacting with an environment to maximize a reward signal over time.
Exploration vs. Exploitation:
Exploration refers to trying new actions to discover potentially better rewards.
Exploitation means choosing the action that gives the highest known reward based on past experience.
The agent interacts with the environment by taking actions, receiving rewards, and updating its policy to maximize future rewards.

Reinforcement Learning


原 PPT 独立图片资源
Reinforcement Learning
原 PPT 独立图片资源

https://www.slideshare.net/slideshow/the-fundamentals-of-machine-learning/238653218

0.3 Machine Learning Basics 0.3.4 Optimization Methods



原 PPT 独立图片资源

0.3 Machine Learning Basics 0.3.4 Optimization Methods

原 PPT 独立图片资源

Optimization for Machine Learning


原 PPT 独立图片资源
Optimization for Machine Learning
Optimization: find the best parameters 𝜃 for the machine  𝑀  that minimize the loss function (or cost function)  ℒ(𝑀(𝜃)) , where  𝑀(𝜃)  represents the machine’s predictions based on parameters  𝜃 .
min𝜃 ℒ(𝑀(𝜃))+𝜆 ℛ(𝜃)
Where:
ℛ(𝜃) is the regularization term (e.g., L1 or L2 regularization).
𝜆  is the regularization strength.

Gradient Decent


原 PPT 独立图片资源
Gradient Decent

Gradient descent
https://easyai.tech/en/ai-definition/gradient-descent/
Objective: Minimize loss function 𝐽(𝜃) by updating parameters 𝜃.
Update Rule:
𝜃 =𝜃 −𝛼 ∇𝜃 𝐽(𝜃)
Challenges:
Choosing Learning Rate.
Local Minima

Policy Gradient


原 PPT 独立图片资源
Policy Gradient
Sample trajectories;
Compute return  𝑅𝑡;
Compute the gradient:
∇𝜃 𝐽(𝜃) = 𝔼𝜋(𝜃)[ ∇𝜃 𝑙𝑜𝑔 𝜋𝜃(𝑎𝑡|𝑠𝑡)⋅𝑅𝑡]
Update the policy;
Repeat until convergence.

原 PPT 独立图片资源
https://jonathan-hui.medium.com/rl-policy-gradients-explained-9b13b688b146

Expectation Maximization


原 PPT 独立图片资源
Expectation Maximization
Goal: Maximize the likelihood 𝐿(𝜃 | 𝑋) =∑𝑍𝑃(𝑋, 𝑍 | 𝜃)  given latent variables 𝑍.
E-step (Expectation):
Estimate the latent variables 𝑃(𝑍 | 𝑋, 𝜃).
M-step (Maximization):
Update the parameters 𝜃 to maximize  𝔼[𝑙𝑜𝑔 𝑃(𝑋, 𝑍 | 𝜃)] .

undefined
https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm

0.3 Machine Learning Basics 0.3.5 Generalization and Model Selection



原 PPT 独立图片资源

0.3 Machine Learning Basics 0.3.5 Generalization and Model Selection

原 PPT 独立图片资源

Generalization, Underfitting and Overfitting


原 PPT 独立图片资源
Generalization, Underfitting and Overfitting
Generalization: The ability of a model to perform well on unseen data.
Underfitting: Model cannot capture the underlying patterns of the data.
Overfitting: Model learns the noise or random fluctuations in the training data, rather than the underlying patterns.

Degree 1 MSE = 4.08e-01(+/- 4.25e-01), Degree 4 MSE = 4.32e-02(+/- 7.08e-02), Degree 15 MSE = 1.82e+08(+/- 5.46e+08)
https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html

Generalization, Underfitting and Overfitting


原 PPT 独立图片资源
Generalization, Underfitting and Overfitting

https://vitalflux.com/overfitting-underfitting-concepts-interview-questions/
Overfitting and Underfitting in Machine Learning

Statistical Learning Setting


原 PPT 独立图片资源
Statistical Learning Setting
Given:
Data Distribution: (𝑋,𝑌)
Model: set ℱ of candidate predictors of the form 𝑓:𝒳↦𝒴
Loss function: 𝑙:𝒴×𝒴↦ℝ+
Goal: 𝑓∗=argmin𝑓∈ℱ⁡𝐸[𝑙(𝑌,𝑓(𝑋))]

Introduction to Statistical Learning Theory. J. Saketha Nath (IIT Bombay)

Statistical Learning Setting


原 PPT 独立图片资源
Statistical Learning Setting
Given a sampled training data set
𝐷={(𝑥1,𝑦1),(𝑥2,𝑦2), …,(𝑥𝑚,𝑦𝑚)}
Empirical Error: {1𝑚∑𝑖=1𝑚𝑙(𝑌𝑖,𝑓(𝑋𝑖))}𝑚=1∞ →     p       𝐸[𝑙(𝑌, 𝑓(𝑋))]
We can obtain a machine 𝑓^ by minimize the empirical error;
We study the statistical property of 𝑓^ given different sampling form data distributions.

Bias Variance Trade-off


原 PPT 独立图片资源
The bias is the difference between the expected value of the model’s prediction and the true value of the target


The variance measures how much the model’s prediction fluctuates when trained on different training sets.
Bias Variance Trade-off

https://scott.fortmann-roe.com/docs/BiasVariance.html
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Bias Variance Trade-off


原 PPT 独立图片资源
Bias Variance Trade-off

https://scott.fortmann-roe.com/docs/BiasVariance.html
原 PPT 独立图片资源

Regularization


原 PPT 独立图片资源
Regularization
Simpler Model
Smoother Model
Add Noise
Early Stop

An image that shows how regularization tries to reduce model flexibility in an effort to reduce and minimize validation error.
https://www.sharpsightlabs.com/blog/regularization-explained/

Model Selection


原 PPT 独立图片资源
Model Selection

原 PPT 独立图片资源
https://insights.blackcoffer.com/the-workflow-of-a-machine-learning-artificial-intelligence-project/
Resampling Methods: Cross-Validation

0.3 Machine Learning Basics 0.3.6 Suggested Readings



原 PPT 独立图片资源

0.3 Machine Learning Basics 0.3.6 Suggested Readings

原 PPT 独立图片资源

Books


原 PPT 独立图片资源
Books
The Elements of Statistical Learning: Data Mining, Inference, and Prediction.by Trevor Hastie, Robert Tibshirani, Jerome Friedman https://hastie.su.domains/ElemStatLearn/

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源

Natural Language UnderstandingDeep Learning Basic


原 PPT 独立图片资源
Natural Language
Understanding

Deep Learning Basic
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
History of Deep Learning
Why Deep Learning
Feed-forward Neural Networks and Back-Propagation
Convolution Neural Networks
Recurrent Neural Networks

0.4 Deep Learning Basics 0.4.1 History of Deep Learning



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.1 History of Deep Learning

原 PPT 独立图片资源

Neural Bionics


原 PPT 独立图片资源
Neural Bionics

M-P Neuron Model (1943, McCulloch & Pitts): The first formalized and systematized artificial neuron model. It can implement basic logic gates (such as AND, OR, and NOT gates), thereby demonstrating that neural networks can perform logical computations.
Lightbox
原 PPT 独立图片资源
Hebb’s learning rule (1949, Hebb): Inspired by the principle of synaptic plasticity observed in the human brain: “neurons that fire together, wire together.” The first learning rule for neural networks to solve classification problems.

Expectation, Despair, and Hope


原 PPT 独立图片资源
Expectation, Despair, and Hope
1958‌: Frank Rosenblatt invents the ‌Perceptron‌, the first trainable single-layer neural network.
‌1969‌: Minsky exposes the limitations of linear separability in perceptrons, triggering the ‌first AI winter‌.
‌1986‌: Hinton, Rumelhart, and others introduce ‌Backpropagation‌, enabling training of multi-layer networks.

‌Dormancy and Brewing


原 PPT 独立图片资源
‌Dormancy and Brewing
‌1989‌: Yann LeCun develops an early ‌Convolutional Neural Network (CNN)‌ for digit recognition.
‌1997‌: Hochreiter & Schmidhuber propose ‌LSTM‌, revolutionizing sequence modeling.
‌‌2006‌: Hinton’s ‌Deep Belief Networks (DBNs)‌ reignite interest in "deep learning.”
However, in this small-data phrase, SVM dominate this period.

Dominate the Domain AI


原 PPT 独立图片资源
Dominate the Domain AI
‌2012‌: ‌AlexNet‌ dominates ImageNet (error rate: 15.3%), sparking the deep learning boom.
‌2014‌: Ian Goodfellow invents ‌GANs‌; VGGNet and ‌GoogleNet‌ advance image classification.
‌2015‌: ‌ResNet‌ uses residual connections to train networks with 1,000+ layers (ImageNet error: 3.57%).
‌2017‌: The ‌Transformer‌ architecture (Vaswani et al.) reshapes NLP.

AGI


原 PPT 独立图片资源
AGI
‌2018‌: ‌BERT‌ (Google) redefines NLP; ‌GPT-1‌ pioneers pre-trained language models.
‌2020‌: ‌GPT-3‌ demonstrates few-shot learning (175B parameters).
‌2022‌: ‌ChatGPT‌ goes viral, Pretrain-SFT-RLHF as road map to AGI;
‌2023‌: ‌GPT-4‌ achieves near-human expertise;
‌2024‌: GPT-o1/DeepSeek-R1 introduce the RL based deep-think;
2025: Agents, Coding, Math, …

0.4 Deep Learning Basics 0.4.2 Why Deep Learning



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.2 Why Deep Learning

原 PPT 独立图片资源

Traditional Machine Learning


原 PPT 独立图片资源
Traditional Machine Learning
Data

Feature

Shallow Machine

Why Neural Nets Failed in Small Data Period?


原 PPT 独立图片资源
Why Neural Nets Failed in Small Data Period?
Small Data Period (Pre-2010s): Hundreds, at most Thousands samples
Recall the Bias-Variance Trade-off, complex machines will have large variance when sample size is small, thus is easy to overfit.
Shallow machines dominate this period.
Shallow NN is identical to kernel machines. Deep NN are too complex;
NN is sensitive to initialization.

原 PPT 独立图片资源
https://scott.fortmann-roe.com/docs/BiasVariance.html

How Hinton Solve this Problem


原 PPT 独立图片资源
How Hinton Solve this Problem

原 PPT 独立图片资源
1.Hinton, G. E. & Salakhutdinov, R. R. Reducing the Dimensionality of Data with Neural Networks. Science 313, 504–507 (2006).

Deep Learning


原 PPT 独立图片资源
Deep Learning
“Depth … is defined for circuits. A circuit is a directed acyclic graph where each node is associated with a computation, and whose output results are used by the successors of that node. … The depth of a circuit is the longest path from an input to an output node. ”
“…(Yao, 1985), showing that logic gates circuits of depth-2 require exponential size to implement d-bit parity where a deep circuit of depth O(log(d)) could implement it with O(d) nodes. H ̊astad (1986) then showed that there are functions computable with a polynomial-size logic gate circuit of depth k that require exponential size when restricted to depth k − 1 (Ha ̊stad, 1986).”
Think: what is the architecture depth of decision trees?

Bengio, Y. & Delalleau, O. On the Expressive Power of Deep Architectures.

Non-local Learning


原 PPT 独立图片资源
Non-local Learning
Local Learning
Learning depends only on local signals (e.g., Hebbian learning: synaptic updates rely only on the activity of connected neurons).
Pros: biologically plausible; Cons: hard to capture complex global dependencies.
Non-local Learning
Parameter updates require global or distant signals (e.g., backpropagation gradients).
Pros: captures long-range dependencies and global structure with higher-level abstraction.
Deep Learning is Non-Local Learning
Deep learning’s success partly relies on non-local mechanism across layers and modules.

Representation Learning


原 PPT 独立图片资源
Representation Learning
Deep Learning = Learning Multiple Levels of Representation
“The intent is to discover more abstract features in the higher levels of the representation, which hopefully make it easier to separate from each other the various explanatory factors extent in the data.”
Deep representations lie within the human brain and human knowledge structures, among other areas.

Bengio, Y. & Delalleau, O. On the Expressive Power of Deep Architectures.

End-to-End Learning


原 PPT 独立图片资源
End-to-End Learning

Zhilu Chen, Xinming Huang Published in IEEE Intelligent Vehicles… 11 June 2017
It connect every component to the final loss;
Thus make each hidden component more closely cooperative.
Figure 1 from End-to-end learning for lane keeping of self-driving cars |  Semantic Scholar

Case Study: Decision Tree


原 PPT 独立图片资源
Case Study: Decision Tree
Shallow Learner
In circuit terms, always equivalent to a depth-2 AND–OR structure.
Local Learner
Training relies on local greedy splits (e.g., information gain / Gini).
No global error optimization or cross-layer coordination.
Not Representation Learning
Merely partitions the input space into regions → outputs predictions.
Does not generate new feature representations or hierarchical abstractions.
Not End-to-End Learning
Learns by stage-wise tree construction, not by unified differentiable optimization.

原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.3 Feedforward Neural Networks and Backpropagation



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.3 Feedforward Neural Networks and Backpropagation

原 PPT 独立图片资源

Artificial Neuron


原 PPT 独立图片资源
Artificial Neuron

https://en.wikipedia.org/wiki/Artificial_neuron

原 PPT 独立图片资源

Activation Functions


原 PPT 独立图片资源
Activation Functions

Types of Activation Functions in Deep Learning | by Ravjot Singh |  DataDrivenInvestor
https://medium.datadriveninvestor.com/types-of-activation-functions-in-deep-learning-e7c2a48d3242

Feedforward Neural Network


原 PPT 独立图片资源
Feedforward Neural Network
Input Sample with fixed dimension (feature vector);
Output Target also with fixed dimension (probabilities of classes);

原 PPT 独立图片资源
https://medium.com/analytics-vidhya/linear-algebra-how-uses-in-artificial-intelligence-2e1e001c65

Back-Propagation


原 PPT 独立图片资源
Back-Propagation

https://analyticsarora.com/8-unique-machine-learning-interview-questions-on-backpropagation/


8 Unique Machine Learning Interview Questions on Backpropagation -  Analytics Arora
原 PPT 独立图片资源
https://www.3blue1brown.com/lessons/backpropagation-calculus

Residual Block and ResNet


原 PPT 独立图片资源
Residual Block and ResNet



原 PPT 独立图片资源
原 PPT 独立图片资源
Residual Block
ResNet 18
When networks goes deeper, their performances will degrade (not overfitting, as the training performance also drop);
ResNet extends the depth to 100-1000, with better performance.
https://d2l.ai/chapter_convolutional-modern/resnet.html

0.4 Deep Learning Basics 0.4.4 Convolutional Neural Networks



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.4 Convolutional Neural Networks

原 PPT 独立图片资源

From Fully Connected Layers to Convolutions


原 PPT 独立图片资源
From Fully Connected Layers to Convolutions



For images, fully connected layers are as follows:



Translation Invariance:

Locality:

Convolution:
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html

Proof in Vision


原 PPT 独立图片资源
Proof in Vision



原 PPT 独立图片资源
https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation

Convolution and Pooling Operation


原 PPT 独立图片资源
Convolution and Pooling Operation



原 PPT 独立图片资源
原 PPT 独立图片资源
convolution
pooling
https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation

Convolution Network (LeNet)


原 PPT 独立图片资源
Convolution Network (LeNet)



原 PPT 独立图片资源
https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation

Convolution Network (AlexNet)


原 PPT 独立图片资源
Convolution Network (AlexNet)



https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation
原 PPT 独立图片资源
LeNet



AlexNet

Convolution Network for Text


原 PPT 独立图片资源
Convolution Network for Text



https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation
原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.5 Recurrent Neural Networks



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.5 Recurrent Neural Networks

原 PPT 独立图片资源

Sequential Data


原 PPT 独立图片资源
Sequential Data
Neuron/Electron Signal;
Sound and voice;
Language;
Multi-step Action Procedure;

原 PPT 独立图片资源
原 PPT 独立图片资源

Coping with the Sequential Data (I)


原 PPT 独立图片资源
Coping with the Sequential Data (I)
Probabilistic Sequence Model:



Variant number of input tokens: 𝑤1,⋯, 𝑤𝑖−1.
To employee a feedforward neural model, we need a fix-size window;

原 PPT 独立图片资源
原 PPT 独立图片资源

Coping with the Sequential Data (II)


原 PPT 独立图片资源
Coping with the Sequential Data (II)
Simple and efficient;
But window cannot be arbitrarily large;
Forget any thing before the window;
We need new neural architecture.

原 PPT 独立图片资源

Recurrent Architecture


原 PPT 独立图片资源
Recurrent Architecture
Introduce inner-state to memorize history;
Update the inner-state when meet new token;
Make decision based on newest inner-state.
原 PPT 独立图片资源
原 PPT 独立图片资源

Language as an example


原 PPT 独立图片资源
Language as an example
原 PPT 独立图片资源

Capacity


原 PPT 独立图片资源
Capacity
If training vanilla neural nets is optimization over functions, training recurrent nets is optimization over programs.
RNNs are Turing-Complete in the sense that they can simulate arbitrary programs (with proper weights).

Training


原 PPT 独立图片资源
Training
Calculate the loss of output to the ground truth;
For sequence generation task, calculate loss at each generation and sum up;


Apply gradient-descend to optimize the parameters using BPTT;
For large dataset, stochastic gradient descend (SGD) should be applied;
Time consuming since the dependencies between hidden-state (cannot parallelize along generation step T);
原 PPT 独立图片资源

Backpropagation Through Time


原 PPT 独立图片资源
Backpropagation Through Time
Apply the Backpropagation along the sequence (time dimension).
The error of each output step should be backpropagated through the sequence.
Details will be discussed later.
原 PPT 独立图片资源

Vanila RNN


原 PPT 独立图片资源
Vanila RNN
ℎ𝑡=σ(𝑊ℎℎℎ𝑡−1+𝑊ℎ𝑥𝑥𝑡+𝑏)
Inner-State: a fix-dimensional vector;
State-Update: Activation of linear combination of inner-state and new input;

Details in Gradient Computing


原 PPT 独立图片资源
Details in Gradient Computing
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Gradient Vanish/Exploding


原 PPT 独立图片资源
Gradient Vanish/Exploding
The gradient involves potentially very large powers of 𝑊ℎℎ𝑇.
If its eigenvalues smaller than 1, the gradient vanishes;
If its eigenvalues larger than 1, the gradient explodes/diverges.
The essential problem of vanilla RNN: numerically unstable.

Long Short-Term Memory (LSTM)


原 PPT 独立图片资源
Long Short-Term Memory (LSTM)
Intuition: Control the long-term dependency:
when the hidden state should be updated and when it should be reset;
Gated Memory Cell:  with an internal state and a number of multiplicative gates that determine whether:
a given input should impact the internal state (the input gate);
the internal state should be flushed to 0 (the forget gate);
the internal state should be allowed to impact the cell’s output (the output gate).

LSTM: (1) The Gates


原 PPT 独立图片资源
LSTM: (1) The Gates
原 PPT 独立图片资源
原 PPT 独立图片资源

LSTM: (2) The Input Node


原 PPT 独立图片资源
LSTM: (2) The Input Node
原 PPT 独立图片资源
原 PPT 独立图片资源

LSTM: (3) Memory Cell Internal State


原 PPT 独立图片资源
LSTM: (3) Memory Cell Internal State
原 PPT 独立图片资源
原 PPT 独立图片资源

LSTM : (4) Hidden State


原 PPT 独立图片资源
LSTM : (4) Hidden State
原 PPT 独立图片资源
原 PPT 独立图片资源

Gated Recurrent Unit (GRU)


原 PPT 独立图片资源
Gated Recurrent Unit (GRU)
GRU: A simplified LSTM with faster computation.
Two Gates: reset gate, and update gate;
Only one state;

GRU: (1) Gates


原 PPT 独立图片资源
GRU: (1) Gates
原 PPT 独立图片资源
原 PPT 独立图片资源

GRU: (2) Candidate Hidden State


原 PPT 独立图片资源
GRU: (2) Candidate Hidden State
原 PPT 独立图片资源
原 PPT 独立图片资源

GRU: (3) Hidden State


原 PPT 独立图片资源
GRU: (3) Hidden State
原 PPT 独立图片资源
原 PPT 独立图片资源

Summary


原 PPT 独立图片资源
Summary
What should we do to make the gates works as expected?
Nothing. We just design the mechanism, enlarge the capacity, then feed the data. Machine will learn how to behave well.
原 PPT 独立图片资源
原 PPT 独立图片资源

Input-Output Paradigms


原 PPT 独立图片资源
Input-Output Paradigms
原 PPT 独立图片资源

Deep RNN


原 PPT 独立图片资源
Deep RNN
Stack RNN layers;
Each layer take hidden-states of previous-layer as input;
Can memorize a lot, analyze deeply, and accomplish difficult task:
Large amount of hidden-states;
Deep hidden-states;
原 PPT 独立图片资源

Bidirectional RNN


原 PPT 独立图片资源
Bidirectional RNN
Uni-directional RNN know history and generate future;
Bidirectional RNN know both history and future and finish tasks:
Fill the blank;
Tagging the items of the sequence;
Predict attributes of the sequence.
原 PPT 独立图片资源

Encoder-Decoder


原 PPT 独立图片资源
Encoder-Decoder
Based on Representation Learning: anything can be represented by embedding vectors;
Encoder: takes a variable-length sequence as input produce embedding vectors as the representations of that sequence;
Decoder generate the output from the representative embedding vectors.
原 PPT 独立图片资源

Sequence Generation


原 PPT 独立图片资源
Sequence Generation
Given input representative embeddings 𝑐, consider the problem of generating a sequence of items {𝑦1,⋯,𝑦𝑇} from a vocabulary 𝒴.


Finding the sequence maximizing above probability would have a unacceptable cost .
We must fall back on an efficient solution.
原 PPT 独立图片资源
原 PPT 独立图片资源

Greedy Search


原 PPT 独立图片资源
Greedy Search
At each step, select the token with the highest conditional probability;
Simple but sub-optimal;
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Beam Search


原 PPT 独立图片资源
Beam Search
At each step, consider 𝑘 candidates with the top conditional sequence probability;
原 PPT 独立图片资源

Applications: Sequence Classification


原 PPT 独立图片资源
Applications: Sequence Classification
Feed the last hidden-state to the classifier;
Widely used in NLP task such as text classification;
Text classification model based on recurrent neural network.

Applications: Sequence Labeling


原 PPT 独立图片资源
Applications: Sequence Labeling
Predict a label for each item of input sequence.
Widely used in NLP task like NER;
bilstm

Applications: Sequence Prediction


原 PPT 独立图片资源
Applications: Sequence Prediction
Modeling the problem as a sequence prediction task;
Typical application: Machine Translation, AGI LLM;
原 PPT 独立图片资源
原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.6 Suggested Readings



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.6 Suggested Readings

原 PPT 独立图片资源

Books


原 PPT 独立图片资源
Books
Deep Learning, Ian Goodfellow and Yoshua Bengio and Aaron Courville https://www.deeplearningbook.org/

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源