Natural Language UnderstandingDeep Learning Basic


原 PPT 独立图片资源
Natural Language
Understanding

Deep Learning Basic
Mingming Sun
BIMSA

Outline


原 PPT 独立图片资源
Outline
History of Deep Learning
Why Deep Learning
Feed-forward Neural Networks and Back-Propagation
Convolution Neural Networks
Recurrent Neural Networks

0.4 Deep Learning Basics 0.4.1 History of Deep Learning



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.1 History of Deep Learning

原 PPT 独立图片资源

Neural Bionics


原 PPT 独立图片资源
Neural Bionics

M-P Neuron Model (1943, McCulloch & Pitts): The first formalized and systematized artificial neuron model. It can implement basic logic gates (such as AND, OR, and NOT gates), thereby demonstrating that neural networks can perform logical computations.
Lightbox
原 PPT 独立图片资源
Hebb’s learning rule (1949, Hebb): Inspired by the principle of synaptic plasticity observed in the human brain: “neurons that fire together, wire together.” The first learning rule for neural networks to solve classification problems.

Expectation, Despair, and Hope


原 PPT 独立图片资源
Expectation, Despair, and Hope
1958‌: Frank Rosenblatt invents the ‌Perceptron‌, the first trainable single-layer neural network.
‌1969‌: Minsky exposes the limitations of linear separability in perceptrons, triggering the ‌first AI winter‌.
‌1986‌: Hinton, Rumelhart, and others introduce ‌Backpropagation‌, enabling training of multi-layer networks.

‌Dormancy and Brewing


原 PPT 独立图片资源
‌Dormancy and Brewing
‌1989‌: Yann LeCun develops an early ‌Convolutional Neural Network (CNN)‌ for digit recognition.
‌1997‌: Hochreiter & Schmidhuber propose ‌LSTM‌, revolutionizing sequence modeling.
‌‌2006‌: Hinton’s ‌Deep Belief Networks (DBNs)‌ reignite interest in "deep learning.”
However, in this small-data phrase, SVM dominate this period.

Dominate the Domain AI


原 PPT 独立图片资源
Dominate the Domain AI
‌2012‌: ‌AlexNet‌ dominates ImageNet (error rate: 15.3%), sparking the deep learning boom.
‌2014‌: Ian Goodfellow invents ‌GANs‌; VGGNet and ‌GoogleNet‌ advance image classification.
‌2015‌: ‌ResNet‌ uses residual connections to train networks with 1,000+ layers (ImageNet error: 3.57%).
‌2017‌: The ‌Transformer‌ architecture (Vaswani et al.) reshapes NLP.

AGI


原 PPT 独立图片资源
AGI
‌2018‌: ‌BERT‌ (Google) redefines NLP; ‌GPT-1‌ pioneers pre-trained language models.
‌2020‌: ‌GPT-3‌ demonstrates few-shot learning (175B parameters).
‌2022‌: ‌ChatGPT‌ goes viral, Pretrain-SFT-RLHF as road map to AGI;
‌2023‌: ‌GPT-4‌ achieves near-human expertise;
‌2024‌: GPT-o1/DeepSeek-R1 introduce the RL based deep-think;
2025: Agents, Coding, Math, …

0.4 Deep Learning Basics 0.4.2 Why Deep Learning



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.2 Why Deep Learning

原 PPT 独立图片资源

Traditional Machine Learning


原 PPT 独立图片资源
Traditional Machine Learning
Data

Feature

Shallow Machine

Why Neural Nets Failed in Small Data Period?


原 PPT 独立图片资源
Why Neural Nets Failed in Small Data Period?
Small Data Period (Pre-2010s): Hundreds, at most Thousands samples
Recall the Bias-Variance Trade-off, complex machines will have large variance when sample size is small, thus is easy to overfit.
Shallow machines dominate this period.
Shallow NN is identical to kernel machines. Deep NN are too complex;
NN is sensitive to initialization.

原 PPT 独立图片资源
https://scott.fortmann-roe.com/docs/BiasVariance.html

How Hinton Solve this Problem


原 PPT 独立图片资源
How Hinton Solve this Problem

原 PPT 独立图片资源
1.Hinton, G. E. & Salakhutdinov, R. R. Reducing the Dimensionality of Data with Neural Networks. Science 313, 504–507 (2006).

Deep Learning


原 PPT 独立图片资源
Deep Learning
“Depth … is defined for circuits. A circuit is a directed acyclic graph where each node is associated with a computation, and whose output results are used by the successors of that node. … The depth of a circuit is the longest path from an input to an output node. ”
“…(Yao, 1985), showing that logic gates circuits of depth-2 require exponential size to implement d-bit parity where a deep circuit of depth O(log(d)) could implement it with O(d) nodes. H ̊astad (1986) then showed that there are functions computable with a polynomial-size logic gate circuit of depth k that require exponential size when restricted to depth k − 1 (Ha ̊stad, 1986).”
Think: what is the architecture depth of decision trees?

Bengio, Y. & Delalleau, O. On the Expressive Power of Deep Architectures.

Non-local Learning


原 PPT 独立图片资源
Non-local Learning
Local Learning
Learning depends only on local signals (e.g., Hebbian learning: synaptic updates rely only on the activity of connected neurons).
Pros: biologically plausible; Cons: hard to capture complex global dependencies.
Non-local Learning
Parameter updates require global or distant signals (e.g., backpropagation gradients).
Pros: captures long-range dependencies and global structure with higher-level abstraction.
Deep Learning is Non-Local Learning
Deep learning’s success partly relies on non-local mechanism across layers and modules.

Representation Learning


原 PPT 独立图片资源
Representation Learning
Deep Learning = Learning Multiple Levels of Representation
“The intent is to discover more abstract features in the higher levels of the representation, which hopefully make it easier to separate from each other the various explanatory factors extent in the data.”
Deep representations lie within the human brain and human knowledge structures, among other areas.

Bengio, Y. & Delalleau, O. On the Expressive Power of Deep Architectures.

End-to-End Learning


原 PPT 独立图片资源
End-to-End Learning

Zhilu Chen, Xinming Huang Published in IEEE Intelligent Vehicles… 11 June 2017
It connect every component to the final loss;
Thus make each hidden component more closely cooperative.
Figure 1 from End-to-end learning for lane keeping of self-driving cars |  Semantic Scholar

Case Study: Decision Tree


原 PPT 独立图片资源
Case Study: Decision Tree
Shallow Learner
In circuit terms, always equivalent to a depth-2 AND–OR structure.
Local Learner
Training relies on local greedy splits (e.g., information gain / Gini).
No global error optimization or cross-layer coordination.
Not Representation Learning
Merely partitions the input space into regions → outputs predictions.
Does not generate new feature representations or hierarchical abstractions.
Not End-to-End Learning
Learns by stage-wise tree construction, not by unified differentiable optimization.

原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.3 Feedforward Neural Networks and Backpropagation



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.3 Feedforward Neural Networks and Backpropagation

原 PPT 独立图片资源

Artificial Neuron


原 PPT 独立图片资源
Artificial Neuron

https://en.wikipedia.org/wiki/Artificial_neuron

原 PPT 独立图片资源

Activation Functions


原 PPT 独立图片资源
Activation Functions

Types of Activation Functions in Deep Learning | by Ravjot Singh |  DataDrivenInvestor
https://medium.datadriveninvestor.com/types-of-activation-functions-in-deep-learning-e7c2a48d3242

Feedforward Neural Network


原 PPT 独立图片资源
Feedforward Neural Network
Input Sample with fixed dimension (feature vector);
Output Target also with fixed dimension (probabilities of classes);

原 PPT 独立图片资源
https://medium.com/analytics-vidhya/linear-algebra-how-uses-in-artificial-intelligence-2e1e001c65

Back-Propagation


原 PPT 独立图片资源
Back-Propagation

https://analyticsarora.com/8-unique-machine-learning-interview-questions-on-backpropagation/


8 Unique Machine Learning Interview Questions on Backpropagation -  Analytics Arora
原 PPT 独立图片资源
https://www.3blue1brown.com/lessons/backpropagation-calculus

Residual Block and ResNet


原 PPT 独立图片资源
Residual Block and ResNet



原 PPT 独立图片资源
原 PPT 独立图片资源
Residual Block
ResNet 18
When networks goes deeper, their performances will degrade (not overfitting, as the training performance also drop);
ResNet extends the depth to 100-1000, with better performance.
https://d2l.ai/chapter_convolutional-modern/resnet.html

0.4 Deep Learning Basics 0.4.4 Convolutional Neural Networks



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.4 Convolutional Neural Networks

原 PPT 独立图片资源

From Fully Connected Layers to Convolutions


原 PPT 独立图片资源
From Fully Connected Layers to Convolutions



For images, fully connected layers are as follows:



Translation Invariance:

Locality:

Convolution:
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html

Proof in Vision


原 PPT 独立图片资源
Proof in Vision



原 PPT 独立图片资源
https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation

Convolution and Pooling Operation


原 PPT 独立图片资源
Convolution and Pooling Operation



原 PPT 独立图片资源
原 PPT 独立图片资源
convolution
pooling
https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation

Convolution Network (LeNet)


原 PPT 独立图片资源
Convolution Network (LeNet)



原 PPT 独立图片资源
https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation

Convolution Network (AlexNet)


原 PPT 独立图片资源
Convolution Network (AlexNet)



https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation
原 PPT 独立图片资源
LeNet



AlexNet

Convolution Network for Text


原 PPT 独立图片资源
Convolution Network for Text



https://d2l.ai/chapter_convolutional-neural-networks/conv-layer.html#fig-correlation
原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.5 Recurrent Neural Networks



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.5 Recurrent Neural Networks

原 PPT 独立图片资源

Sequential Data


原 PPT 独立图片资源
Sequential Data
Neuron/Electron Signal;
Sound and voice;
Language;
Multi-step Action Procedure;

原 PPT 独立图片资源
原 PPT 独立图片资源

Coping with the Sequential Data (I)


原 PPT 独立图片资源
Coping with the Sequential Data (I)
Probabilistic Sequence Model:



Variant number of input tokens: 𝑤1,⋯, 𝑤𝑖−1.
To employee a feedforward neural model, we need a fix-size window;

原 PPT 独立图片资源
原 PPT 独立图片资源

Coping with the Sequential Data (II)


原 PPT 独立图片资源
Coping with the Sequential Data (II)
Simple and efficient;
But window cannot be arbitrarily large;
Forget any thing before the window;
We need new neural architecture.

原 PPT 独立图片资源

Recurrent Architecture


原 PPT 独立图片资源
Recurrent Architecture
Introduce inner-state to memorize history;
Update the inner-state when meet new token;
Make decision based on newest inner-state.
原 PPT 独立图片资源
原 PPT 独立图片资源

Language as an example


原 PPT 独立图片资源
Language as an example
原 PPT 独立图片资源

Capacity


原 PPT 独立图片资源
Capacity
If training vanilla neural nets is optimization over functions, training recurrent nets is optimization over programs.
RNNs are Turing-Complete in the sense that they can simulate arbitrary programs (with proper weights).

Training


原 PPT 独立图片资源
Training
Calculate the loss of output to the ground truth;
For sequence generation task, calculate loss at each generation and sum up;


Apply gradient-descend to optimize the parameters using BPTT;
For large dataset, stochastic gradient descend (SGD) should be applied;
Time consuming since the dependencies between hidden-state (cannot parallelize along generation step T);
原 PPT 独立图片资源

Backpropagation Through Time


原 PPT 独立图片资源
Backpropagation Through Time
Apply the Backpropagation along the sequence (time dimension).
The error of each output step should be backpropagated through the sequence.
Details will be discussed later.
原 PPT 独立图片资源

Vanila RNN


原 PPT 独立图片资源
Vanila RNN
ℎ𝑡=σ(𝑊ℎℎℎ𝑡−1+𝑊ℎ𝑥𝑥𝑡+𝑏)
Inner-State: a fix-dimensional vector;
State-Update: Activation of linear combination of inner-state and new input;

Details in Gradient Computing


原 PPT 独立图片资源
Details in Gradient Computing
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Gradient Vanish/Exploding


原 PPT 独立图片资源
Gradient Vanish/Exploding
The gradient involves potentially very large powers of 𝑊ℎℎ𝑇.
If its eigenvalues smaller than 1, the gradient vanishes;
If its eigenvalues larger than 1, the gradient explodes/diverges.
The essential problem of vanilla RNN: numerically unstable.

Long Short-Term Memory (LSTM)


原 PPT 独立图片资源
Long Short-Term Memory (LSTM)
Intuition: Control the long-term dependency:
when the hidden state should be updated and when it should be reset;
Gated Memory Cell:  with an internal state and a number of multiplicative gates that determine whether:
a given input should impact the internal state (the input gate);
the internal state should be flushed to 0 (the forget gate);
the internal state should be allowed to impact the cell’s output (the output gate).

LSTM: (1) The Gates


原 PPT 独立图片资源
LSTM: (1) The Gates
原 PPT 独立图片资源
原 PPT 独立图片资源

LSTM: (2) The Input Node


原 PPT 独立图片资源
LSTM: (2) The Input Node
原 PPT 独立图片资源
原 PPT 独立图片资源

LSTM: (3) Memory Cell Internal State


原 PPT 独立图片资源
LSTM: (3) Memory Cell Internal State
原 PPT 独立图片资源
原 PPT 独立图片资源

LSTM : (4) Hidden State


原 PPT 独立图片资源
LSTM : (4) Hidden State
原 PPT 独立图片资源
原 PPT 独立图片资源

Gated Recurrent Unit (GRU)


原 PPT 独立图片资源
Gated Recurrent Unit (GRU)
GRU: A simplified LSTM with faster computation.
Two Gates: reset gate, and update gate;
Only one state;

GRU: (1) Gates


原 PPT 独立图片资源
GRU: (1) Gates
原 PPT 独立图片资源
原 PPT 独立图片资源

GRU: (2) Candidate Hidden State


原 PPT 独立图片资源
GRU: (2) Candidate Hidden State
原 PPT 独立图片资源
原 PPT 独立图片资源

GRU: (3) Hidden State


原 PPT 独立图片资源
GRU: (3) Hidden State
原 PPT 独立图片资源
原 PPT 独立图片资源

Summary


原 PPT 独立图片资源
Summary
What should we do to make the gates works as expected?
Nothing. We just design the mechanism, enlarge the capacity, then feed the data. Machine will learn how to behave well.
原 PPT 独立图片资源
原 PPT 独立图片资源

Input-Output Paradigms


原 PPT 独立图片资源
Input-Output Paradigms
原 PPT 独立图片资源

Deep RNN


原 PPT 独立图片资源
Deep RNN
Stack RNN layers;
Each layer take hidden-states of previous-layer as input;
Can memorize a lot, analyze deeply, and accomplish difficult task:
Large amount of hidden-states;
Deep hidden-states;
原 PPT 独立图片资源

Bidirectional RNN


原 PPT 独立图片资源
Bidirectional RNN
Uni-directional RNN know history and generate future;
Bidirectional RNN know both history and future and finish tasks:
Fill the blank;
Tagging the items of the sequence;
Predict attributes of the sequence.
原 PPT 独立图片资源

Encoder-Decoder


原 PPT 独立图片资源
Encoder-Decoder
Based on Representation Learning: anything can be represented by embedding vectors;
Encoder: takes a variable-length sequence as input produce embedding vectors as the representations of that sequence;
Decoder generate the output from the representative embedding vectors.
原 PPT 独立图片资源

Sequence Generation


原 PPT 独立图片资源
Sequence Generation
Given input representative embeddings 𝑐, consider the problem of generating a sequence of items {𝑦1,⋯,𝑦𝑇} from a vocabulary 𝒴.


Finding the sequence maximizing above probability would have a unacceptable cost .
We must fall back on an efficient solution.
原 PPT 独立图片资源
原 PPT 独立图片资源

Greedy Search


原 PPT 独立图片资源
Greedy Search
At each step, select the token with the highest conditional probability;
Simple but sub-optimal;
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Beam Search


原 PPT 独立图片资源
Beam Search
At each step, consider 𝑘 candidates with the top conditional sequence probability;
原 PPT 独立图片资源

Applications: Sequence Classification


原 PPT 独立图片资源
Applications: Sequence Classification
Feed the last hidden-state to the classifier;
Widely used in NLP task such as text classification;
Text classification model based on recurrent neural network.

Applications: Sequence Labeling


原 PPT 独立图片资源
Applications: Sequence Labeling
Predict a label for each item of input sequence.
Widely used in NLP task like NER;
bilstm

Applications: Sequence Prediction


原 PPT 独立图片资源
Applications: Sequence Prediction
Modeling the problem as a sequence prediction task;
Typical application: Machine Translation, AGI LLM;
原 PPT 独立图片资源
原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.6 Suggested Readings



原 PPT 独立图片资源

0.4 Deep Learning Basics 0.4.6 Suggested Readings

原 PPT 独立图片资源

Books


原 PPT 独立图片资源
Books
Deep Learning, Ian Goodfellow and Yoshua Bengio and Aaron Courville https://www.deeplearningbook.org/

Any Questions?



原 PPT 独立图片资源
Any Questions?
原 PPT 独立图片资源