Intelligent System


原 PPT 独立图片资源
Natural Language Understanding
Intelligent System
Mingming Sun
BIMSA

Intelligent System: Capabilities


原 PPT 独立图片资源
Intelligent System: Capabilities
4.1 System Architecture
4.2 Perception and Generation
4.3 Memory and Knowledge
4.4 Action and Execution
4.5 Planning and Control
4.6 Coordination and Collaboration
4.7 Learning and Evaluation

4.1 System Architecture 4.1.1 General-Purpose AI



原 PPT 独立图片资源

4.1 System Architecture 4.1.1 General-Purpose AI

原 PPT 独立图片资源

Artificial General Intelligence


原 PPT 独立图片资源
Artificial General Intelligence
Traditional AI: Solving specific problems in specific domain
General-purpose AI: transfer across tasks and domains
Capability dimensions, not an AGI test:
All-Tasks of one Modal such as Language in one Model
All-Modal in one Model

AGI: All-Tasks in One Modal


原 PPT 独立图片资源
AGI: All-Tasks in One Modal
Task Specific Model:
Input Data → Model → Prediction
Just need to understand the mapping between data;
The model does not need to understand the task.
AGI Model:
Input Data + Task Description → Model → Prediction
Also need to understand the task.

AGI: All-Modal in One Model


原 PPT 独立图片资源
AGI: All-Modal in One Model
Modal Specific Model:
Language: RNN/Transformer
Vision: CNN/Diffusion
Speech: HMM
Smell: ?
AGI Model:
Data in all modal → Model → Prediction
Mixed input of multi-modal data;
Unified model architecture.

4.1 System Architecture 4.1.2 Language Models as Components



原 PPT 独立图片资源

4.1 System Architecture 4.1.2 Language Models as Components

原 PPT 独立图片资源

Language Model Capabilities


原 PPT 独立图片资源
Language Model Capabilities
Understand the Domain Input -- Language
Understand the Task -- Instructions
Output High Quality Response -- Human Preference

Learning Language Model Capabilities


原 PPT 独立图片资源
Learning Language Model Capabilities
Pretraining: Understand the Language
Instruction Fine-tuning: Understand the Task
Reinforcement Learning with Human Feedback: Align with Human Preference

4.1 System Architecture 4.1.3 Agents and System Architecture



原 PPT 独立图片资源

4.1 System Architecture 4.1.3 Agents and System Architecture

原 PPT 独立图片资源

Intelligent System


原 PPT 独立图片资源
Intelligent System
Till now, we talked about neural models.
An intelligent system also needs explicit task state:
Retain goals, observations, actions and outcomes;
Intelligent need to interact with outer-world:
Human can explore external knowledge;
Human can use existing tools;
Human can write code to control the objects in the world.
Real system/organizations involves interaction of multiple intelligent individuals.

Agent!


原 PPT 独立图片资源
Agent!
Long-term goal of AI
Agent models connect observations, goals and actions.
“An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.” -- Russell & Norvig
An agent selects actions using observations and state;
An intelligent system may use one or multiple agents.

Agent


原 PPT 独立图片资源
Agent
https://langchain-ai.github.io/langgraph/tutorials/workflows/#agent
原 PPT 独立图片资源

Why Agents today?


原 PPT 独立图片资源
Why Agents today?
The unprecedented reasoning capabilities of large language models;
Progress in tool manipulation and environmental interaction;
Sophisticated memory architectures that support long-term experience accumulation.
Through collaboration, competition, and evolutionary dynamics, multi-agent systems can tackle highly complex tasks.

Perception and Generation


原 PPT 独立图片资源
Natural Language Understanding
Perception and Generation
Mingming Sun
BIMSA

Multi-Modal Understanding and Generation


原 PPT 独立图片资源
Understanding and Generation
4.2.1 Multimodality
4.2.2 Modal Representation
4.2.3 Multi-Modal Fusion
4.2.4 Modal Generation
4.2.5 Case Study: Nano Banana

4.2 Perception and Generation 4.2.1 Multimodality



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.1 Multimodality

原 PPT 独立图片资源

What is a Modality


原 PPT 独立图片资源
What is a Modality
A modality refers to a form of information or sensory channel — such as text, image, audio, video, or touch.
A multimodal system involves multiple types of data modalities, and is capable of jointly perceiving, understanding, and processing them.
Goal of multimodal models: to simulate how humans use multiple senses to understand the world holistically.
原 PPT 独立图片资源
https://www.appen.com/multi-modal-ai

Why Do We Need Multimodal Models?


原 PPT 独立图片资源
Why Do We Need Multimodal Models?
People naturally use a combination of vision, speech, and language to communicate.
Real-world inputs are inherently multimodal
Autonomous driving: Combines camera images, radar data, and GPS
Healthcare: Interpreting scans + doctor notes + patient speech
Customer service bots: Understand user voice, detect emotion, display visuals
Virtual assistants (e.g., Siri, Alexa): Use voice, text, context

Why Do We Need Multimodal Models?


原 PPT 独立图片资源
Why Do We Need Multimodal Models?
Limitations of Single-Modality Models
Information Loss: Relying on a single modality (e.g., text only or image only) can lead to incomplete understanding.
Example: A caption “A man is holding something” lacks detail if the image isn’t considered.
Ambiguity: Single-modality input often leads to uncertainty or confusion.
Example: The spoken word “bank” could mean a financial institution or a riverbank — image context can resolve this.

Evolution of Multimodal Models


原 PPT 独立图片资源
Evolution of Multimodal Models
Pre-AGI times: Mapping between different modals for specific task
The principle of audio-visual speech recognition. 
https://www.researchgate.net/figure/The-principle-of-audio-visual-speech-recognition_fig1_261268684
Showcasing the location of Metatags from the Open Graph Protocol
https://programmingpercy.tech/blog/opengraph-protocol-how-and-why/

Evolution of Multimodal Models


原 PPT 独立图片资源
Evolution of Multimodal Models
Nowadays
https://www.youtube.com/watch?v=2PShpVgQo2M

How to Achieve Multimodal Understanding?


原 PPT 独立图片资源
How to Achieve Multimodal Understanding?
Modal Representation
Feature Extraction
Embedding
Across all Modals
Muti-Modal Fusion
Early Fusion
Late Fusion
Joint Fusion
Modal Generation
Generative Model
Across all Models

4.2 Perception and Generation 4.2.2 Modal Representation



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.2 Modal Representation

原 PPT 独立图片资源

Single Modal Representation: Language


原 PPT 独立图片资源
Single Modal Representation: Language
Word2Vec
BERT
原 PPT 独立图片资源
(Mikolov et al. 2013)
原 PPT 独立图片资源
https://medium.com/data-science/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270

Single Modal Representation: Vision


原 PPT 独立图片资源
Single Modal Representation: Vision
CNNs: LeNet, Alex Net, ZFNet, GoogLeNet/Inception, VGG Net, ResNet
AlexNet architecture used as the baseline model for the analysis of results on the CIFAR-C data set. Light yellow indicates a convolution, while darker yellow is a ReLU function. Similarly, light
https://www.researchgate.net/figure/AlexNet-architecture-used-as-the-baseline-model-for-the-analysis-of-results-on-the_fig5_339756908
原 PPT 独立图片资源
https://medium.com/analytics-vidhya/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5
原 PPT 独立图片资源
https://medium.com/analytics-vidhya/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5

Single Modal Representation: Vision


原 PPT 独立图片资源
Single Modal Representation: Vision
Vision Transformer: ViT
https://paperswithcode.com/method/vision-transformer
原 PPT 独立图片资源

Single Modal Representation: Voice


原 PPT 独立图片资源
Single Modal Representation: Voice
Wave2Vec
BERT
https://arxiv.org/pdf/1904.05862
https://sh-tsang.medium.com/brief-review-hubert-self-supervised-speech-representation-learning-by-masked-prediction-of-e8baf7909c4a
原 PPT 独立图片资源
原 PPT 独立图片资源

Cross Modal Representation


原 PPT 独立图片资源
Cross Modal Representation
Learn a shared semantic space for different modalities (e.g., image, text).
Each modality is independently encoded into vectors, but aligned through training.
Bridging Modalities: A Survey of Cross-Modal Image-Text Retrieval
https://www.iece.org/article/abs/cjif.2024.361895

Why Align Modalities


原 PPT 独立图片资源
Why Align Modalities
Different modalities describe the same concept (e.g., a cat in text and image).
We want:

Aligned representations enable cross-modal retrieval, zero-shot learning, etc.
原 PPT 独立图片资源

Contrastive Learning: The Engine Behind


原 PPT 独立图片资源
Contrastive Learning: The Engine Behind
Sample positive image-text pairs (𝐼, 𝑇+) , and negative pairs (𝐼, 𝑇−).
Use contrastive loss:


Contrastive loss is a special form of cross-entropy loss, treating the matching pair as the positive class and all others as negatives in a softmax over the batch.
Objective: pull positive pairs together, push apart negative pairs.
原 PPT 独立图片资源

CLIP: Large-Scale Cross-Modal Training


原 PPT 独立图片资源
CLIP: Large-Scale Cross-Modal Training
Proposed by OpenAI (2021); trained on 400M image-text pairs.
Architecture:
Image encoder: Vision Transformer (ViT)
Text encoder: Transformer
Trained via symmetric contrastive loss.
Supports zero-shot classification via prompt-based text queries.

CLIP: Large-Scale Cross-Modal Training


原 PPT 独立图片资源
CLIP: Large-Scale Cross-Modal Training
原 PPT 独立图片资源
https://openai.com/index/clip/

ALIGN: Scaling Contrastive Learning


原 PPT 独立图片资源
ALIGN: Scaling Contrastive Learning
Google Research, 2021
Trained on 1.8B noisy image-text pairs from the web.
Simpler than CLIP:
Image: EfficientNet
Text: BERT
No symmetric loss; only image-to-text direction.
Relies on data scale + simple architecture.
Insight: Even noisy web-scale data works if quantity is large enough.

Key Ideas in Cross-Modal Contrastive Representation


原 PPT 独立图片资源
Key Ideas in Cross-Modal Contrastive Representation
Cross-modal representation enables:
Retrieval
Zero-shot learning
Prompt-based classification
Core ingredients:
Independent encoders
Contrastive objective
Large paired datasets
CLIP and ALIGN show the power of contrastive learning + scale.
But we need more thorough interaction between modals.

4.2 Perception and Generation 4.2.3 Multi-Modal Fusion



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.3 Multi-Modal Fusion

原 PPT 独立图片资源

Understanding Modality Fusion


原 PPT 独立图片资源
Understanding Modality Fusion
Definition: the process of integrating information from multiple modalities (e.g., image + text) to perform a task.
Objective:
Capture inter-modal interaction;
Enable joint reasoning and decision making.
Output:
Unified representation;
Task-specific predictions (e.g., answer, caption, class).
原 PPT 独立图片资源
https://medium.com/@raj.pulapakura/multimodal-models-and-fusion-a-complete-guide-225ca91f6861
原 PPT 独立图片资源

Types of Modality Fusion


原 PPT 独立图片资源
Types of Modality Fusion
Type
Description
Example
Early Fusion
Concatenate raw inputs or early features
Multimodal embeddings
Late Fusion
Process each modality independently, fuse at output
Ensemble or late classifier
Middle/Intermediate Fusion
Use attention layers to allow one modality to condition on the other
LXMERT, BLIP

Types of Modality Fusion


原 PPT 独立图片资源
Types of Modality Fusion
Illustration of early fusion, late fusion, and middle fusion methods used by multimodal fusion networks.
https://www.researchgate.net/figure/llustration-of-early-fusion-late-fusion-and-middle-fusion-methods-used-by-multimodal_fig2_362028535

Cross-Attention Fusion: Fine-Grained Modality Interaction


原 PPT 独立图片资源
Cross-Attention Fusion: Fine-Grained Modality Interaction
Key idea: One modality attends to the other to capture alignment & dependency.
Mechanism:
Queries from modality A (e.g., text);
Keys/values from modality B (e.g., image);
Output:
modality-A-aware features guided by modality B.
Supports: visual grounding, co-reference, reasoning.

LXMERT: Separate Encoders + Cross-Modality Layers


原 PPT 独立图片资源
LXMERT: Separate Encoders + Cross-Modality Layers
Separate encoders for language and vision.
Multi-layer cross-attention to allow deep interaction.
Pretrained with multiple objectives:
Masked Language Modeling;
Image-Text Matching;
Visual Question Answering.

LXMERT: Separate Encoders + Cross-Modality Layers


原 PPT 独立图片资源
LXMERT: Separate Encoders + Cross-Modality Layers
原 PPT 独立图片资源
https://arxiv.org/pdf/1908.07490

BLIP: Bootstrapping Language-Image Pre-training


原 PPT 独立图片资源
BLIP: Bootstrapping Language-Image Pre-training
Use a large amount of noisy web image-caption pair;
Vision-language pretraining;
Optimize following losses:
Image-Text Contrastive Loss for align embedding
Image-Text Matching Loss via image-grounded text encoder for matching positive pairs;
Language Modeling Loss via image-grounded text decoder for generating the captions.

BLIP: Bootstrapping Language-Image Pre-training


原 PPT 独立图片资源
BLIP: Bootstrapping Language-Image Pre-training
原 PPT 独立图片资源
https://arxiv.org/pdf/2201.12086

UNITER: One Transformer to Encode Both Modalities


原 PPT 独立图片资源
UNITER: One Transformer to Encode Both Modalities
Concatenates image and text embeddings as a single sequence;
Uses self-attention across both modalities;
No separate branches—fully joint modeling;
Strong performance on VQA, retrieval, NLVR2.

UNITER: One Transformer to Encode Both Modalities


原 PPT 独立图片资源
UNITER: One Transformer to Encode Both Modalities
https://arxiv.org/pdf/1909.11740
原 PPT 独立图片资源

From Specialized Pipelines to Unified Representation


原 PPT 独立图片资源
From Specialized Pipelines to Unified Representation
Traditional multimodal models: Use modality-specific encoders and fusion modules.
Unified representation: Different modalities (text, image, audio, video) are all projected into a shared token space, processed by a single encoder.
Goal: Enable a general-purpose model to handle arbitrary modality combinations in a consistent architecture.
Example: GPT-4V handles both text and image inputs as sequences of tokens.

Motivation for Unification


原 PPT 独立图片资源
Motivation for Unification
Simplifies architecture: fewer modality-specific components.
Easier transfer learning and multi-task learning.
Matches the trend of scaling (foundation models).
Moves toward “everything is a sequence” perspective:
Text → token sequence;
Image → patch sequence;
Audio → spectrogram token sequence.
Image of size 224 x 224 → 196 patches → 196 tokens.

GOAT : Multimodal, Multitask, Multi-environment Agent


原 PPT 独立图片资源
GOAT : Multimodal, Multitask, Multi-environment Agent
All inputs converted to tokens (image patches, joint angles, text).
All outputs are tokens (actions, text, pixels).
One Transformer processes all: “A generalist agent.”
“Same network, different context.”

GOAT : Multimodal, Multitask, Multi-environment Agent


原 PPT 独立图片资源
GOAT : Multimodal, Multitask, Multi-environment Agent
原 PPT 独立图片资源
原 PPT 独立图片资源
https://deepmind.google/discover/blog/a-generalist-agent/

Flamingo: Visual Context in a Text Model


原 PPT 独立图片资源
Flamingo: Visual Context in a Text Model
Frozen language model (e.g., Chinchilla) + Perceiver Resampler for visual input.
Image/video features injected as interleaved prompts.
Allows few-shot multimodal QA, captioning, etc.

Flamingo: Visual Context in a Text Model


原 PPT 独立图片资源
Flamingo: Visual Context in a Text Model
原 PPT 独立图片资源
原 PPT 独立图片资源
https://arxiv.org/pdf/2204.14198#page=21.98

Flamingo: Visual Context in a Text Model


原 PPT 独立图片资源
Flamingo: Visual Context in a Text Model
Frozen language model (e.g., Chinchilla) + Perceiver Resampler for visual input.
Image/video features injected as interleaved prompts.
Allows few-shot multimodal QA, captioning, etc.

GPT-4V : Unified Token Sequence for Vision & Text


原 PPT 独立图片资源
GPT-4V : Unified Token Sequence for Vision & Text
Image input → tokenization → inserted into text (embedding) stream.
Same decoder-only Transformer processes all inputs.
Enables VQA, captioning, OCR, reasoning — all with prompting only.

GPT-4V : Unified Token Sequence for Vision & Text


原 PPT 独立图片资源
GPT-4V : Unified Token Sequence for Vision & Text
原 PPT 独立图片资源
https://arxiv.org/pdf/2309.17421

Summary


原 PPT 独立图片资源
Summary
Modalities offer complementary information.
Fusion enables Reasoning and Decision
Not just combination, but understanding across modalities
Provide comprehensive information for better Solution Generation.

4.2 Perception and Generation 4.2.4 Modal Generation



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.4 Modal Generation

原 PPT 独立图片资源

Language Generation


原 PPT 独立图片资源
Language Generation
Hidden Representation for each step;
For each representation, a token classifier classifies it into a token.
Two Level:
Embedding/Representation Level
Token Level

How to Generate Image?– Discrete Approach VQ-VAE


原 PPT 独立图片资源
How to Generate Image?-- Discrete Approach VQ-VAE
Make a Image Patch Codebook
Image patches are corresponds to the indexes in the codebook – Image Token.
Generating Image Tokens just like Language Generation;
Recover the image from Image Tokens.

DALL-E: Learning the CodeBook via dVAE


原 PPT 独立图片资源
DALL-E: Learning the CodeBook via dVAE
原 PPT 独立图片资源
https://medium.com/@zaiinn440/how-openais-dall-e-works-da24ac6c12fa

DALL-E: Encode and Decode


原 PPT 独立图片资源
DALL-E: Encode and Decode
https://medium.com/@zaiinn440/how-openais-dall-e-works-da24ac6c12fa
原 PPT 独立图片资源
原 PPT 独立图片资源

DALL-E: Generation via Transformer


原 PPT 独立图片资源
DALL-E: Generation via Transformer
https://medium.com/@zaiinn440/how-openais-dall-e-works-da24ac6c12fa
原 PPT 独立图片资源

How to Generate Image?– Continuous Approach – Diffusion


原 PPT 独立图片资源
How to Generate Image?-- Continuous Approach -- Diffusion
原 PPT 独立图片资源
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process

How to Generate Image?– Continuous Approach – Diffusion


原 PPT 独立图片资源
How to Generate Image?-- Continuous Approach -- Diffusion
https://medium.com/@kemalpiro/step-by-step-visual-introduction-to-diffusion-models-235942d2f15c

How to Generate Image?– Continuous Approach – Diffusion


原 PPT 独立图片资源
How to Generate Image?-- Continuous Approach -- Diffusion
https://medium.com/@kemalpiro/step-by-step-visual-introduction-to-diffusion-models-235942d2f15c

Diffusion: Forward Process


原 PPT 独立图片资源
Diffusion: Forward Process
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Diffusion: Backward Process


原 PPT 独立图片资源
Diffusion: Backward Process
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Diffusion: Learning and Loss


原 PPT 独立图片资源
Diffusion: Learning and Loss
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Diffusion: Algorithms


原 PPT 独立图片资源
Diffusion: Algorithms
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process
原 PPT 独立图片资源
𝜖𝜃(𝑥,𝑡) is the neural network we need to learn.
Networks with various architectures can be applied.
It could be dependent on other input, such class labels or input texts.

4.2 Perception and Generation 4.2.5 Case Study: Nano Banana



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.5 Case Study: Nano Banana

原 PPT 独立图片资源

Nano Banana Pro


原 PPT 独立图片资源
Nano Banana Pro
https://blog.google/technology/ai/nano-banana-pro/
Infographic about plants using nano banana pro

Nano Banana Pro


原 PPT 独立图片资源
Nano Banana Pro
https://blog.google/technology/ai/nano-banana-pro/
Infographic about making chai made using Nano Banana Pro

Nano Banana Pro


原 PPT 独立图片资源
Nano Banana Pro
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Nano Banana Pro is just Released


原 PPT 独立图片资源
Nano Banana Pro is just Released
原 PPT 独立图片资源

How Nano Banana Pro Does This?


原 PPT 独立图片资源
How Nano Banana Pro Does This?
https://my.infocaptor.com/hub/summaries/google-for-developers/behind-the-scenes-of-google-s-state-of-the-art-nano-banana-image-model-H6ZXujE1qBA?utm_source=chatgpt.com
Evaluation Signal:
“Koshik has been, you know, talking about it for a long time. One of the like biggest advocates of it. Uh, and we were kind of like brushing him off for a long time about how like, you know, this guy‘s a little crazy. Like he’s really obsessed with text rendering. Um, but eventually it kind of became like one of the staple things we looked at. And you can kind of think about it like um when the model learns how to do this like structure for text, it‘s also kind of able to learn other structure in an image as well. And like in an image you have these like um different like frequencies and you can have like structure which you can think of and but you can also have like texture and stuff like this. So it really gives you signal into like how good the model is at generating the structure of the scene.”
“I think humans uh I was actually a little bit skeptical of the human raider approach to doing evals for image generation. But I think what at least I‘ve realized over time is uh when you have enough humans looking at enough prompts across uh a variety of categories, you actually do get quite a bit of good signal. But obviously this is expensive. You don’t want to always be uh asking a bunch of humans uh to grade images. So looking at this text rendering metric for example while a model is training gives you great signal as to whether it‘s performing like you expect. That’s super interesting.”

How Nano Banana Pro Does This?


原 PPT 独立图片资源
How Nano Banana Pro Does This?
https://my.infocaptor.com/hub/summaries/google-for-developers/behind-the-scenes-of-google-s-state-of-the-art-nano-banana-image-model-H6ZXujE1qBA?utm_source=chatgpt.com
Interleaved Generation (Interaction between Understanding and Generation):
So uh one of the magics of interleaf generation is that it offers you to do a new paradigm for image generation right like so if you have a very complex prompt you know you‘re talking about six different edits what if I go with like 50 different edits right so uh now that the model has a really good mechanism to grab information from the context like pixel perfect and use it in the next turn what you can do is you can ask the model to break down the complex prompt either it is editing or for image generation into multiple steps and do edits like one by one over different steps. So for the first one you do this like edits like this five different things and then for the for the next one the next five and so on so forth.
So it‘s like very similar to the test on compute that we have on the language side, right? So, so you spend more flops and and you let the model to bring uh basically this thinking in the uh in the kind of like the pixel space plus breaking it down into smaller pieces that you can like really nail down uh that specific stage but like accumulate it you can do whatever complex task you want right so I think like again this is the magic of interleaf generation that you can think about you know incremental generation of like really complex images as opposed to traditional way of doing it which was like really push pushing hard for getting the best image in one shot, right? Like at the end of the day there‘s a there’s a capacity that you can push the middle. You know, at some point you realize that okay, you know, with 100 details, uh we cannot do that. But when you have this one interle generation breaking it steps, you can always go for any capacity in any complexity that you want to generate. One of the one of the things that's always top of mind for me, especially as like you're uh Nicole, you're also the PM for uh for our imagined models.

Perception, Generation and System State



原 PPT 独立图片资源
Perception and Generation in a Stateful System
原 PPT 独立图片资源
Observe, express, and retain what matters

4.3 Memory and Knowledge 4.3.1 Memory



原 PPT 独立图片资源

4.3 Memory and Knowledge 4.3.1 Memory

原 PPT 独立图片资源

Why Do Agents Need Memory?


原 PPT 独立图片资源
Why Do Agents Need Memory?
Traditional LLMs:
Stateless / context-window only; every conversation “starts from scratch”
Real-world needs:
Long-horizon, multi-turn tasks: project assistants, learning coaches, dev copilots
Personalization: user preferences, past mistakes, long-term goals
Goals of agent memory:
Persistent learning across sessions
Automatically “recall” the most relevant past experiences and knowledge for the current task

Type of Memories


原 PPT 独立图片资源
Type of Memories
Inspired by human memory:
Semantic memory: facts, concepts, general knowledge
Episodic memory: concrete interactions and task experiences
Procedural memory: skills, strategies, workflows
Multi-level memory in typical systems:
Working / short-term memory: current conversation and active state
Long-term / persistent memory: cross-session, structured storage
Reflective memory: summaries and self-improvement based on past episodes

Recent Direction 1 — Structured and Hierarchical Memory


原 PPT 独立图片资源
Recent Direction 1 — Structured and Hierarchical Memory
Structured memory vs. flat vector retrieval:
Instead of “dump everything into a single vector store,” combine multiple structures: lists, trees, graphs, knowledge graphs, etc.
Representative work:
AriGraph: uses a “semantic knowledge graph + episodic nodes/edges” as a world model, significantly improving agents in text-based games
Zep / temporal KGs: graph-structured, time-aware memory for multi-turn dialogue and event tracking
Trend:
From simple bi-directional vector search to structured world models + hierarchical retrieval + reflective summarization

AriGraph


原 PPT 独立图片资源
AriGraph
Ariadne agent and its results
https://github.com/AIRI-Institute/AriGraph

Recent Direction 2 — Learning-Based and Adaptive Memory Policies


原 PPT 独立图片资源
Recent Direction 2 — Learning-Based and Adaptive Memory Policies
Problem: what to store, when to store, and how to retrieve are still often defined by hand (thresholds, scores, top-k, etc.).
Representative work:
Adaptive memory optimization for LLM agents: Uses off-policy / on-policy learning to let agents learn “what is worth remembering” and “how to encode it.”
Memory-augmented LLM agents : Cross-task training + dynamic memory retrieval to improve sample efficiency in multi-task, multi-turn instruction following.
Core trend:
Memory strategies are no longer purely hard-coded, but can be optimized via reinforcement learning or meta-learning.

Learning to Memorize


原 PPT 独立图片资源
Learning to Memorize
https://arxiv.org/pdf/2508.16629
原 PPT 独立图片资源

4.3 Memory and Knowledge 4.3.2 External Knowledge and Retrieval



原 PPT 独立图片资源

4.3 Memory and Knowledge 4.3.2 External Knowledge and Retrieval

原 PPT 独立图片资源

Agent need External Knowledge


原 PPT 独立图片资源
Agent need External Knowledge
Parameter Efficiency & Coverage
Impossible to internalize the ever-expanding corpus of human knowledge
On-demand retrieval fills rare or specialized information gaps
Up-to-Date & Dynamic Information
Static model parameters become stale as the world evolves
Live access to news, databases, and APIs ensures current answers
Grounding & Multi-Modality
Ties language to real-world entities (objects, sensor streams, images)
Knowledge graphs provide explicit commonsense (e.g., physics, causality)
Memory & Long-Term Interaction
External “memory” (vector DBs, key–value stores) for session history
Enables personalization and sustained, coherent dialogues

Coping with External Knowledge


原 PPT 独立图片资源
Coping with External Knowledge
External Knowledge
Recent News/Documents
Private Document/Email
Knowledge Graphs, Data Base
Private Images/Videos
…
How to use these knowledge?
Retrieve relevant information;
Generate response using these retrieved information.

RAG: Retrieval-Augmented Generation


原 PPT 独立图片资源
RAG: Retrieval-Augmented Generation
原 PPT 独立图片资源
https://github.com/hkproj/retrieval-augmented-generation-notes/blob/main/Slides.pdf

RAG Road Maps


原 PPT 独立图片资源
RAG Road Maps
https://github.com/hkproj/retrieval-augmented-generation-notes/blob/main/Slides.pdf
原 PPT 独立图片资源

What to Retrieve?


原 PPT 独立图片资源
What to Retrieve?
Different Information Level: Document, Chunk, Sentence, Words
High Level:
Rich Context & Coherence, Higher Recall, Simpler Indexing
Lower Precision / More Noise, Context‐Window Bloat
Suitable when you need comprehensive background or discourse coherence.
Low Level:
High Precision & Conciseness, Efficient Context Usage, Fine-Grained Control
Lack of Broader Context, Lower Recall
Suitable when you want pinpoint accuracy and to conserve prompt tokens.

Graph RAG


原 PPT 独立图片资源
Graph RAG
原 PPT 独立图片资源
https://arxiv.org/pdf/2404.16130
原 PPT 独立图片资源
https://microsoft.github.io/graphrag/

Knowledge Graph, Table, SQL Database, …


原 PPT 独立图片资源
Knowledge Graph, Table, SQL Database, …
Overview of PIKE-RAG Framework
https://github.com/microsoft/PIKE-RAG

Recall: How to Retrieve?


原 PPT 独立图片资源
Recall: How to Retrieve?
Inverted index + BM25
Embedding Vector + ANN Search
https://www.pinecone.io/learn/series/faiss/hnsw/
The search process through the multi-layer structure of an HNSW graph.
原 PPT 独立图片资源
https://medium.com/@mhammadkhan/neural-re-ranking-models-c0a67278f626

When to Retrieve


原 PPT 独立图片资源
When to Retrieve
User ‘s Intention: Web search Button
Intention Detection: Detect for each request in the dialog: Notebook LM
Low Confidence Detected: Call retrieve when unreliable answer found.
原 PPT 独立图片资源
原 PPT 独立图片资源
https://afforai.com/comparisons/notebook-lm-alternative

When to Retrieve


原 PPT 独立图片资源
When to Retrieve
https://arxiv.org/pdf/2305.06983
原 PPT 独立图片资源
原 PPT 独立图片资源

4.4 Action and Execution 4.4.1 Tool Calling



原 PPT 独立图片资源

4.4 Action and Execution 4.4.1 Tool Calling

原 PPT 独立图片资源

Agent need to take Action


原 PPT 独立图片资源
Agent need to take Action
Beyond “Tell” to “Do”
Closing the loop: from understanding to executing tasks (e.g., sending emails, scheduling)
Tool-Calling & External APIs
Interface with calculators, search engines, IoT devices, business systems
Enables multi-step workflows under LLM orchestration
Agentic Autonomy
Persistent agents can monitor, adapt, and intervene in real-world processes
Frees users from manual coordination of complex tasks

Tool Calling


原 PPT 独立图片资源
Tool Calling
Definition: Enabling an LLM to invoke external functions or services (APIs, calculators, search engines) during conversation.
Why it matters:
Keeps knowledge up-to-date beyond training cutoff
Adds deterministic computation (math, code execution)
Connects LLMs to real-world data and actions
Introduced by Open AI at 2023.

Define Tools


原 PPT 独立图片资源
Define Tools
原 PPT 独立图片资源
https://medium.com/@developer.yasir.pk/tool-calling-for-llms-a-detailed-tutorial-a2b4d78633e2

Prepare the Conversation


原 PPT 独立图片资源
Prepare the Conversation
https://medium.com/@developer.yasir.pk/tool-calling-for-llms-a-detailed-tutorial-a2b4d78633e2
原 PPT 独立图片资源

Pass Tools to the LLM


原 PPT 独立图片资源
Pass Tools to the LLM
https://medium.com/@developer.yasir.pk/tool-calling-for-llms-a-detailed-tutorial-a2b4d78633e2

原 PPT 独立图片资源

Interpret LLM Output


原 PPT 独立图片资源
Interpret LLM Output
https://medium.com/@developer.yasir.pk/tool-calling-for-llms-a-detailed-tutorial-a2b4d78633e2

原 PPT 独立图片资源

Execute Tool Calls


原 PPT 独立图片资源
Execute Tool Calls
https://medium.com/@developer.yasir.pk/tool-calling-for-llms-a-detailed-tutorial-a2b4d78633e2

原 PPT 独立图片资源

Incorporate Tool Results


原 PPT 独立图片资源
Incorporate Tool Results
https://medium.com/@developer.yasir.pk/tool-calling-for-llms-a-detailed-tutorial-a2b4d78633e2

原 PPT 独立图片资源

Continue Conversation


原 PPT 独立图片资源
Continue Conversation
https://medium.com/@developer.yasir.pk/tool-calling-for-llms-a-detailed-tutorial-a2b4d78633e2

原 PPT 独立图片资源

4.4 Action and Execution 4.4.2 Coding



原 PPT 独立图片资源

4.4 Action and Execution 4.4.2 Coding

原 PPT 独立图片资源

Agent should write Code


原 PPT 独立图片资源
Agent should write Code
Automating Development
Rapid prototyping: generate boilerplate, tests, and integration snippets
Lower barrier for non-programmers to implement data pipelines and analyses
Self-Improving Systems
Models can modify their own tooling or workflows in response to new requirements
Enables continuous adaptation without full manual redevelopment
Explainable & Reproducible
Generated code with comments provides a clear audit trail of the model’s logic

From Tool Calling to Coding


原 PPT 独立图片资源
From Tool Calling to Coding
Coding is the interface to control the natural world;
Tool Calling: calling human written code;
Coding: generating code to satisfy new requirements.

Coding is Long-Term Goal of LLM


原 PPT 独立图片资源
Coding is Long-Term Goal of LLM
原 PPT 独立图片资源

https://arxiv.org/pdf/2406.00515

Coding contributes a lot in Advances of LLM


原 PPT 独立图片资源
Coding contributes a lot in Advances of LLM
https://arxiv.org/pdf/2303.10420
原 PPT 独立图片资源

Training to Generate Code


原 PPT 独立图片资源
Training to Generate Code
https://arxiv.org/pdf/2406.00515
原 PPT 独立图片资源

Data is Foundation


原 PPT 独立图片资源
Data is Foundation
https://arxiv.org/pdf/2406.00515
原 PPT 独立图片资源
原 PPT 独立图片资源

Code Act


原 PPT 独立图片资源
Code Act
https://github.com/xingyaoww/code-act
Overview

4.5 Planning and Control 4.5.1 Agent Control and Planning

The Agent Chooses the Next Operation

agent controllerAgent state enters a controller, which selects answer, deliberation or external action. Environment observations return to evidence and memory; stop conditions are explicit. State + evidenceGoal, history, budgetControllerAnswer / think / actExecute + observeModel, tool, environmentAssess / stopCheck outcome + costnew observations → update state / memory

More thinking changes a candidate. An action may reveal new evidence or change the environment.

Updating context, beliefs or memory is not necessarily a parameter update.
Stop on a checked result, a justified handoff, a budget limit, or lack of a useful next action.

Ana: Think More, or Obtain New Evidence?

Current gap Useful next operation What it can resolve
Policy translated incorrectly Re-read policy; repair formalization Semantic fidelity
Proof search unfinished Allocate solver time Formal consequence
Filing record unreliable Query authoritative filing system Evidence about F
Source conflict unresolved Inspect provenance / request review Dependence and factual dispute

P(R ∣ O) = 11/21 concerns review priority. It does not authorize approval.
Choose an operation that addresses the actual uncertainty.

Case Study: Plan and Execute


原 PPT 独立图片资源
Case Study: Plan and Execute
plan-and-execute diagram
https://langchain-ai.github.io/langgraph/tutorials/plan-and-execute/plan-and-execute/

4.5 Planning and Control 4.5.2 System I / II in Agents



原 PPT 独立图片资源

4.5 Planning and Control 4.5.2 System I / II in Agents

原 PPT 独立图片资源

System I / II: An Operational Analogy

Dimension Direct response Additional deliberation
Computation Generate the answer directly Reason, branch, revise or check
Useful when Current policy and evidence suffice Extra work can resolve uncertainty
Failure Confident shortcut Repeated error; unnecessary work
Measure Success, actual compute, latency, evidence gained

Fast is not automatically shallow. Slow is not automatically correct.
These labels do not establish human-like mental systems inside an LLM.

No fixed “95% / 5%” allocation is assumed. A direct answer still requires neural computation.

Training, Decoding and Agent Control

Layer What changes? Example
Training Model parameters SFT, RLVR, distillation
Decoding / inference Candidate trajectory and budget Sampling, search, reasoning tokens
Agent control Next operation and environment state Call a tool, observe, retry, stop

A single model can support several modes. A search can repeatedly call a direct policy.

Nucleus sampling is a decoding choice, not a definition of System I.
Public o1 evidence does not establish a specific MCTS + CriticGPT architecture.

4.5 Planning and Control 4.5.3 Adaptive Reasoning

Recent Work: Different Control Problems

Question Representative work What is controlled?
Expose two modes Qwen3 (May 2025) Prompt / template + budget
Learn when to use each Thinkless; AdaptThink (2025) Mode-selection policy
Predict required budget Sonata (ICLR 2026) Compute allocation before reasoning
Make future direct answers better System 2 distillation (2024) Parameters, using selected outputs
Check simple-task behavior S1-Bench (IJCAI 2026) Evaluation, not a controller

Research selection checked on 2026-11-15. Versions and primary sources are recorded in the reading page.

Qwen3: Make Both Modes Available

qwen modeMode-fusion data Thinking + direct answers → Post-training Learn both behaviors → Template + budget Choose / constrain mode → Response With or without long CoT Mode-fusion dataThinking + direct answersPost-trainingLearn both behaviorsTemplate + budgetChoose / constrain modeResponseWith or without long CoT

The May 2025 report provides a controllable interface.
Mode availability alone does not demonstrate an optimal learned router.

§4.3: thinking-mode fusion. §4.7: budget analysis. These claims refer to the original unified-mode version.

Thinkless: Learn the Mode Token

thinkless modeWarm-up distillation Short + long examples → Select control token <short> or <think> → Generate + score Correctness; short bias → DeGRPO update Control / response losses Warm-up distillationShort + long examplesSelect control token<short> or <think>Generate + scoreCorrectness; short biasDeGRPO updateControl / response losses
Table 1 · 1.5B Baseline → Thinkless accuracy Mean response tokens
GSM8K 83.47% → 84.18% 1919 → 624
MATH-500 86.08% → 81.84% 5675 → 2555

Token savings can accompany an accuracy loss. Thinking frequency, token count and wall time are different metrics.

AdaptThink: Prefer Direct Answers Under a Constraint

adaptthink modeReference policy Estimate task reward → Sample both modes Balanced exploration → Constrained objective Favor NoThinking, Retain task quality → Policy update Learn a mode per input Reference policyEstimate task rewardSample both modesBalanced explorationConstrained objectiveFavor NoThinkingRetain task qualityPolicy updateLearn a mode per input

Cold start: a long-thinking model may never sample direct answers without explicit exploration.

Importance sampling: correct for the sampling distribution while learning from both modes.

The performance constraint is an optimization target—not a proof that every future answer is correct.

AdaptThink: Quality and Cost Together

DeepSeek-R1-Distill-Qwen-1.5B · original Thinking → AdaptThink · Table 1

Dataset Accuracy (%) Response tokens NoThinking share
GSM8K 79.0 → 83.1 978 → 480 86.9%
MATH500 80.6 → 82.0 4887 → 1782 76.8%
AIME 2024 29.4 → 31.0 12073 → 6679 40.4%

Three-dataset macro average: +2.4 pp accuracy; −53.0% response length.

16K evaluation context; temperature 0.6. AIME: 16 samples per problem, averaged—not best-of-16. Token reduction is not measured wall-clock speedup.

Sonata: Predict a Budget Before Thinking

sonata budgetOffline calibration Multi-path consistency → Train small adapter Predict agreement, from hidden states → Online query prefill Read representation → Allocate budget Then generate reasoning Offline calibrationMulti-path consistencyTrain small adapterPredict agreementfrom hidden statesOnline query prefillRead representationAllocate budgetThen generate reasoning

Self-consistency is a proxy for thinking need, not a truth certificate.

“Latent space” here supplies features to a budget predictor. It does not mean all reasoning becomes a Coconut-style latent chain.

Sonata: What the 2026 Evidence Supports

Reported in ICLR 2026 proceedings

20–60% fewer thinking tokens at the same accuracy across reported model / task settings.

Models include Qwen3-8B / 32B, GPT-OSS-120B, Qwen3-235B-A22B.

Read the boundary

Tasks include AIME25, GSM8K, MATH500, GPQA and LiveCodeBench.

Not a universal per-query guarantee.
Not a tool-use success benchmark.
Not the same as measured latency.

Evaluate calibration cost, routing overhead, difficult-tail failures and actual task quality.

Version frozen to the conference proceedings; later author summaries report different experiment ranges.

4.6 Coordination and Collaboration 4.6.1 Workflows and Coordination



原 PPT 独立图片资源

4.6 Coordination and Collaboration 4.6.1 Workflows and Coordination

原 PPT 独立图片资源

Workflows and Multi-Agent Systems


原 PPT 独立图片资源
Workflows and Multi-Agent Systems
https://langchain-ai.github.io/langgraph/tutorials/workflows/#agent
原 PPT 独立图片资源

Prompt Chaining


原 PPT 独立图片资源
Prompt Chaining
https://langchain-ai.github.io/langgraph/tutorials/workflows/#agent
原 PPT 独立图片资源

Parallelization


原 PPT 独立图片资源
Parallelization
https://langchain-ai.github.io/langgraph/tutorials/workflows/#agent
原 PPT 独立图片资源

Routing


原 PPT 独立图片资源
Routing
https://langchain-ai.github.io/langgraph/tutorials/workflows/#agent
原 PPT 独立图片资源

Orchestrator-Worker


原 PPT 独立图片资源
Orchestrator-Worker
https://langchain-ai.github.io/langgraph/tutorials/workflows/#agent
原 PPT 独立图片资源

Evaluator-optimizer


原 PPT 独立图片资源
Evaluator-optimizer
https://langchain-ai.github.io/langgraph/tutorials/workflows/#agent
原 PPT 独立图片资源

4.6 Coordination and Collaboration 4.6.2 Multi-Agent Case Studies

Case Study: Research and Report


原 PPT 独立图片资源
Case Study: Research and Report
https://blog.langchain.dev/langgraph-multi-agent-workflows/
原 PPT 独立图片资源

Case Study: Trading Agent


原 PPT 独立图片资源
Case Study: Trading Agent
TradingAgents: Multi-Agents LLM Financial Trading Framework
原 PPT 独立图片资源

Case Study: Agent Conversation


原 PPT 独立图片资源
Case Study: Agent Conversation
https://medium.com/@tsiciliani/multi-agent-conversation-with-autogen-ai-52a48240e698
原 PPT 独立图片资源

Case Study: Agent Town


原 PPT 独立图片资源
Case Study: Agent Town
https://hai.stanford.edu/news/computational-agents-exhibit-believable-humanlike-behavior
Image of the generative agents virtual town
Flow chart of the generative agent memory

4.7 Learning and Evaluation 4.7.1 Feedback and System Optimization



原 PPT 独立图片资源

4.7 Learning and Evaluation 4.7.1 Feedback and System Optimization

原 PPT 独立图片资源

Search, Learn, Evaluate: A Conditional Cycle

improvement cyclePolicy Propose candidates → Search + check Improve / filter outputs → Training Update parameters → Held-out evaluation Measure gain and cost PolicyPropose candidatesSearch + checkImprove / filter outputsTrainingUpdate parametersHeld-out evaluationMeasure gain and cost

Better search outputs may teach a better policy—if supervision is useful and transferable.

Possible failure: biased checks, copied errors, no useful reward, distribution shift.

Stop or revise: no held-out gain, worse reliability, excessive cost, exhausted evidence.

An iterative procedure is not evidence of unlimited improvement or inevitable superintelligence.

Case Study: Alpha Evolve


原 PPT 独立图片资源
Case Study: Alpha Evolve
https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/
原 PPT 独立图片资源

Agent Paradigm


原 PPT 独立图片资源
Agent Paradigm
https://arxiv.org/pdf/2510.05592
原 PPT 独立图片资源

Agent Flow


原 PPT 独立图片资源
Agent Flow
https://arxiv.org/pdf/2510.05592
原 PPT 独立图片资源

Agent Flow


原 PPT 独立图片资源
Agent Flow
https://arxiv.org/pdf/2510.05592
原 PPT 独立图片资源

4.7 Learning and Evaluation 4.7.2 Distillation and Transfer

Distillation: Can Slow Work Improve Fast Responses?

system2 distillationExtra computation Generate stronger outputs → Select targets Consistency / filtering → Answer-only SFT Do not retain rationale → Direct response Evaluate on held-out tasks Extra computationGenerate stronger outputsSelect targetsConsistency / filteringAnswer-only SFTDo not retain rationaleDirect responseEvaluate on held-out tasks

Yu et al. (2024): several methods transfer useful behavior; transfer is task-dependent.

Self-consistency filtering is not formal verification.
This experiment removes intermediate reasoning from the training target; other distillation designs can retain it.

Distillation Can Fail: Keep the Negative Result

Llama-2-70B-chat · GSM8K · k = 1 · original Table 4

Condition Accuracy Generated tokens
Direct few-shot (no CoT) 7.58% 57
Zero-shot CoT teacher 52.77% 270
Answer-only CoT distillation 7.13% 18

Fewer output tokens did not preserve the teacher’s math accuracy.

This is one base model and one self-supervised answer-only recipe. It does not show that all reasoning distillation is impossible.

S1-Bench: Test the Easy End Too

IJCAI 2026: evaluate simple tasks, response cost and consistency across repeated samples.

Table 3 pass@1 acc@5 (all 5 correct) Mean tokens
DS-R1-1.5B 81.47% 54.50% 489.54
DS-R1 (671B) 100.00% 100.00% 646.40

Good hard-task scores do not settle simple-task reliability or efficiency.

T=0.6; top-p=0.95; k=5; max 10K. Paper-defined metrics use formatting checks and GPT-4o judging. “System 1” is a benchmark label, not a diagnosis of a mental mechanism.

4.7 Learning and Evaluation 4.7.3 Evaluation and Integration

Evaluate the Whole Controller

Compare on the same tasks Report together
Always direct / always deliberate Correctness, hard-constraint violations
Fixed budget / learned allocation Tokens, actual latency, tail latency
Reason only / retrieve evidence Tool cost, evidence gained, retries
Before / after training Held-out transfer, training cost, regressions

Count routing, failed attempts and tool calls. Preserve the same checking rules.
A policy that refuses every task is cheap; report coverage as well as accepted-case accuracy.

The Agent System as a Whole

agent course handoffControl policy What should happen next? → Tool calling Execute a typed action → Memory + knowledge Retain / retrieve evidence → Feedback + evaluation Update state; check goals Control policyWhat should happen next?Tool callingExecute a typed actionMemory + knowledgeRetain / retrieve evidenceFeedback + evaluationUpdate state; check goals

Tools, code, memory and retrieval provide operations for the controller.
Next: divide work across agents and check the joint result.

A successful local step is not enough: evaluate the complete task, including handoffs and failed attempts.

Thank You!



原 PPT 独立图片资源
Thank You!
原 PPT 独立图片资源