Perception and Generation


原 PPT 独立图片资源
Natural Language Understanding
Perception and Generation
Mingming Sun
BIMSA

Multi-Modal Understanding and Generation


原 PPT 独立图片资源
Understanding and Generation
4.2.1 Multimodality
4.2.2 Modal Representation
4.2.3 Multi-Modal Fusion
4.2.4 Modal Generation
4.2.5 Case Study: Nano Banana

4.2 Perception and Generation 4.2.1 Multimodality



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.1 Multimodality

原 PPT 独立图片资源

What is a Modality


原 PPT 独立图片资源
What is a Modality
A modality refers to a form of information or sensory channel — such as text, image, audio, video, or touch.
A multimodal system involves multiple types of data modalities, and is capable of jointly perceiving, understanding, and processing them.
Goal of multimodal models: to simulate how humans use multiple senses to understand the world holistically.
原 PPT 独立图片资源
https://www.appen.com/multi-modal-ai

Why Do We Need Multimodal Models?


原 PPT 独立图片资源
Why Do We Need Multimodal Models?
People naturally use a combination of vision, speech, and language to communicate.
Real-world inputs are inherently multimodal
Autonomous driving: Combines camera images, radar data, and GPS
Healthcare: Interpreting scans + doctor notes + patient speech
Customer service bots: Understand user voice, detect emotion, display visuals
Virtual assistants (e.g., Siri, Alexa): Use voice, text, context

Why Do We Need Multimodal Models?


原 PPT 独立图片资源
Why Do We Need Multimodal Models?
Limitations of Single-Modality Models
Information Loss: Relying on a single modality (e.g., text only or image only) can lead to incomplete understanding.
Example: A caption “A man is holding something” lacks detail if the image isn’t considered.
Ambiguity: Single-modality input often leads to uncertainty or confusion.
Example: The spoken word “bank” could mean a financial institution or a riverbank — image context can resolve this.

Evolution of Multimodal Models


原 PPT 独立图片资源
Evolution of Multimodal Models
Pre-AGI times: Mapping between different modals for specific task
The principle of audio-visual speech recognition. 
https://www.researchgate.net/figure/The-principle-of-audio-visual-speech-recognition_fig1_261268684
Showcasing the location of Metatags from the Open Graph Protocol
https://programmingpercy.tech/blog/opengraph-protocol-how-and-why/

Evolution of Multimodal Models


原 PPT 独立图片资源
Evolution of Multimodal Models
Nowadays
https://www.youtube.com/watch?v=2PShpVgQo2M

How to Achieve Multimodal Understanding?


原 PPT 独立图片资源
How to Achieve Multimodal Understanding?
Modal Representation
Feature Extraction
Embedding
Across all Modals
Muti-Modal Fusion
Early Fusion
Late Fusion
Joint Fusion
Modal Generation
Generative Model
Across all Models

4.2 Perception and Generation 4.2.2 Modal Representation



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.2 Modal Representation

原 PPT 独立图片资源

Single Modal Representation: Language


原 PPT 独立图片资源
Single Modal Representation: Language
Word2Vec
BERT
原 PPT 独立图片资源
(Mikolov et al. 2013)
原 PPT 独立图片资源
https://medium.com/data-science/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270

Single Modal Representation: Vision


原 PPT 独立图片资源
Single Modal Representation: Vision
CNNs: LeNet, Alex Net, ZFNet, GoogLeNet/Inception, VGG Net, ResNet
AlexNet architecture used as the baseline model for the analysis of results on the CIFAR-C data set. Light yellow indicates a convolution, while darker yellow is a ReLU function. Similarly, light
https://www.researchgate.net/figure/AlexNet-architecture-used-as-the-baseline-model-for-the-analysis-of-results-on-the_fig5_339756908
原 PPT 独立图片资源
https://medium.com/analytics-vidhya/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5
原 PPT 独立图片资源
https://medium.com/analytics-vidhya/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5

Single Modal Representation: Vision


原 PPT 独立图片资源
Single Modal Representation: Vision
Vision Transformer: ViT
https://paperswithcode.com/method/vision-transformer
原 PPT 独立图片资源

Single Modal Representation: Voice


原 PPT 独立图片资源
Single Modal Representation: Voice
Wave2Vec
BERT
https://arxiv.org/pdf/1904.05862
https://sh-tsang.medium.com/brief-review-hubert-self-supervised-speech-representation-learning-by-masked-prediction-of-e8baf7909c4a
原 PPT 独立图片资源
原 PPT 独立图片资源

Cross Modal Representation


原 PPT 独立图片资源
Cross Modal Representation
Learn a shared semantic space for different modalities (e.g., image, text).
Each modality is independently encoded into vectors, but aligned through training.
Bridging Modalities: A Survey of Cross-Modal Image-Text Retrieval
https://www.iece.org/article/abs/cjif.2024.361895

Why Align Modalities


原 PPT 独立图片资源
Why Align Modalities
Different modalities describe the same concept (e.g., a cat in text and image).
We want:

Aligned representations enable cross-modal retrieval, zero-shot learning, etc.
原 PPT 独立图片资源

Contrastive Learning: The Engine Behind


原 PPT 独立图片资源
Contrastive Learning: The Engine Behind
Sample positive image-text pairs (𝐼, 𝑇+) , and negative pairs (𝐼, 𝑇−).
Use contrastive loss:


Contrastive loss is a special form of cross-entropy loss, treating the matching pair as the positive class and all others as negatives in a softmax over the batch.
Objective: pull positive pairs together, push apart negative pairs.
原 PPT 独立图片资源

CLIP: Large-Scale Cross-Modal Training


原 PPT 独立图片资源
CLIP: Large-Scale Cross-Modal Training
Proposed by OpenAI (2021); trained on 400M image-text pairs.
Architecture:
Image encoder: Vision Transformer (ViT)
Text encoder: Transformer
Trained via symmetric contrastive loss.
Supports zero-shot classification via prompt-based text queries.

CLIP: Large-Scale Cross-Modal Training


原 PPT 独立图片资源
CLIP: Large-Scale Cross-Modal Training
原 PPT 独立图片资源
https://openai.com/index/clip/

ALIGN: Scaling Contrastive Learning


原 PPT 独立图片资源
ALIGN: Scaling Contrastive Learning
Google Research, 2021
Trained on 1.8B noisy image-text pairs from the web.
Simpler than CLIP:
Image: EfficientNet
Text: BERT
No symmetric loss; only image-to-text direction.
Relies on data scale + simple architecture.
Insight: Even noisy web-scale data works if quantity is large enough.

Key Ideas in Cross-Modal Contrastive Representation


原 PPT 独立图片资源
Key Ideas in Cross-Modal Contrastive Representation
Cross-modal representation enables:
Retrieval
Zero-shot learning
Prompt-based classification
Core ingredients:
Independent encoders
Contrastive objective
Large paired datasets
CLIP and ALIGN show the power of contrastive learning + scale.
But we need more thorough interaction between modals.

4.2 Perception and Generation 4.2.3 Multi-Modal Fusion



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.3 Multi-Modal Fusion

原 PPT 独立图片资源

Understanding Modality Fusion


原 PPT 独立图片资源
Understanding Modality Fusion
Definition: the process of integrating information from multiple modalities (e.g., image + text) to perform a task.
Objective:
Capture inter-modal interaction;
Enable joint reasoning and decision making.
Output:
Unified representation;
Task-specific predictions (e.g., answer, caption, class).
原 PPT 独立图片资源
https://medium.com/@raj.pulapakura/multimodal-models-and-fusion-a-complete-guide-225ca91f6861
原 PPT 独立图片资源

Types of Modality Fusion


原 PPT 独立图片资源
Types of Modality Fusion
Type
Description
Example
Early Fusion
Concatenate raw inputs or early features
Multimodal embeddings
Late Fusion
Process each modality independently, fuse at output
Ensemble or late classifier
Middle/Intermediate Fusion
Use attention layers to allow one modality to condition on the other
LXMERT, BLIP

Types of Modality Fusion


原 PPT 独立图片资源
Types of Modality Fusion
Illustration of early fusion, late fusion, and middle fusion methods used by multimodal fusion networks.
https://www.researchgate.net/figure/llustration-of-early-fusion-late-fusion-and-middle-fusion-methods-used-by-multimodal_fig2_362028535

Cross-Attention Fusion: Fine-Grained Modality Interaction


原 PPT 独立图片资源
Cross-Attention Fusion: Fine-Grained Modality Interaction
Key idea: One modality attends to the other to capture alignment & dependency.
Mechanism:
Queries from modality A (e.g., text);
Keys/values from modality B (e.g., image);
Output:
modality-A-aware features guided by modality B.
Supports: visual grounding, co-reference, reasoning.

LXMERT: Separate Encoders + Cross-Modality Layers


原 PPT 独立图片资源
LXMERT: Separate Encoders + Cross-Modality Layers
Separate encoders for language and vision.
Multi-layer cross-attention to allow deep interaction.
Pretrained with multiple objectives:
Masked Language Modeling;
Image-Text Matching;
Visual Question Answering.

LXMERT: Separate Encoders + Cross-Modality Layers


原 PPT 独立图片资源
LXMERT: Separate Encoders + Cross-Modality Layers
原 PPT 独立图片资源
https://arxiv.org/pdf/1908.07490

BLIP: Bootstrapping Language-Image Pre-training


原 PPT 独立图片资源
BLIP: Bootstrapping Language-Image Pre-training
Use a large amount of noisy web image-caption pair;
Vision-language pretraining;
Optimize following losses:
Image-Text Contrastive Loss for align embedding
Image-Text Matching Loss via image-grounded text encoder for matching positive pairs;
Language Modeling Loss via image-grounded text decoder for generating the captions.

BLIP: Bootstrapping Language-Image Pre-training


原 PPT 独立图片资源
BLIP: Bootstrapping Language-Image Pre-training
原 PPT 独立图片资源
https://arxiv.org/pdf/2201.12086

UNITER: One Transformer to Encode Both Modalities


原 PPT 独立图片资源
UNITER: One Transformer to Encode Both Modalities
Concatenates image and text embeddings as a single sequence;
Uses self-attention across both modalities;
No separate branches—fully joint modeling;
Strong performance on VQA, retrieval, NLVR2.

UNITER: One Transformer to Encode Both Modalities


原 PPT 独立图片资源
UNITER: One Transformer to Encode Both Modalities
https://arxiv.org/pdf/1909.11740
原 PPT 独立图片资源

From Specialized Pipelines to Unified Representation


原 PPT 独立图片资源
From Specialized Pipelines to Unified Representation
Traditional multimodal models: Use modality-specific encoders and fusion modules.
Unified representation: Different modalities (text, image, audio, video) are all projected into a shared token space, processed by a single encoder.
Goal: Enable a general-purpose model to handle arbitrary modality combinations in a consistent architecture.
Example: GPT-4V handles both text and image inputs as sequences of tokens.

Motivation for Unification


原 PPT 独立图片资源
Motivation for Unification
Simplifies architecture: fewer modality-specific components.
Easier transfer learning and multi-task learning.
Matches the trend of scaling (foundation models).
Moves toward “everything is a sequence” perspective:
Text → token sequence;
Image → patch sequence;
Audio → spectrogram token sequence.
Image of size 224 x 224 → 196 patches → 196 tokens.

GOAT : Multimodal, Multitask, Multi-environment Agent


原 PPT 独立图片资源
GOAT : Multimodal, Multitask, Multi-environment Agent
All inputs converted to tokens (image patches, joint angles, text).
All outputs are tokens (actions, text, pixels).
One Transformer processes all: “A generalist agent.”
“Same network, different context.”

GOAT : Multimodal, Multitask, Multi-environment Agent


原 PPT 独立图片资源
GOAT : Multimodal, Multitask, Multi-environment Agent
原 PPT 独立图片资源
原 PPT 独立图片资源
https://deepmind.google/discover/blog/a-generalist-agent/

Flamingo: Visual Context in a Text Model


原 PPT 独立图片资源
Flamingo: Visual Context in a Text Model
Frozen language model (e.g., Chinchilla) + Perceiver Resampler for visual input.
Image/video features injected as interleaved prompts.
Allows few-shot multimodal QA, captioning, etc.

Flamingo: Visual Context in a Text Model


原 PPT 独立图片资源
Flamingo: Visual Context in a Text Model
原 PPT 独立图片资源
原 PPT 独立图片资源
https://arxiv.org/pdf/2204.14198#page=21.98

Flamingo: Visual Context in a Text Model


原 PPT 独立图片资源
Flamingo: Visual Context in a Text Model
Frozen language model (e.g., Chinchilla) + Perceiver Resampler for visual input.
Image/video features injected as interleaved prompts.
Allows few-shot multimodal QA, captioning, etc.

GPT-4V : Unified Token Sequence for Vision & Text


原 PPT 独立图片资源
GPT-4V : Unified Token Sequence for Vision & Text
Image input → tokenization → inserted into text (embedding) stream.
Same decoder-only Transformer processes all inputs.
Enables VQA, captioning, OCR, reasoning — all with prompting only.

GPT-4V : Unified Token Sequence for Vision & Text


原 PPT 独立图片资源
GPT-4V : Unified Token Sequence for Vision & Text
原 PPT 独立图片资源
https://arxiv.org/pdf/2309.17421

Summary


原 PPT 独立图片资源
Summary
Modalities offer complementary information.
Fusion enables Reasoning and Decision
Not just combination, but understanding across modalities
Provide comprehensive information for better Solution Generation.

4.2 Perception and Generation 4.2.4 Modal Generation



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.4 Modal Generation

原 PPT 独立图片资源

Language Generation


原 PPT 独立图片资源
Language Generation
Hidden Representation for each step;
For each representation, a token classifier classifies it into a token.
Two Level:
Embedding/Representation Level
Token Level

How to Generate Image?– Discrete Approach VQ-VAE


原 PPT 独立图片资源
How to Generate Image?-- Discrete Approach VQ-VAE
Make a Image Patch Codebook
Image patches are corresponds to the indexes in the codebook – Image Token.
Generating Image Tokens just like Language Generation;
Recover the image from Image Tokens.

DALL-E: Learning the CodeBook via dVAE


原 PPT 独立图片资源
DALL-E: Learning the CodeBook via dVAE
原 PPT 独立图片资源
https://medium.com/@zaiinn440/how-openais-dall-e-works-da24ac6c12fa

DALL-E: Encode and Decode


原 PPT 独立图片资源
DALL-E: Encode and Decode
https://medium.com/@zaiinn440/how-openais-dall-e-works-da24ac6c12fa
原 PPT 独立图片资源
原 PPT 独立图片资源

DALL-E: Generation via Transformer


原 PPT 独立图片资源
DALL-E: Generation via Transformer
https://medium.com/@zaiinn440/how-openais-dall-e-works-da24ac6c12fa
原 PPT 独立图片资源

How to Generate Image?– Continuous Approach – Diffusion


原 PPT 独立图片资源
How to Generate Image?-- Continuous Approach -- Diffusion
原 PPT 独立图片资源
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process

How to Generate Image?– Continuous Approach – Diffusion


原 PPT 独立图片资源
How to Generate Image?-- Continuous Approach -- Diffusion
https://medium.com/@kemalpiro/step-by-step-visual-introduction-to-diffusion-models-235942d2f15c

How to Generate Image?– Continuous Approach – Diffusion


原 PPT 独立图片资源
How to Generate Image?-- Continuous Approach -- Diffusion
https://medium.com/@kemalpiro/step-by-step-visual-introduction-to-diffusion-models-235942d2f15c

Diffusion: Forward Process


原 PPT 独立图片资源
Diffusion: Forward Process
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Diffusion: Backward Process


原 PPT 独立图片资源
Diffusion: Backward Process
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Diffusion: Learning and Loss


原 PPT 独立图片资源
Diffusion: Learning and Loss
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Diffusion: Algorithms


原 PPT 独立图片资源
Diffusion: Algorithms
https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#reverse-diffusion-process
原 PPT 独立图片资源
𝜖𝜃(𝑥,𝑡) is the neural network we need to learn.
Networks with various architectures can be applied.
It could be dependent on other input, such class labels or input texts.

4.2 Perception and Generation 4.2.5 Case Study: Nano Banana



原 PPT 独立图片资源

4.2 Perception and Generation 4.2.5 Case Study: Nano Banana

原 PPT 独立图片资源

Nano Banana Pro


原 PPT 独立图片资源
Nano Banana Pro
https://blog.google/technology/ai/nano-banana-pro/
Infographic about plants using nano banana pro

Nano Banana Pro


原 PPT 独立图片资源
Nano Banana Pro
https://blog.google/technology/ai/nano-banana-pro/
Infographic about making chai made using Nano Banana Pro

Nano Banana Pro


原 PPT 独立图片资源
Nano Banana Pro
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源
原 PPT 独立图片资源

Nano Banana Pro is just Released


原 PPT 独立图片资源
Nano Banana Pro is just Released
原 PPT 独立图片资源

How Nano Banana Pro Does This?


原 PPT 独立图片资源
How Nano Banana Pro Does This?
https://my.infocaptor.com/hub/summaries/google-for-developers/behind-the-scenes-of-google-s-state-of-the-art-nano-banana-image-model-H6ZXujE1qBA?utm_source=chatgpt.com
Evaluation Signal:
“Koshik has been, you know, talking about it for a long time. One of the like biggest advocates of it. Uh, and we were kind of like brushing him off for a long time about how like, you know, this guy‘s a little crazy. Like he’s really obsessed with text rendering. Um, but eventually it kind of became like one of the staple things we looked at. And you can kind of think about it like um when the model learns how to do this like structure for text, it‘s also kind of able to learn other structure in an image as well. And like in an image you have these like um different like frequencies and you can have like structure which you can think of and but you can also have like texture and stuff like this. So it really gives you signal into like how good the model is at generating the structure of the scene.”
“I think humans uh I was actually a little bit skeptical of the human raider approach to doing evals for image generation. But I think what at least I‘ve realized over time is uh when you have enough humans looking at enough prompts across uh a variety of categories, you actually do get quite a bit of good signal. But obviously this is expensive. You don’t want to always be uh asking a bunch of humans uh to grade images. So looking at this text rendering metric for example while a model is training gives you great signal as to whether it‘s performing like you expect. That’s super interesting.”

How Nano Banana Pro Does This?


原 PPT 独立图片资源
How Nano Banana Pro Does This?
https://my.infocaptor.com/hub/summaries/google-for-developers/behind-the-scenes-of-google-s-state-of-the-art-nano-banana-image-model-H6ZXujE1qBA?utm_source=chatgpt.com
Interleaved Generation (Interaction between Understanding and Generation):
So uh one of the magics of interleaf generation is that it offers you to do a new paradigm for image generation right like so if you have a very complex prompt you know you‘re talking about six different edits what if I go with like 50 different edits right so uh now that the model has a really good mechanism to grab information from the context like pixel perfect and use it in the next turn what you can do is you can ask the model to break down the complex prompt either it is editing or for image generation into multiple steps and do edits like one by one over different steps. So for the first one you do this like edits like this five different things and then for the for the next one the next five and so on so forth.
So it‘s like very similar to the test on compute that we have on the language side, right? So, so you spend more flops and and you let the model to bring uh basically this thinking in the uh in the kind of like the pixel space plus breaking it down into smaller pieces that you can like really nail down uh that specific stage but like accumulate it you can do whatever complex task you want right so I think like again this is the magic of interleaf generation that you can think about you know incremental generation of like really complex images as opposed to traditional way of doing it which was like really push pushing hard for getting the best image in one shot, right? Like at the end of the day there‘s a there’s a capacity that you can push the middle. You know, at some point you realize that okay, you know, with 100 details, uh we cannot do that. But when you have this one interle generation breaking it steps, you can always go for any capacity in any complexity that you want to generate. One of the one of the things that's always top of mind for me, especially as like you're uh Nicole, you're also the PM for uh for our imagined models.

Perception, Generation and System State



原 PPT 独立图片资源
Perception and Generation in a Stateful System
原 PPT 独立图片资源
Observe, express, and retain what matters