Handbooks / Generative AI & LLMs / Chapter 10

Multimodal Models

7 pages · ~5 min✓ Reviewed

Builds on Model Serving and Inference Optimization.

Generative AI & LLMsPage 1

Multimodal Models: The Mental Model

1. The VLM Idea

VLM: image + text in, text out
Image becomes embeddings the LLM reads like extra tokens
Three parts: vision encoder (usually a ViT), projector, LLM decoder
Projector: linear layer or MLP, sometimes cross-attention

Vision encoderViT
Projectorlinear / MLP
LLM decodertext out

2. Fusion Styles

	Early fusion	Cross-attention
Example	LLaVA	Flamingo
Image tokens	Join the LLM sequence	Kept separate, attended across

3. One Shared Space

Modality	Pipeline
Image	patches → embeddings
Audio	spectrogram → embeddings
Text	subword tokens → embeddings

Projector

It maps vision features into the LLM embedding dimension, so image vectors fit the decoder.

4. Interview Point

Three VLM parts? Vision encoder, projector, LLM decoder
Early vs cross-attention? Tokens join the sequence vs separate streams attended across
Why project? Vision features must match the LLM embedding dimension

Generative AI & LLMsPage 2

Image Tokens, Patches and Contrastive Training

1. Patches Become Tokens

Splitfixed 16x16 patches
Flattenpixels to vector
Projectlinear layer
Add positionembeddings

Each patch = one image token
Tokens = (H/P) x (W/P)
Double the resolution: 4x tokens and far more attention cost

Image	Patch	Grid	Tokens
224x224	16x16	14x14	196
336x336	14x14	24x24	576
448x448	16x16	28x28	784

2. CLIP Contrastive Loss

Image and text encoders trained jointly on image-caption pairs
Matching pairs get high cosine similarity in a shared space
Batch of N: N diagonal positives, N²-N negatives

Generative AI & LLMsPage 3

Diffusion Basics for Image Generation

1. Forward and Reverse

Clean imagex0
Add Gaussian noiseT steps
Pure noisexT
Learned denoiserstep by step

Forward: add Gaussian noise over T steps until pure noise
Reverse: a learned denoiser removes noise step by step
Train: random timestep t, noise a clean image, predict the added noise
Loss = MSE between predicted and true noise; net is a U-Net or transformer

2. Latent Diffusion

Runs in a compressed VAE latent space, not pixel space
Far cheaper than pixel diffusion (Stable Diffusion)
The VAE decoder turns the final latent into an image

3. Conditioning and Sampling

Text encoder (CLIP or T5) feeds cross-attention in the denoiser
Classifier-free guidance blends conditional and unconditional predictions

Knob	Effect

Generative AI & LLMsPage 4

Speech-to-Text and Text-to-Speech

1. Speech to Text

Waveform
Log-melspectrogram
Encoder-decodere.g. Whisper
Text tokens

Whisper wants 16 kHz mono audio in 30 s windows
WER = (substitutions + deletions + insertions) / reference words
Lower is better; the standard ASR metric

2. Text to Speech

Normalize textnumbers, abbreviations
Acoustic modelmel or audio tokens
VocoderHiFi-GAN / neural codec
Waveform

The vocoder turns a mel spectrogram or audio tokens into the waveform

3. Streaming, Quality, Ethics

	Streaming	Batch
Output	Partial transcripts	Final transcript
Latency	Low	Higher
Accuracy

Generative AI & LLMsPage 5

Multimodal Prompting Tips and Common Mistakes

1. Prompt Layout

Image first, then the question: 'in the chart above'
Several images: label them Image 1, Image 2
Ask for JSON fields, a table, or transcribe-then-answer, not 'describe this'

text

Image 1: [sales chart]
Transcribe the axis labels, then answer.
Reply as JSON: {"peak_month": str}

label, transcribe, structured output

2. Resolution and Cost

Problem	Fix
Small text, thin lines, dense tables	Crop or tile the region
Many tokens, cost, latency	Resize to needed detail
Exact characters matter	OCR first, then the VLM

3. Common Mistakes

VLMs hallucinate: verify counts, spatial relations, fine reading
Image generation: not 'don't mention X', use a negative prompt
STT: check sample rate and noise first

4. Interview Point

Generative AI & LLMsPage 6

Multimodal Cheat Sheet

1. Vision and CLIP

Item	Rule
Patch tokens	(H/P) x (W/P)
VLM	vision encoder + projector + LLM
CLIP	contrastive loss on NxN similarity matrix, learned temperature

2. Diffusion

Forward: add noise; reverse: predict noise
Latent diffusion denoises in a VAE latent
Text enters by cross-attention
Higher guidance scale: better prompt fit, less diversity

3. Speech and Prompting

	Pipeline	Key fact
ASR	log-mel → encoder-decoder (Whisper)	16 kHz, 30 s windows, WER
TTS	acoustic model → vocoder	vocoder makes the waveform

Generative AI & LLMsPage 7

Check yourself

A ViT with 14x14 patches reads a 224x224 image. You switch to 448x448. How many image tokens now?

224: 16x16 = 256 tokens. 448: 32x32 = 1024 tokens.
Doubling resolution gives 4x tokens, so attention costs far more.

A CLIP batch holds 8 image-caption pairs. How many negative pairs are in the similarity matrix?

The matrix is 8x8 = 64 entries.
8 diagonal positives leave 64 - 8 = 56 negatives.

The prompt "a street, no cars" produces an image full of cars. Why, and what is the fix?

The text encoder embeds the word "cars", so it pushes the image toward cars.
Move it to the negative prompt field instead.

Multimodal Models

1. The VLM Idea

2. Fusion Styles

3. One Shared Space

4. Interview Point

1. Patches Become Tokens

2. CLIP Contrastive Loss

1. Forward and Reverse

2. Latent Diffusion

3. Conditioning and Sampling

1. Speech to Text

2. Text to Speech

3. Streaming, Quality, Ethics

1. Prompt Layout

2. Resolution and Cost

3. Common Mistakes

4. Interview Point

1. Vision and CLIP

2. Diffusion

3. Speech and Prompting

3. Zero-Shot Classification

4. Interview Point

4. Interview Point