Multimodal Large Language Models

What is Multimodal?

Multimodal = A single model that processes multiple types of data:

• Text
• Images
• Audio (some models)
• Video (some models)

Text-only LLM:
Input: "What's in this image?" 
Output: "I can't see images"

Multimodal LLM:
Input: [Image] "What's in this image?"
Output: "The image shows a cat sleeping on a bed"

Multimodal Models

GPT-4V (OpenAI)

Capabilities:
- Read text from images (OCR)
- Describe what's in images
- Answer questions about images
- Read charts, diagrams, graphs
- Understand layouts

Example:
User: [Image of menu] "What's the most expensive item?"
GPT-4V: [Reads menu, analyzes] "The lobster at $45"

Claude 3 (Anthropic)

3 versions:
- Opus: Most capable (slower, expensive)
- Sonnet: Balanced
- Haiku: Fast, cheap

Can analyze images, read documents, understand layouts

Other Models

LLaVA: Open-source, multimodal
Gemini (Google): Text + image + code
Qwen-VL: Open-source vision-language

Architecture: How Vision × Language Works

Image
  ↓
Vision Encoder (like ViT - Vision Transformer)
  ↓
Image embeddings
  ↓
LLM (text processor)
  ↓
Text output

Example:
[Dog image] → Vision encoder → [visual embedding] → LLM → "This is a golden retriever"

Key Innovation: Vision-Language Alignment

Training multimodal models:

1. Take image
2. Get image embedding (from vision model)
3. Get text description
4. Get text embedding (from language model)
5. Train to align: image embedding ≈ text embedding

This alignment allows:

• Image-to-text (captioning)
• Text-to-image search
• VQA (Visual Question Answering)

Real-World Applications

Document Understanding

Upload: Invoice, contract, form
Query: "Extract customer name and total amount"
Multimodal LLM: "Customer: John Doe, Total: $1,234.56"

Better than OCR because it understands context!

Medical Imaging Analysis

Upload: X-ray, MRI scan
Query: "What abnormalities do you see?"
LLM: "There appears to be... (medical analysis)"

Note: Current models aren't certified for medical use - need expert review

E-commerce Product Analysis

Upload: Product image
Query: "Describe this product in 50 words for a product listing"
LLM: "Premium leather handbag with spacious interior..."

Accessibility

Image description for blind users:
Image: [Photo of sunset]
Multimodal LLM: "A stunning sunset over the ocean with golden and pink clouds"

Challenges

Hallucinations in Vision

Image: A red car
Multimodal LLM: "This is a blue car" (wrong color)

More common in images than text!

Context Length with Images

Images take many tokens to encode
- 1 image = 1000-5000 tokens
- Limits how many images in one request

Solutions:
- Compress images
- Multiple API calls
- New models with longer context (Gemini 1.5: 1M tokens!)

Cost

Processing images is expensive (more tokens)
GPT-4V: $0.01-0.03 per image

Fine-tuning: $0.012-0.018 per 1M tokens
(Much more expensive than text-only)

Future: Unified Multimodal AI

Coming soon:
- Audio understanding (transcribe, answer questions about audio)
- Video understanding (understand video content)
- 3D understanding (process 3D models, point clouds)
- Real-time streaming (live video input)

Vision: One model that truly understands all modalities