Frontier · Intermediate
Multimodal AI
Vision-language models, image and audio understanding, and building apps that mix modalities.
Why it matters
Frontier models read images, documents and audio natively. Many real tasks, such as forms, charts and screenshots, are multimodal.
Key ideas
- Vision transformers
- Contrastive image-text training
- Vision-language models
- Document and chart understanding
- Speech in and out
You will be able to
- Extract data from documents
- Evaluate a VLM on your images
- Design a multimodal pipeline
Primary reading
- An Image is Worth 16x16 Words (ViT) (opens in a new tab)Dosovitskiy et al. · 2020
- Learning Transferable Visual Models From Natural Language Supervision (CLIP) (opens in a new tab)Radford et al. · 2021