Vision Language Models

Vision-Language Models = image/video and text pretraining

• Image Encoder: Converts images into dense feature embeddings.
• Embedding Projector: Aligns image and text embeddings into a common space.
• Text Encoder / Decoder: Processes or generates textual output based on the image features.

Dual Encoder (Contrastive Models)

• Separate encoders for image and text.
• Project embeddings into a shared space.
• Optimized using contrastive learning, e.g., InfoNCE:

$${\mathcal{L}}_{\text{contrastive}}=-\frac{1}{N}\sum _i\log \frac{\exp (\text{sim}(v_i,t_i)/\tau )}{{\sum }_j\exp (\text{sim}(v_i,t_j)/\tau )}$$

where $v_i$ and $t_i$ are image and text embeddings, $\text{sim}$ is cosine similarity, and $\tau$ is a temperature hyperparameter.

• Advantages: Fast retrieval, scalable.
• Examples: CLIP, ALIGN.

Encoder-Decoder (Generative Models)

• Image encoder extracts visual features.
• Text decoder generates output based on features.
• Optimized with cross-entropy loss for text generation.
• Useful for captioning, VQA.
• Examples: BLIP, OFA, Flamingo.

Unified Transformer Models

• Images and text tokens processed in a single transformer.
• Supports multimodal understanding and generation.
• Can perform instruction-following and reasoning tasks.
• Examples: Kosmos-2, GPT-4 multimodal.

Image Encoders

• CNN-based: ResNet, EfficientNet – good for local features.
• ViT-based: Vision Transformers – capture global context.
• Output: feature vectors or patch embeddings.

Text Encoders / Decoders

• Text encoders: BERT, RoBERTa – produce text embeddings.
• Text decoders: GPT-style autoregressive transformers.
• Can be autoregressive (predict next token) or masked language modeling (predict masked tokens).

Alignment Techniques

Contrastive Loss (as above) aligns image and text embeddings.

Cross-attention allows text queries to attend to image features:

$$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$

• Images provide keys ($K$) and values ($V$), text queries ($Q$) attend to them.
• Fine-grained alignment is crucial for generation tasks.

Pretraining Objectives

• Contrastive pretraining: Learn a shared embedding space.
• Captioning / Generation: Learn to generate text from images.
• Masked multimodal modeling: Mask parts of image or text and predict them.
• Instruction tuning: Train on vision-language tasks with natural language prompts.