# Vision Language Models

> Run VLM inference (Qwen2-VL, LLaVA, CLIP) via Transformers: captioning, VQA, zero-shot classification, chart data extraction. Use when captioning images, answering visual questions, or classifying images with no labels.

- Skill: `pavel-kravchenko/vision-language-models` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pavel-kravchenko/vision-language-models`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pavel-kravchenko/vision-language-models/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: pavel-kravchenko (https://skillmd.com/u/pavel-kravchenko)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pavel-kravchenko/vision-language-models

---


# Vision-Language Models (VLM) Inference

## When to Use

- Captioning or describing an image, micrograph, gel, or scientific figure in natural language
- Visual question answering (VQA): "what does this plot/chart/image show?"
- Zero-shot image classification with CLIP when no labeled training set exists
- Extracting structured data (axis values, trends, labels) from a chart or figure image
- Reasoning across multiple images in one prompt (e.g. before/after, replicate gels, time series)

Not for: retrieval over many document pages (use `ai-science-vision-rag` for ColPali-style page retrieval + RAG) or image generation (use `generative-imaging`).

## Version Compatibility

`transformers` >=4.45, `torch` >=2.1, `qwen-vl-utils` >=0.0.8, Python >=3.10. Models: `Qwen/Qwen2-VL-2B-Instruct` / `-7B-Instruct`, `llava-hf/llava-v1.6-mistral-7b-hf`, `OpenGVLab/InternVL2-8B`, `openai/clip-vit-base-patch32`.

## Prerequisites

```bash
pip install transformers accelerate torch pillow qwen-vl-utils
```

Familiarity with PIL `Image` objects and the HF `AutoModel`/`AutoProcessor` API. A GPU with >=8GB VRAM is needed for 7B-class VLMs (2B variants and CLIP run on CPU, slowly); 4-bit quantization (`bitsandbytes`) roughly halves that requirement.

## Zero-Shot Image Classification (CLIP)

**Goal:** classify an image against an arbitrary, unlabeled set of candidate text descriptions.
**Approach:** encode the image and each candidate label with CLIP's joint embedding space, then rank labels by cosine similarity — no fine-tuning or training data required.

```python
import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

def zero_shot_classify(image: Image.Image, candidate_labels: list[str]) -> list[tuple[str, float]]:
    """Rank candidate_labels by CLIP similarity to image; returns (label, prob) sorted desc."""
    inputs = processor(text=candidate_labels, images=image, return_tensors="pt", padding=True)
    with torch.no_grad():
        outputs = model(**inputs)
    probs = outputs.logits_per_image.softmax(dim=1)[0]  # (n_labels,)
    ranked = sorted(zip(candidate_labels, probs.tolist()), key=lambda x: -x[1])
    return ranked

# image = Image.open("microscopy_field.png")
# ranked = zero_shot_classify(image, ["fluorescent nuclei stain", "brightfield tissue section", "western blot"])
```

## Captioning and VQA (Qwen2-VL)

**Goal:** describe an image or answer a free-text question about it, including multi-image comparisons.
**Approach:** build a chat-style message with `{"type": "image", ...}` and `{"type": "text", ...}` content blocks, apply the processor's chat template, and generate greedily for factual (low-hallucination) answers.

```python
import torch
from PIL import Image
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

def describe_image(image: Image.Image, question: str = "Describe this image in detail.") -> str:
    """Answer a question about a single image (captioning, VQA)."""
    messages = [{"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": question},
    ]}]
    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    image_inputs, _ = process_vision_info(messages)
    inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)
    with torch.no_grad():
        ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
    return processor.decode(ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)

def compare_images(images: list[Image.Image], question: str) -> str:
    """Reason jointly over multiple images (e.g. before/after, replicate gels, time series)."""
    content = [{"type": "image", "image": img} for img in images]
    content.append({"type": "text", "text": question})
    messages = [{"role": "user", "content": content}]
    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    image_inputs, _ = process_vision_info(messages)
    inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)
    with torch.no_grad():
        ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
    return processor.decode(ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
```

## Extracting Structured Data from Charts/Figures

**Goal:** turn a chart/plot image into machine-readable values instead of prose.
**Approach:** instruct the VLM to answer strictly in JSON, then parse defensively — VLMs occasionally wrap JSON in prose or markdown fences.

```python
import json
import re

def extract_chart_data(image: Image.Image, fields: list[str]) -> dict:
    """Ask the VLM to read `fields` off a chart image and return them as parsed JSON."""
    prompt = (
        f"Read this chart and return ONLY a JSON object with keys: {fields}. "
        "No prose, no markdown fences."
    )
    raw = describe_image(image, prompt)
    match = re.search(r"\{.*\}", raw, re.DOTALL)  # strip any accidental fencing/prose
    if not match:
        raise ValueError(f"No JSON found in VLM output: {raw!r}")
    return json.loads(match.group(0))

# extract_chart_data(chart_img, ["x_axis_label", "y_axis_label", "peak_value", "trend"])
```

## Pitfalls

- **Hallucination on structured extraction**: VLMs invent plausible-looking numbers for low-quality or cropped charts — always spot-check `extract_chart_data` output against the source image, and use `do_sample=False` (greedy decoding) for factual tasks.
- **Memory**: Qwen2-VL-7B needs ~16GB GPU RAM in bf16, ~8GB in 4-bit (`load_in_4bit=True` via `bitsandbytes`); use the 2B variant or CLIP-only workflows on CPU.
- **Resolution vs cost tradeoff**: Qwen2-VL uses dynamic resolution (`min_pixels`/`max_pixels` on the processor) — higher resolution improves fine text/axis-label reading but each image costs more visual tokens and slows generation.
- **CLIP is not generative**: it only scores image-label similarity; use it for classification/retrieval, not captioning or VQA — use Qwen2-VL/LLaVA for those.
- **This is not OCR**: for dense, small-font text (e.g. full-page scanned documents) a dedicated OCR pipeline or `ai-science-vision-rag`'s page-retrieval approach is more reliable than asking a VLM to transcribe everything.

## See Also

- `ai-science-vision-rag` — ColPali-style page retrieval + RAG for multi-page document QA
- `generative-imaging` — image generation/diffusion counterpart
- `ai-science-llm-finetuning` — fine-tuning a VLM/LLM for domain-specific tasks
- `document-rag` — text-based RAG pipeline for non-visual documents