Vision Language Models

Vision-Language Models (VLM) Inference

When to Use

• Captioning or describing an image, micrograph, gel, or scientific figure in natural language
• Visual question answering (VQA): "what does this plot/chart/image show?"
• Zero-shot image classification with CLIP when no labeled training set exists
• Extracting structured data (axis values, trends, labels) from a chart or figure image
• Reasoning across multiple images in one prompt (e.g. before/after, replicate gels, time series)

Not for: retrieval over many document pages (use ai-science-vision-rag for ColPali-style page retrieval + RAG) or image generation (use generative-imaging).

Version Compatibility

transformers >=4.45, torch >=2.1, qwen-vl-utils >=0.0.8, Python >=3.10. Models: Qwen/Qwen2-VL-2B-Instruct / -7B-Instruct, llava-hf/llava-v1.6-mistral-7b-hf, OpenGVLab/InternVL2-8B, openai/clip-vit-base-patch32.

Prerequisites

pip install transformers accelerate torch pillow qwen-vl-utils

Familiarity with PIL Image objects and the HF AutoModel/AutoProcessor API. A GPU with >=8GB VRAM is needed for 7B-class VLMs (2B variants and CLIP run on CPU, slowly); 4-bit quantization (bitsandbytes) roughly halves that requirement.

Zero-Shot Image Classification (CLIP)

Goal: classify an image against an arbitrary, unlabeled set of candidate text descriptions. Approach: encode the image and each candidate label with CLIP's joint embedding space, then rank labels by cosine similarity — no fine-tuning or training data required.

import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

def zero_shot_classify(image: Image.Image, candidate_labels: list[str]) -> list[tuple[str, float]]:
    """Rank candidate_labels by CLIP similarity to image; returns (label, prob) sorted desc."""
    inputs = processor(text=candidate_labels, images=image, return_tensors="pt", padding=True)
    with torch.no_grad():
        outputs = model(**inputs)
    probs = outputs.logits_per_image.softmax(dim=1)[0]  # (n_labels,)
    ranked = sorted(zip(candidate_labels, probs.tolist()), key=lambda x: -x[1])
    return ranked

# image = Image.open("microscopy_field.png")
# ranked = zero_shot_classify(image, ["fluorescent nuclei stain", "brightfield tissue section", "western blot"])

Captioning and VQA (Qwen2-VL)

Goal: describe an image or answer a free-text question about it, including multi-image comparisons. Approach: build a chat-style message with {"type": "image", ...} and {"type": "text", ...} content blocks, apply the processor's chat template, and generate greedily for factual (low-hallucination) answers.

import torch
from PIL import Image
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

def describe_image(image: Image.Image, question: str = "Describe this image in detail.") -> str:
    """Answer a question about a single image (captioning, VQA)."""
    messages = [{"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": question},
    ]}]
    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    image_inputs, _ = process_vision_info(messages)
    inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)
    with torch.no_grad():
        ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
    return processor.decode(ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)

def compare_images(images: list[Image.Image], question: str) -> str:
    """Reason jointly over multiple images (e.g. before/after, replicate gels, time series)."""
    content = [{"type": "image", "image": img} for img in images]
    content.append({"type": "text", "text": question})
    messages = [{"role": "user", "content": content}]
    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    image_inputs, _ = process_vision_info(messages)
    inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)
    with torch.no_grad():
        ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
    return processor.decode(ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)

Extracting Structured Data from Charts/Figures

Goal: turn a chart/plot image into machine-readable values instead of prose. Approach: instruct the VLM to answer strictly in JSON, then parse defensively — VLMs occasionally wrap JSON in prose or markdown fences.

import json
import re

def extract_chart_data(image: Image.Image, fields: list[str]) -> dict:
    """Ask the VLM to read `fields` off a chart image and return them as parsed JSON."""
    prompt = (
        f"Read this chart and return ONLY a JSON object with keys: {fields}. "
        "No prose, no markdown fences."
    )
    raw = describe_image(image, prompt)
    match = re.search(r"\{.*\}", raw, re.DOTALL)  # strip any accidental fencing/prose
    if not match:
        raise ValueError(f"No JSON found in VLM output: {raw!r}")
    return json.loads(match.group(0))

# extract_chart_data(chart_img, ["x_axis_label", "y_axis_label", "peak_value", "trend"])

Pitfalls

• Hallucination on structured extraction: VLMs invent plausible-looking numbers for low-quality or cropped charts — always spot-check extract_chart_data output against the source image, and use do_sample=False (greedy decoding) for factual tasks.
• Memory: Qwen2-VL-7B needs ~16GB GPU RAM in bf16, ~8GB in 4-bit (load_in_4bit=True via bitsandbytes); use the 2B variant or CLIP-only workflows on CPU.
• Resolution vs cost tradeoff: Qwen2-VL uses dynamic resolution (min_pixels/max_pixels on the processor) — higher resolution improves fine text/axis-label reading but each image costs more visual tokens and slows generation.
• CLIP is not generative: it only scores image-label similarity; use it for classification/retrieval, not captioning or VQA — use Qwen2-VL/LLaVA for those.
• This is not OCR: for dense, small-font text (e.g. full-page scanned documents) a dedicated OCR pipeline or ai-science-vision-rag's page-retrieval approach is more reliable than asking a VLM to transcribe everything.

See Also

• ai-science-vision-rag — ColPali-style page retrieval + RAG for multi-page document QA
• generative-imaging — image generation/diffusion counterpart
• ai-science-llm-finetuning — fine-tuning a VLM/LLM for domain-specific tasks
• document-rag — text-based RAG pipeline for non-visual documents