Vlm Model Enabler

VLM Model Enabler

Enables a new VLM model in the GenAI VLM pipeline. Follows a strict 4-step workflow with checkpoints.

Input

• HuggingFace model_id and task (e.g. google/gemma-3n-E2B-it image-text-to-text), or path to an already exported OpenVINO IR directory.
• Recommended prerequisite: .model_analysis/<model_type>_analysis.md produced by the model-analysis agent. If absent, the caller (or the user, if running this skill standalone) should invoke model-analysis first to avoid duplicating upstream inspection here. This skill does not invoke other agents itself.

Working Directory

All intermediate assets for one model go in .model_enabler/<model_type>/. Determine <model_type> from config.json and create the directory if it does not exist. Never share this directory between different model types; this keeps parallel and resumed enablement runs isolated.

Rules

• Create .py files for all experiments — avoid inline multi-line Python snippets; keep shell commands minimal and only for build/run steps.
• After each step, verify all checkpoint files exist before proceeding. If any are missing, create them.
• Use the todo list tool to track step progress.

Reference

• genai-vlm-architecture.md — GenAI VLM pipeline interfaces and new-model checklist. Read the "Adding a New Model — Checklist" section before Step 2.
• model-inference-text-to-text.md — Supplementary test templates and debugging tips for text-only mode (Step 2).
• model-inference-image-text.md — Supplementary test templates, preprocessing utilities list, and debugging tips for image-text mode (Step 3).

Step 1 — GenAI Enablement Design

Goal: Map the model into the GenAI VLM pipeline using the upstream analysis report.

1.1 Obtain the analysis report

Read .model_analysis/<model_type>_analysis.md.

If it does not exist, prefer asking the caller to invoke the model-analysis agent first (it is the canonical source for upstream facts). If that is not possible — e.g. running this skill standalone without the agent available — produce an equivalent report yourself by following the procedure in .github/agents/model-analysis.agent.md and write the result to the same path. Do not skip this artifact; later steps depend on it.

1.2 Design the mapping

Using the report, decide:

• Closest GenAI model: compare the report's sub-model layout, preprocessing, and special tokens against existing implementations in src/cpp/src/visual_language/*/classes.hpp. Pick the closest one as the reference.
• Required changes: list files to create or modify (enum in vlm_config.hpp, new <model_type>/classes.{hpp,cpp}, factory registrations, etc.). Use genai-vlm-architecture.md "Adding a New Model — Checklist" as the structure.
• Gaps: anything in the report that existing GenAI infrastructure does not cover (custom position IDs, extra LM inputs, dynamic image tiling, etc.).

Checkpoint

Append a ## GenAI Enablement Design section to .model_analysis/<model_type>_analysis.md:

## GenAI Enablement Design
- Closest GenAI model: <name> — because <reason>
- Required changes:
  - <file>: <what changes>
- Gaps: <items needing new infrastructure>

Do not proceed to Step 2 until this section exists.

Step 2 — Text-Only Inference

Goal: VLMPipeline generates correct text with text-only input, matching optimum-intel exactly.

2.1 Implement C++ changes

Follow the checklist in genai-vlm-architecture.md "Adding a New Model":

1. Add enum to vlm_config.hpp, string mapping to vlm_config.cpp
2. Create <model_type>/classes.hpp and <model_type>/classes.cpp
3. Register in vision_encoder.cpp and inputs_embedder.cpp factories
4. Add friend class to inputs_embedder.hpp
5. Implement stubs: VisionEncoder::encode() throws, get_inputs_embeds() handles text-only path
6. Handle any model-specific requirements (custom position IDs, extra LM inputs, etc.)

2.2 Build

For normal local development, install the current checkout with nightly OpenVINO packages:

pip install --pre -U . --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

This is the default validation path. Do not validate edited source through a previously installed OpenVINO GenAI wheel.

After building, verify:

python -c "import openvino, openvino_genai; print(openvino.__version__); print(openvino_genai.__file__)"

Confirm the imported module comes from this checkout's build/install output.

Use an OpenVINO source build only when the change requires it or a native loading/ABI mismatch cannot be resolved with the installed runtime. In that case, build OpenVINO and OpenVINO GenAI from source in the same environment as described in src/docs/BUILD.md; do not treat adding an arbitrary library directory to LD_LIBRARY_PATH as proof of compatibility.

Fix all compilation errors.

2.3 Verify

Create and run .model_enabler/<model_type>/test_text_only_compare.py — compare GenAI vs optimum-intel with do_sample=False on 3 prompts.

Checkpoint

• Build succeeds
• .model_enabler/<model_type>/test_text_only_compare.py shows exact match on all prompts

Do not proceed to Step 3 until text-only output matches optimum-intel exactly.

Step 3 — Image-Text Inference

Goal: VLMPipeline generates correct text with image input, semantically close to optimum-intel.

3.1 Implement vision preprocessing + encoding

Replace the VisionEncoder stub. Key references:

• preprocessor_config.json for resize/normalization params
• clip.hpp/clip.cpp for available resize utilities (bilinear_resize, bicubic_resize, etc.)
• The closest model implementation identified in Step 1

3.2 Implement embedding merge

Update get_inputs_embeds() to handle the non-empty images case: insert vision embeddings at placeholder token positions.

Before reporting a tokenizer/export blocker, compare Hugging Face and OpenVINO tokenization for every required image/chat special token. If the OpenVINO tokenizer drops or splits added tokens, recover only the configured special token IDs in the model-specific GenAI input embedder. This is needed because get_inputs_embeds() uses image-placeholder IDs to locate where vision embeddings must be merged:

1. Read tokenizer.json added tokens and the relevant token IDs from model and processor configuration.
2. Split the normalized prompt around required added-token strings.
3. Tokenize ordinary text spans with the OpenVINO tokenizer and splice the exact added-token IDs back into input_ids.
4. Verify image-placeholder positions exist and their count matches the vision embedding rows before merging.
5. Validate the recovery with real image-text generation and the Optimum comparison.

Do not hard-code a Hub repository ID or silently replace missing tokens. For models exposing image_pad_token_id, assert or explicitly check that the final tokenized input_ids contains that ID and that its occurrence count equals the number of image-embedding rows inserted by the model-specific embedder. A prompt containing only the textual placeholder is not sufficient.

3.3 Build and verify

Rebuild, then create and run:

• .model_enabler/<model_type>/test_image_text_compare.py — compare GenAI vs optimum-intel on 3 image prompts

If output differs, locate the first divergent component rather than judging only decoded text. Compare, in order:

1. chat template and token IDs;
2. processor outputs, shapes, value ranges, layouts, masks, and spatial grids;
3. vision embeddings and projector/merger outputs;
4. image-token insertion positions and counts;
5. language-model inputs, logits, and generated token IDs.

Also inspect effective precision at every stage. Models may use dtype, torch_dtype, nested vision/text precision fields, or cast inputs inside forward(). Verify actual tensor and parameter dtypes; do not assume an fp32 export merely because the command requested fp32.

Checkpoint

• Build succeeds
• .model_enabler/<model_type>/test_image_text_compare.py shows semantically similar outputs

Step 4 — Repository Test Coverage

Every newly enabled model must add repository tests:

1. Find the tiny-random model id:
   • First infer it from the optimum-intel model description, tests, or release notes when available.
   • If it is not documented there, look it up directly on HuggingFace Hub.
   • Matching the optimum-intel-internal-testing/tiny-random-* prefix, for example optimum-intel-internal-testing/tiny-random-gemma4-unified-it.
2. Add the model to VLM Python tests:
   • Prefer extending tests/python_tests/test_vlm_pipeline.py with the tiny-random model id, prompt image tag, video tag if applicable, resolution, and any targeted skip/xfail entry required by an already-tracked issue.
   • Add a dedicated tests/python_tests/test_<model_type>_*.py only when the new model requires behavior that does not fit the shared VLM pipeline suite.
   • Use the existing VLM fixtures and converted-model cache helpers; do not use the full-size model in repository tests.
3. Validate locally:
   • Run the narrow pytest target for the added or modified tests from the activated virtual environment.
   • If the test cannot run locally, document the exact command, blocker, and expected CI coverage.

Apply model-specific dependency entries consistently to the relevant Linux, manylinux, and Windows VLM workflows. Do not update a shared dependency for all models when only the newly enabled architecture requires it.

Before adding a tiny-model ID to a test matrix, verify that it exists, is accessible without a developer's private cache, preserves the real architecture identity, and can execute the requested generation path. Never add a guessed or not-yet-published Hub ID and then treat HTTP 401/403/404 as a passing test. If repository policy requires a hosted fixture and publication is unavailable, report a test-fixture blocker and leave enablement incomplete. If the test infrastructure supports a deterministic local constructor, prefer that over a newly uploaded model and cache it using repository conventions.

Do not report successful enablement after GenAI source changes unless the test is added and its exact pytest result is recorded. If local execution is blocked, add the test and report the exact blocker and expected CI coverage. Confirm that the narrow pytest command collected at least one test; a passing command with every test deselected is not validation.

Final Deliverables

Before declaring the model enabled:

• .model_analysis/<model_type>_analysis.md exists with the ## GenAI Enablement Design section
• All test scripts exist and pass in .model_enabler/<model_type>/
• Tiny-random repository coverage is added under tests/python_tests/
• Narrow pytest command/result, or the exact local blocker, is recorded
• git diff --name-only contains the intended source, test, and docs files only
• No debug prints, scratch artifacts, local absolute paths, or unrelated edits remain