Model Checker

Model Checker

Validates that a HuggingFace model exported via optimum-intel works correctly with OpenVINO GenAI pipelines and passes accuracy benchmarks.

When to Use

• A new model was added to optimum-intel and needs GenAI validation
• Verify a HuggingFace model exports to OpenVINO IR and runs inference
• Check model accuracy after conversion using who-what-benchmark

Inputs

The user must provide:

• model_id: HuggingFace model identifier (e.g. tencent/HY-MT1.5-1.8B) or path to an existing OpenVINO IR directory (e.g. /tmp/my_model_ir). When a local path is provided, export is skipped automatically.
• task: optimum-cli export task. Supported values:
  • text-generation-with-past
  • image-text-to-text
  • text-to-image
  • image-to-image
  • feature-extraction
  • text-classification
  • text-to-video
  • automatic-speech-recognition

Prerequisites

Ensure the Python virtual environment is activated before running any commands.

1. Locate the virtual environment — check for common directories at the repository root: .venv/, venv/, env/. Use list_dir to find it. If none is found, ask the user for its location.
2. Check if already activated: if which python or where python points inside the virtual environment, it's already activated. If not, proceed to activate it.
3. Activate based on the current platform:
   • Linux/macOS: source <venv_path>/bin/activate
   • Windows (cmd): <venv_path>\Scripts\activate.bat
   • Windows (PowerShell): <venv_path>\Scripts\Activate.ps1
4. The background terminal doesn't inherit the venv activation. Run it with the venv activated in the same command.

Procedure

Step 1: Run check_model.py

Run the checker script from the repository root:

python3 .github/skills/model-checker/scripts/check_model.py \
    --model-id <model_id_or_path> \
    --task <export_task> \
    --work-dir .model_enabler/<model_type>/model_checker

Determine <model_type> from the model's config.json. When --model-id is a path to an existing directory, the script treats it as a pre-converted OpenVINO IR model and automatically skips the export step. Keep each model type in its own work directory so parallel or resumed enablement runs do not overwrite one another's artifacts.

Run python3 .github/skills/model-checker/scripts/check_model.py --help for the full argument reference including defaults. The --work-dir is where all intermediate files, logs, and outputs will be stored. Do not pipe with any additional logging or redirection — the script handles its own logging.

Skip flags (for re-runs after a fix)

When a previous run already passed some steps (e.g. export succeeded but inference test failed), use skip flags to avoid repeating expensive passed steps:

• --skip-export — reuse existing IR in <work-dir>/model_ir instead of re-exporting (avoids re-downloading weights). When --model-id is a local path, export is bypassed automatically and that directory is used directly instead of <work-dir>/model_ir.
• --skip-llm-bench — skip the llm_bench inference test
• --skip-wwb — skip the who-what-benchmark accuracy check
• --skip-wwb-ground-truth — skip WWB ground-truth collection and reuse <work-dir>/wwb/gt.csv; useful when iterating on GenAI target evaluation after ground truth was already collected

Do not use skip flags on the first run. Only use them when retrying after a targeted fix.

Step 2: Interpret Results

The script logs progress for each step and exits with code 0 (pass) or non-zero (fail).

Pass criteria:

• Export: exit code 0

• Inference test (llm_bench): exit code 0, metrics line logged

• WWB accuracy (depends on --wwb-base mode):

  • --wwb-base optimum (default):
    1. Optimum ground truth generation: exit code 0
    2. GenAI target evaluation: similarity ≥ SIMILARITY_THRESHOLD
  • --wwb-base hf:
    1. HF ground truth generation: exit code 0
    2. Optimum target evaluation: similarity ≥ SIMILARITY_THRESHOLD
    3. GenAI target evaluation: similarity ≥ SIMILARITY_THRESHOLD

  Note: the WWB step is skipped automatically for automatic-speech-recognition (no WWB support).

Log files: each tool writes its own dedicated log; paths are printed during execution. When a step fails, read the corresponding log for the full traceback and context before drawing any conclusions.

work-dir: work-dir is in current workspace, prefer to use tool calls to access logs and outputs instead of custom bash commands.

Step 3: Report Results

Results format:

• Model: <model_id> (<task>)
• Validation: PASSED / FAILED
• Performance (if passed):
  • 1st token latency, 2nd token latency, throughput
  • Optimum similarity / GenAI similarity (if applicable)
• Logs: paths to export log, llm_bench log, WWB logs
• Failed step analysis (if failed): summary of the failure and relevant log path for details

Security

• NEVER invoke optimum-cli, wwb, or llm_bench directly. Always go through check_model.py.
• NEVER modify model_id — pass it exactly as provided by the user.