# Model Verify Flagos

> Verify the serving stack with a user-specified target model. Runs twice: first with FlagGems/FlagCX disabled (isolate model-specific errors), then with full multi-chip stack enabled. Diffs the two runs to pinpoint which layer caused any failure.

- Skill: `flagos-ai/model-verify-flagos` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add flagos-ai/model-verify-flagos`
- Raw SKILL.md: https://api.skillmd.com/api/skills/flagos-ai/model-verify-flagos/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: flagos-ai (https://skillmd.com/u/flagos-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/flagos-ai/model-verify-flagos

---


<!--
 Copyright 2026 FlagOS Contributors

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->


# Target Model Verification

Same layer-peeling approach as env-verify, but with the real target model that may
require multi-GPU tensor parallelism. Runs the model twice (without and with
multi-chip stack) and diffs results to isolate failures.

## Skill Components

```
model-verify/
├── SKILL.md                            # This file — execution flow
├── scripts/
│   └── diff_analysis.py                # Compare Run A vs Run B, classify errors (JSON)
└── references/
    └── multichip-errors.md             # Multi-chip error patterns and diff truth table
```

**Reused from env-verify:**
- `env-verify/scripts/run_offline_inference.py` — Phase A test (parameterized)
- `env-verify/scripts/test_serve_mode.py` — Phase B test (parameterized)
- `env-verify/references/error-classification.md` — Layer-based error rules

## Prerequisites

- Running container with software stack installed (from `install-stack`)
- env-verify completed (at least Phase A passed)
- User must provide model path

If invoked standalone, ask for container name, vendor, and model path.
If invoked from `/flagrelease`, these are passed as context.

## Execution Flow

### Step 1: Get Model Info from User

**Ask the user for** (use AskUserQuestion if not provided):

1. **Model path** (required) — local path inside container OR ModelScope/HuggingFace ID
2. **--tensor-parallel-size** (optional) — default to GPU count
3. **Additional vllm args** (optional)

Get default TP size:
```bash
docker exec <CONTAINER> python3 -c "
import torch; print(torch.cuda.device_count() if torch.cuda.is_available() else 1)
"
```

**If user does not provide model path → ask and wait. Do not guess.**

### Step 2: Download Model (if needed)

If model path is a remote ID (not starting with `/`):

```bash
docker exec <CONTAINER> python3 -c "
from modelscope import snapshot_download
snapshot_download('<MODEL_ID>', local_dir='/data/models/<MODEL_NAME>')
"
```

If local directory, verify `config.json` exists:
```bash
docker exec <CONTAINER> test -f <MODEL_PATH>/config.json
```

**Timeout:** 600s for large model downloads.

### Step 3: Run A — WITHOUT Multi-Chip Stack

Copy the test scripts from env-verify into the container (if not already there):

```bash
docker cp <ENV_VERIFY_DIR>/scripts/run_offline_inference.py <CONTAINER>:/tmp/
docker cp <ENV_VERIFY_DIR>/scripts/test_serve_mode.py <CONTAINER>:/tmp/
```

**Phase A (offline):**
```bash
docker exec <CONTAINER> bash -c '
export USE_FLAGGEMS=0
unset FLAGCX_PATH
timeout 300 python3 /tmp/run_offline_inference.py \
    --model <MODEL_PATH> \
    --tp <TP_SIZE> \
    --trust-remote-code
' > /tmp/run_a_offline.json
```

**Phase B (serve):**
```bash
docker exec <CONTAINER> bash -c '
export USE_FLAGGEMS=0
unset FLAGCX_PATH
timeout 360 python3 /tmp/test_serve_mode.py \
    --model <MODEL_PATH> \
    --tp <TP_SIZE> \
    --trust-remote-code \
    --health-timeout 300
' > /tmp/run_a_serve.json
```

### Step 4: Run B — WITH Full Multi-Chip Stack

**Skip logic:** If ALL of FlagGems, FlagTree, FlagCX failed install → skip Run B.
Report: "Run B skipped: no multi-chip packages installed."
Check install-stack results to decide.

**Phase A (offline):**
```bash
docker exec <CONTAINER> bash -c '
export USE_FLAGGEMS=1
export FLAGCX_PATH=/tmp/FlagCX
export VLLM_PLUGINS=fl
timeout 300 python3 /tmp/run_offline_inference.py \
    --model <MODEL_PATH> \
    --tp <TP_SIZE> \
    --trust-remote-code
' > /tmp/run_b_offline.json
```

**Phase B (serve):**
```bash
docker exec <CONTAINER> bash -c '
export USE_FLAGGEMS=1
export FLAGCX_PATH=/tmp/FlagCX
export VLLM_PLUGINS=fl
timeout 360 python3 /tmp/test_serve_mode.py \
    --model <MODEL_PATH> \
    --tp <TP_SIZE> \
    --trust-remote-code \
    --health-timeout 300
' > /tmp/run_b_serve.json
```

### Step 5: Diff Analysis

Copy and run `scripts/diff_analysis.py` to compare the two runs:

```bash
docker cp <SKILL_DIR>/scripts/diff_analysis.py <CONTAINER>:/tmp/
docker exec <CONTAINER> python3 /tmp/diff_analysis.py \
    --run-a /tmp/run_a_offline.json \
    --run-b /tmp/run_b_offline.json
```

Read `references/multichip-errors.md` to interpret the diff and classify errors.

### Step 6: Produce Report

```json
{
  "status": "PASS | PARTIAL | FAIL",
  "stage": "model-verify",
  "model": "<MODEL_PATH>",
  "tensor_parallel_size": 8,
  "run_a_without_multichip": {
    "flags": {"USE_FLAGGEMS": "0", "FLAGCX_PATH": "unset"},
    "phase_a_offline": "PASS | FAIL",
    "phase_b_serve": "PASS | FAIL",
    "output_sample": "...",
    "errors": []
  },
  "run_b_with_multichip": {
    "flags": {"USE_FLAGGEMS": "1", "FLAGCX_PATH": "/tmp/FlagCX"},
    "skipped": false,
    "phase_a_offline": "PASS | FAIL",
    "phase_b_serve": "PASS | FAIL",
    "output_sample": "...",
    "errors": []
  },
  "diff_analysis": {
    "conclusion": "BOTH_PASS | MULTICHIP_ERROR | SAME_ERROR | DIFFERENT_ERRORS",
    "detail": "...",
    "multichip_component": "FlagGems | FlagTree | FlagCX | plugin | null",
    "recommended_stack": "full | base | none"
  }
}
```

**`recommended_stack`** — tells downstream skills which stack to use:
- `full` — Run B passed (USE_FLAGGEMS=1, FLAGCX_PATH set)
- `base` — only Run A passed (USE_FLAGGEMS=0, FLAGCX_PATH unset)
- `none` — Run A also failed (model can't serve)

**Status logic:**
- `PASS` — both Run A and Run B succeed
- `PARTIAL` — Run A passes, Run B fails
- `FAIL` — Run A fails

## Error Handling

| Failure | Behavior |
|---------|----------|
| Model path not provided | Ask user, wait |
| Model path not found | Report exact path, exit with error |
| Model too large for memory | Report OOM, suggest reducing TP or dtype |
| TP > available GPUs | Report "requested TP=X but only Y available" |
| Server hangs | Kill after timeout, capture last logs |
| Run A and Run B both fail | Report both errors separately |

**Rule:** Run BOTH runs regardless of individual failures. Maximize error coverage.

## Timeout Rules

| Operation | Timeout |
|-----------|---------|
| Model download | 600s |
| Phase A (offline) | 300s |
| Phase B (serve + test) | 360s |