# H3 Multimodal Lora Training

> Plan, audit, configure, run when authorized, evaluate, and package MiniMax H3 LoRA training for images, video, stereo audio, motion, and true Ref2VA reference conditioning. Use when a user wants an auditable H3 LoRA dataset or training run; do not use for prompt-only Hailuo generation or unrelated image-model LoRA training.

- Skill: `chengyansen-ai/h3-multimodal-lora-training` (Agent Skill, multi-file: 10 files)
- Install (CLI): `npx skillmds@latest add chengyansen-ai/h3-multimodal-lora-training`
- Raw SKILL.md: https://api.skillmd.com/api/skills/chengyansen-ai/h3-multimodal-lora-training/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: chengyansen-ai (https://skillmd.com/u/chengyansen-ai)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/chengyansen-ai/h3-multimodal-lora-training

---


# H3 Multimodal LoRA Training

Turn a training idea and local multimodal dataset into a reproducible H3 LoRA package through exactly ten gated steps. Treat the workflow as a state machine: finish and verify the current layer before entering the next one. Never call a LoRA “Ref2VA-trained” merely because it can be loaded by a Ref2VA inference workflow.

## Start here

1. Read `references/task-routing.md` and classify the request before proposing parameters.
2. Initialize a project with `scripts/h3_project.py init` when the user wants files created.
3. Record progress with `scripts/h3_project.py status` and `advance`; do not report a later step as complete while an earlier gate is open.
4. Read only the reference associated with the active step:
   - dataset design or audit: `references/dataset-contract.md`
   - trainer choice or configuration: `references/backend-recipes.md`
   - loss, timestep, shift, AdaLN, or distillation: `references/h3-math-and-guards.md`
   - LoRA variants, loss weighting, Turbo, or new algorithm proposals: `references/algorithm-options.md`
   - checkpoint comparison or delivery: `references/evaluation-and-export.md`
   - provenance or disputed claims: `references/source-ledger.md`

The initializer copies ready-to-adapt examples from `assets/` into the project’s `examples/` directory. Adapt the matching example into the required path; do not edit the example in place and mistake it for a completed gate.

Initialize:

```powershell
python scripts/h3_project.py init --root <project-directory> --name <experiment-name> --route <route> --backend <backend> --model-mode <fl2va-or-ref2va>
```

If `python` is unavailable, locate an existing Python runtime; do not install software without authorization. All bundled scripts use the Python standard library; the selected trainer supplies its own dependencies. `ffprobe` is optional for early preparation but required for final media evidence.

Executable backend priority:

1. `diffsynth`: production Ref2VA adapter for image, video, audio, and video+audio references;
2. `ai-toolkit`: low-memory local adapter for image/video references and embedded target/reference-video sound, with explicit rejections for unsupported audio sidecars;
3. `simpletuner`: FL2VA, joint AV, or audio-only adapter; the reviewed path does not claim Ref2VA training.

## Authority and safety boundary

- Inspection, planning, hashing, local project scaffolding, and configuration generation are reversible preparation.
- Downloading large checkpoints, installing dependencies, launching/resuming training, using paid compute, uploading data, or publishing weights requires the user’s explicit request or approval at that point.
- Never upload private faces, voices, credentials, or unreleased media as part of diagnosis.
- Do not train without a recorded rights basis for every sample. Escalate ambiguous likeness, voice, copyright, minor-safety, or geographic/license questions; do not convert uncertainty into a permissive assumption.
- Preserve existing project files. The project initializer refuses non-empty targets; generated artifacts must not overwrite a run silently.
- Pin the base-model revision and trainer commit. Community defaults are hypotheses, not official H3 requirements.

## Routes

Use one primary route and optional secondary evaluation goals:

| Route | Learns | Minimum useful data | Required evaluation |
|---|---|---|---|
| `identity` | person, character, object, clothing | varied images plus motion clips when possible | identity, prompt adherence, motion preservation |
| `style` | visual rendering or cinematography | diverse subjects/scenes; images and clips | style strength, content leakage, motion preservation |
| `motion` | action, physics, camera movement | temporal clips with structured captions | held-out subjects, trajectory, timing, camera compliance |
| `joint_av` | video and target audio together | real synchronized stereo audio/video clips | video, audio, lip/event sync, silence behavior |
| `true_ref2va` | use of reference image/video/audio conditions | targets explicitly separated from references | zero/one/multi-reference and order-swap tests |
| `turbo` | few-step DMD/Turbo acceleration | teacher/student-specific data | speed-quality frontier and teacher retention |

`turbo` is a separate objective. Never reuse its rank, shifts, or loss recipe as a default identity/style recipe.

## The ten nested steps

### 1. Freeze the outcome

Capture the subject, route candidates, target inference workflow, quality priority, trigger, intended use, budget, deadline, and whether the user wants preparation only or an actual run. Write `records/01-intake.json`.

Gate: one measurable primary outcome, one inference target, and an explicit execution boundary exist. If the request is vague, propose a conservative default and label it as a proposal.

### 2. Prove rights and feasibility

Record dataset rights/consent in `records/02-rights.json`. Inspect GPU model/count/VRAM, system RAM, disk, OS, Python, CUDA, BF16, `ffprobe`, base checkpoint availability, and inference host in `records/02-environment.json`.

Collect machine evidence with:

```powershell
python scripts/probe_environment.py --root <project> --backend-repo <pinned-checkout>
```

Gate: `authorized=true`, every sample can map to a rights record, the H3 license is acceptable for the intended territory/use, and the chosen hardware route is plausible. Do not use inference-memory figures as training estimates.

### 3. Select the learning route

Apply `references/task-routing.md`. Separate these claims:

- `load_compatible`: a LoRA file can be attached to that inference graph;
- `condition_trained`: reference media actually entered the training conditioning sequence;
- `audio_trained`: target audio contributed a non-masked loss.

Write `records/03-route-decision.json` with accepted route, rejected alternatives, evidence, primary backend, fallback backend, and evaluation obligations.

Gate: data semantics and backend capabilities match the claim. A T2V LoRA loadable in Ref2VA remains `condition_trained=false`.

### 4. Build the dataset contract

Create `data/manifest.jsonl` using `references/dataset-contract.md`. Keep target video/audio separate from reference image/video/audio. Split by source or subject, never by adjacent frames from the same source. Captions describe visible/audible variation and temporal structure, not just quality slogans.

Gate: unique IDs, existing local paths, non-empty captions, explicit split, rights IDs, no target/reference self-copy, and all route-required modalities are present.

### 5. Audit and preprocess without leakage

Run:

```powershell
python scripts/lint_captions.py --manifest <project>/data/manifest.jsonl --route <route> --trigger <trigger> --output <project>/reports/04-caption-lint.json
python scripts/audit_dataset.py --manifest <project>/data/manifest.jsonl --rights-ledger <project>/records/02-rights.json --project-root <project> --route <route> --trigger <trigger> --output <project>/reports/05-data-audit.json
```

Use `--hash-media` for exact-file duplicate checks and `--strict` when warnings must fail CI. Review contact sheets and representative clips manually; automated probes cannot prove identity, anatomy, motion, sync, or aesthetics.

Gate: `hard_failures` is empty; every `rights_id` maps to `training_allowed=true`; audit hashes match the current manifest and rights ledger; caption warnings are reviewed; duplicate/leakage findings are resolved; 24 fps, dimensions, duration, frames, 32 kHz stereo, A/V offset, clipping, and silence warnings are either corrected or recorded as backend-handled exceptions.

### 6. Lock one falsifiable experiment

Choose resolution/frame buckets, rank/alpha, target modules, learning rate, optimizer, precision/quantization, checkpoint cadence, validation seeds, shift recipe, target sign, modality loss weights, and distillation guard. Do not present guessed values as user-approved facts.

Create `records/06-experiment-lock.json`:

```powershell
python scripts/h3_project.py lock --root <project> --dataset-manifest <project>/data/manifest.jsonl --profile <project>/configs/training-profile.json --model-revision <sha-or-tag> --trainer-revision <commit>
```

Gate: dataset/profile hashes, base revision, trainer commit, route, conditioning claims, math convention, validation prompts, and one-change-only hypothesis are immutable for this run. Step 7 translates this neutral profile into a pinned backend config without changing its semantics.

### 7. Resolve the pinned backend configuration

Read the selected backend’s local README/config examples at the pinned commit and then `references/backend-recipes.md`. Generate `configs/backend-config.*` plus `records/07-launch-plan.json` containing the exact command, environment, estimated resources, output paths, resume behavior, and stop conditions.

For the three executable adapters run:

```powershell
python scripts/prepare_backend.py --root <project> --backend-repo <pinned-checkout>
```

Add `--materialize hardlink` (or `symlink`/`copy`) for AI Toolkit and SimpleTuner because those trainers consume folder datasets. DiffSynth consumes the generated `configs/diffsynth-metadata.json` directly. The command rejects unknown profile fields, writes requested/resolved/defaulted/rejected semantics to `configs/resolved-profile.json`, and verifies the local Git revision and required source symbols before writing the launch plan.

For AI Toolkit and SimpleTuner, add a source-verified `backend_options.inference_validation` contract based on `assets/configs/inference-validation.example.json`. Do not invent a host CLI. DiffSynth gets a built-in reference-ablation validator. If DiffSynth uses a DeCFG preset adapter, set `distillation_guard=diffsynth_decfg_adapter`, provide `preset_lora_path`, and keep `preset_lora_model=dit`.

Gate: dry-run/config parser succeeds; model variant, dataset schema, LoRA target modules, audio mode, reference mode, shifts, adapter/CFG/drift guard, preview settings, and output format are explicit. Never invent a CLI flag from memory.

### 8. Run a bounded smoke test

Only after execution is authorized, cache a small representative subset and run 50–300 steps. Save frequent checkpoints and fixed-seed base/LoRA-off/LoRA-on samples. Record separate video loss and audio loss when audio is in scope, GPU/RAM/disk peaks, throughput, NaN/OOM events, and sample paths in `reports/08-smoke-test.json`.

Execute the exact locked argv without a shell:

```powershell
python scripts/run_backend.py --root <project> --authorization-token I_AUTHORIZE_H3_GPU_TRAINING
```

Immediately before execution the runner revalidates the launch-plan hash, resolved profile, backend Git SHA, and source-file hashes. It writes `pass` only after a zero exit code, finite observed loss, no OOM/NaN marker, and at least one structurally valid SafeTensors checkpoint. DiffSynth currently exposes a combined H3 video+audio loss; record that observability limit and require held-out audio evaluation before final `audio_trained` claims.

Gate: `decision=pass`; losses are finite; every claimed modality contributes; samples show directional learning without major base drift; resume works; memory headroom is acceptable. Fix the cause and repeat with a new experiment ID rather than silently changing the lock.

### 9. Train, monitor, and select evidence-first

Run the approved schedule with high-frequency checkpoints. Evaluate multiple checkpoints on fixed held-out prompts and seeds. Fill `reports/checkpoint-metrics.csv`, then run:

```powershell
python scripts/score_checkpoints.py --input <project>/reports/checkpoint-metrics.csv --route <route> --output <project>/reports/09-checkpoint-selection.json
```

Write `reports/09-training-summary.json` with actual settings, interruptions, drift, and the chosen—not necessarily final—checkpoint.

Gate: the selected checkpoint wins on the route scorecard and passes base-preservation/artifact floors. Stop early when late checkpoints degrade; more steps are not inherently better.

### 10. Prove inference compatibility and package

Test the chosen LoRA in its declared H3 host with matching base variant, rank/alpha semantics, target modules, LoRA strength range, shifts, scheduler, and any special node/pass. For Ref2VA, test 0/1/multiple references and order changes. For AV, test speech, non-speech, silence, and event sync.

For DiffSynth the backend preparer generates `configs/validate-diffsynth.py` and a fixed-seed 0/1/all/reversed-reference matrix. AI Toolkit and SimpleTuner use the locked inference argv template. After choosing the checkpoint run:

```powershell
python scripts/validate_lora.py --weights <checkpoint.safetensors> --project-root <project> --authorization-token I_AUTHORIZE_H3_GPU_INFERENCE
```

Without `--project-root`, `validate_lora.py` performs structural SafeTensors and LoRA A/B-pair validation only; that is not inference proof. With a project root, each generated sample must also pass ffprobe stream, duration, fps, frame-grid, dimensions, sample-rate, and channel checks declared by the launch plan.

Create `exports/delivery-manifest.json` listing weights, hashes, trigger, recommended strength, base/trainer revisions, conditions actually trained, limitations, sample matrix, license/NOTICE, dataset provenance summary, inference workflow, and reproduction command.

Gate: `reports/10-lora-validation.json` proves structural validity and successful inference samples; a clean environment can load the package; claims do not exceed tests; no dataset media, secrets, optimizer state, or private metadata leaks into the export.

## Non-negotiable H3 invariants

- H3’s public training convention uses `x_sigma=(1-sigma)*x0+sigma*noise`, model timestep `t=1-sigma`, and data-ward target `x0-noise`. Verify backend sign rather than assuming a generic diffusion target.
- Map one base noise level to modality-specific schedules; do not sample unrelated video/audio timesteps for a synchronized target.
- Missing target audio must be masked or rejected, never silently trained as long silence. Audio-only rows must mask video loss.
- Record video and audio losses separately when the backend exposes them. If it exposes only a combined AV loss, record that observability limit and require held-out audio/sync inference before claiming audio learning; a healthy total loss alone is insufficient.
- Public inference commonly uses video shift 12 and audio shift 3, while some training recipes use other empirical shifts. Save the exact training and inference values; do not silently normalize them.
- H3 checkpoints are CFG-distilled. Use and record a supported preservation strategy such as training adapter plus contrastive/CFG-augmented loss, frozen-base drift regularization, or a tested backend-specific equivalent. Always keep early checkpoints.
- Default to excluding AdaLN from ordinary LoRA when cross-compatibility matters. Train or approximate AdaLN only for an explicit, validated backend/model objective.
- H3 video is fixed at 24 fps in the official pipeline. Treat frame-grid and dimension mismatches as explicit preprocessing decisions, not invisible coercions.

Read `references/h3-math-and-guards.md` before changing any of these invariants.

Do not add rsLoRA, LoRA+, DoRA, PiSSA, LoftQ, Min-SNR, sparse attention, or DMD merely because it is newer. Read `references/algorithm-options.md`; treat each as a one-change-only experiment and require backend implementation plus inference round-trip evidence.

## Completion response

Report the current step and gate status, not merely “training completed.” On final delivery include:

- route and exact capability claims;
- dataset counts by modality and split;
- base/trainer revisions and experiment-lock hash;
- selected checkpoint and why it beat the final checkpoint if applicable;
- evaluated inference modes, recommended strength, and known failures;
- local paths to the project, report, configuration, weights, and delivery manifest;
- any action still requiring user authorization.

The skill is complete only after Step 10 passes or the user explicitly requested an earlier stopping point.