Fiftyone Zoo Remote Model

FiftyOne Remote Model Zoo — Integration Guide

When to use

Triggers: new remote zoo source; debugging an existing one; registration "succeeds" but model not loadable; ModuleNotFoundError from DataLoader workers; custom fom.Model / TorchModelMixin; VLM/structured-output integrations.

Not this skill: plugins, operators, panels, brain methods. Route via Phase 0.

Phase 0 — Confirm the integration surface

Required output: one-line confirmation. Example: "Zoo model integration because user wants dataset.apply_model(model) to write predictions." If you cannot write that line, stop.

Phase 1 — Scaffold

Copy template/:

• manifest.json — top-level name required (silent skip if missing).
• __init__.py — exports download_model, load_model, optional resolve_input. Relative imports: from .zoo import ....
• zoo.py — config + model class.

Phase 2 — Implement

• Class hierarchy, properties, predict/predict_all input dispatch → MODEL-CLASS.md.
• Label return types, single-fo.Label rule, coordinates → LABEL-TYPES.md.
• DataLoader pickle, worker import resolution → DATALOADER.md.
• VLM / generative-structured-output (uses generate() with prompts/schemas) → also VLM-PATTERNS.md.

Phase 3 — Validate

• manifest.json has top-level name.
• __init__.py uses relative imports.
• Image ops return single fo.Label (dicts only for video frame-level, integer keys).
• One-known-example coordinate check passed.
• dataset.apply_model(model) runs with default num_workers.
• On macOS, run dataset.apply_model(model, ...) with default num_workers and confirm no ModuleNotFoundError from spawned workers.

On failure, route by symptom:

Key Directives

Canonical names; other files cite by name. Full failure modes and diagnostic moves in DEBUGGING-PRINCIPLES.md.

• Runtime parameters are setters. NEVER make users reconstruct the model to change a generate() / forward kwarg, a prompt, an operation selector, or a post-processing threshold. Why: weights are large; anything that feeds into model(...) is per-call input, not model identity.
• Framework-first. ALWAYS use FiftyOne primitives before custom code. Why: framework classes are worker-importable; yours aren't.
• Worker-pickle constraint. NEVER define pickle-bound objects in zoo.py. Why: spawned DataLoader workers can't import modules loaded via importlib.util.spec_from_file_location.
• Reference implementations need verification. NEVER copy a pattern from another zoo source without running it under multi-worker first. Why: widely-copied references silently break.
• Schema compliance ≠ correctness. NEVER trust schema-conformant outputs as proof of correctness. Why: models echo your wrong field names back unchanged.
• Read specs, don't patch parsers. NEVER patch a parser more than twice — find the format spec. Why: each patch shifts the failure elsewhere; the cycle is unbounded.
• Bounded repair scope. NEVER grow repair logic past a few common malformations. Why: unbounded repair masks model-quality regressions.

Quick reference index

• MANIFEST.md — schema, entry points, idempotent download_model.
• MODEL-CLASS.md — hierarchy, properties, predict dispatch.
• DATALOADER.md — worker pickle WHY, primitives, wrong fixes that look right.
• LABEL-TYPES.md — return types, coordinate normalization.
• DEBUGGING-PRINCIPLES.md — six rules with failure modes and diagnostic moves.
• VLM-PATTERNS.md — tool calling, generation budget, thinking, vision tokens, delimiters, multi-tier parser, coordinate quirks.