open-edge-platform Multimodal Embedding Serving — Dev
Work on the service's source. This skill assumes a repo clone of
edge-ai-libraries with this microservice at
microservices/multimodal-embedding-serving/; if there is no clone, clone the
repo first (git clone https://github.com/open-edge-platform/edge-ai-libraries.git -b main) or — if the user
only wants to use the service — switch to
../multimodal-embedding-serving-user/SKILL.md.
Run all commands from the microservice root.
When to Use
- Add or modify a model-family handler (CLIP/SigLIP/MobileCLIP/CN-CLIP/Blip2/QwenText)
- Run or extend the existing test suite
- Navigate the wrapper → registry → handler architecture before editing
- Build the image from source
- Debug model load, OpenVINO conversion, or import failures
Example Prompts
Sample Problem-solving scenarios this skill handles end-to-end:
| Example | Problem it solves |
|---|---|
| onboard-new-model.md | Onboard a new embedding model family (REST + SDK) |
| update-test-cases.md | Update test cases to cover a newly onboarded model |
Reference Lookup
| File | Load when… |
|---|---|
references/source-map.md |
locating code before editing, or adding a model family (checklist inside) |
references/testing.md |
running/adding tests or smoke-testing changes |
Environment setup
poetry install # Python >=3.10,<3.14
Known limitation: mobileclip and salesforce-lavis are installed only
in the Docker image (see docker/Dockerfile), not via pyproject.toml. In a
bare Poetry env, MobileCLIP and Blip2 handlers fail to import — develop those
families against the container:
export EMBEDDING_MODEL_NAME="MobileCLIP/mobileclip_s0"
source setup.sh && docker compose -f docker/compose.yaml up -d --build
Architecture in one paragraph
src/app.py (routes, input union) → src/wrapper.py (EmbeddingModel, the
high-level API — also the SDK surface) → src/models/registry.py
(factory) → src/models/handlers/<family>_handler.py (per-family
load_model/encode_text/encode_image, PyTorch and optional OpenVINO
paths). src/models/config.py holds the 19-model registry. Map + add-a-model
checklist: references/source-map.md.
Test / verify loop
poetry run python -m unittest tests/test_path_security.py -v # the existing suite
Coverage is thin (path security only) — for behavior changes, smoke-test via
examples/server_examples.py / examples/sdk_examples.py against a small
CLIP model: references/testing.md.
Build from source
export EMBEDDING_MODEL_NAME="CLIP/clip-vit-b-32"
source setup.sh # sourced; hard-fails without the model name
docker compose -f docker/compose.yaml build
docker compose -f docker/compose.yaml up -d
Debug a running instance
docker logs -f multimodal-embedding-serving— model load, conversion, request logs (server runs--log-level debug).curl -s localhost:9777/model/currentand/model/capabilities— ground truth for what's loaded.- Slow first inference = lazy OpenVINO export/compile into
EMBEDDING_OV_MODELS_DIR(/app/ov_models); cached afterwards in theov-modelsvolume.
Contribution gotchas
| Gotcha | Consequence |
|---|---|
src/wrapper.py is a path dependency of multimodal-dataprep (visual-data-preparation-for-retrieval/multimodal-dataprep) |
API changes there ripple into that service — check its usage before changing signatures |
Package is built as a wheel (poetry build; packages maps src/ → multimodal_embedding_serving) |
keep new modules importable under the package name; SDK users import from it |
INFER_BATCH_SIZE compiles OpenVINO models to a fixed batch shape |
handler changes must keep the pad/split logic intact |
| QwenText handlers are text-only by design | don't "fix" the 400 for images; capability flags live on the handler |
| Every new file needs the SPDX header | CI/license scans fail otherwise |