Multimodal Embedding Serving User

Multimodal Embedding Serving — User

Run and call the embedding service. Run commands yourself and relay output. REST base URL: http://localhost:9777 (host port hardcoded by setup.sh; container 8000).

When to Use

• Deploy the embedding service (Docker Compose or standalone) and confirm health
• Embed text, images, or videos over REST on port 9777
• Choose or switch among the CLIP/SigLIP/MobileCLIP/CN-CLIP/Blip2/QwenText models
• Integrate embeddings in-process via the Python SDK wheel
• Diagnose 400/422 errors or model-capability mismatches

Example Prompts

Sample Problem-solving scenarios this skill handles end-to-end:

Docs & deploy files — with or without a clone

All paths below are relative to microservices/multimodal-embedding-serving/ in the edge-ai-libraries repo. No clone? Fetch any of them from GitHub raw:

https://raw.githubusercontent.com/open-edge-platform/edge-ai-libraries/main/microservices/multimodal-embedding-serving/<path>

Load these existing docs only when needed:

1. Context routing — repo clone or standalone? REST or SDK?

• In-process Python integration wanted (no separate server): the service doubles as an SDK — build the wheel with poetry build (needs the repo) and use get_model_handler(...) + EmbeddingModel; see docs/user-guide/sdk-usage.md. Rule of thumb: default to REST; pick the SDK when a Python process embeds heavily and an HTTP hop per item would dominate. Note: MobileCLIP/Blip2 extras exist only in the Docker image, and the wheel is not on PyPI.
• Otherwise detect a clone:

  [ -f setup.sh ] && grep -q 'name = "multimodal-embedding-serving"' pyproject.toml 2>/dev/null \
    && echo REPO || echo STANDALONE
  

  REPO → Step 2 from the microservice root. STANDALONE → fetch the two deploy files, then the exact same Step 2:

  RAW=https://raw.githubusercontent.com/open-edge-platform/edge-ai-libraries/main/microservices/multimodal-embedding-serving
  mkdir -p embedding-serving/docker && cd embedding-serving
  curl -fsSL $RAW/setup.sh -o setup.sh
  curl -fsSL $RAW/docker/compose.yaml -o docker/compose.yaml
  

  Already running (curl -sf localhost:9777/health) → Step 3.

2. Bring-up (identical in both contexts)

1. Pick a model — default CLIP/clip-vit-b-32 (512-dim, text+image+video). Trade-offs: docs/user-guide/supported-models.md.
2. setup.sh must be sourced and hard-fails without EMBEDDING_MODEL_NAME. REGISTRY_URL=intel selects the prebuilt image; --no-build prevents a source build. Run in the background — first start downloads the model:

   bash -c 'export EMBEDDING_MODEL_NAME="CLIP/clip-vit-b-32" REGISTRY_URL=intel TAG=latest \
     && source setup.sh && docker compose -f docker/compose.yaml up -d --no-build'
   

   Intel GPU: also export EMBEDDING_DEVICE=GPU (setup.sh then auto-enables OpenVINO + THROUGHPUT mode).
3. Wait for readiness:

   until curl -sf http://localhost:9777/health; do sleep 5; done
   

3. Capability check (mandatory before non-text inputs)

curl -s http://localhost:9777/model/capabilities

QwenText models are text-only — image/video requests return 400. GET /model/current shows the exact loaded model id to use in requests.

4. Embed

Text (single string or list of strings):

curl -s http://localhost:9777/embeddings -H 'Content-Type: application/json' -d '{
  "model": "CLIP/clip-vit-b-32",
  "input": {"type": "text", "text": "a red truck at a loading dock"},
  "encoding_format": "float"
}'

Response: {"embedding": [...]} — a flat vector for text/image; a list of per-frame vectors for video inputs.

• model must equal the loaded model (else 400).
• Image: {"type":"image_url","image_url":"https://…"} (plain string, not a nested object) or image_base64. Video: video_url/video_base64/ video_frames with segment_config (num_frames default 64, extraction_fps, frame_indexes) — full shapes and examples: docs/user-guide/api-reference.md.

5. Stop / clean

• docker compose -f docker/compose.yaml down
• Volumes ov-models (model caches) and data-prep persist; removing them forces re-downloads — confirm with the user first.

Troubleshooting