# Molmo

> Visual pointing and Q&A via the Molmo VLM served from a self-hosted vLLM endpoint (OpenAI-compatible API). Use when a workflow needs a single pixel coordinate for a named object (point_prompt) or Molmo-grade visual question answering and a vLLM server is available; for an API-only zero-GPU alternative use the gemini-er bundle.

- Skill: `graph-robots/molmo` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add graph-robots/molmo`
- Raw SKILL.md: https://api.skillmd.com/api/skills/graph-robots/molmo/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- License: MIT
- Author: graph-robots (https://skillmd.com/u/graph-robots)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/graph-robots/molmo

---


# molmo

Molmo visual pointing + Q&A. The bundle itself is zero-GPU (httpx client
only), but Molmo has **no hosted API** — you must serve it yourself with
vLLM and point `GAP_MOLMO_BASE_URL` at it. If you can't self-host, use
`gemini-er.detect` (hosted Gemini Robotics-ER) instead.

## Hosting recipe (vLLM)

Lifted from the dev tree's run book (`training/README.md`):

```bash
# Serve Molmo2-8B on an OpenAI-compatible endpoint
CUDA_VISIBLE_DEVICES=0 PYTHONNOUSERSITE=1 \
  python -m vllm.entrypoints.openai.api_server \
  --model allenai/Molmo2-8B \
  --trust-remote-code \
  --dtype bfloat16 \
  --port 8122 \
  --gpu-memory-utilization 0.5 \
  --max-model-len 4096 \
  --max-num-batched-tokens 4096

# Smoke-test
curl -s http://127.0.0.1:8122/v1/models | jq '.data[0].id'

# Point the bundle at it
export GAP_MOLMO_BASE_URL=http://127.0.0.1:8122/v1
```

Operational notes from the dev tree: pin Molmo to its own GPU when running
alongside other perception services — under heavy parallel evaluation it
becomes the throughput bottleneck if co-located; on a dedicated GPU you can
push `--gpu-memory-utilization 0.85 --max-num-batched-tokens 8192` for ~2×
perception throughput. The server can also run on a remote machine and be
port-forwarded in (the 4090 real-robot profile did exactly this).

## Config

| Env                  | Meaning                              | Default              |
|----------------------|--------------------------------------|----------------------|
| `GAP_MOLMO_BASE_URL` | vLLM OpenAI-compatible base URL      | — (required)         |
| `GAP_MOLMO_MODEL`    | Model name served by vLLM            | `allenai/Molmo2-8B`  |

## Notes

- `molmo.point_prompt` sends the canonical `"Point at <query>"` prompt and
  parses all four Molmo point output formats (Molmo2 `<points coords=...>`,
  Molmo1 `<point x= y=>`, legacy `<points x1= y1= ...>`, plain `x, y`
  fallback), converting normalized coordinates to pixels. `found=False`
  means the model emitted no parseable point.
- `molmo.query_yes_no` coerces with the source-verbatim rule: answer is true
  iff `"yes"` appears in the lowercased reply.
- Backend unreachable after 3 retries raises `ToolError` (route `on_error`).