# Mujoco > MuJoCo for lightweight, contact-rich robot manipulation simulation on macOS/Linux, especially single-arm grasping without ROS: headless offscreen rendering, MJCF models, mujoco_menagerie assets, damped-least-squares inverse kinematics, and empirical grasp calibration. Use when: 'MuJoCo', 'MJCF', 'mjpython', 'MUJOCO_GL', 'menagerie', 'SO-101' / SO-ARM100 arm, 'offscreen render', 'inverse kinematics' / 'IK', 'grasp' / 'pick and place' in sim, hand-building a manipulation env, or headless mujoco.Renderer determinism. Load after architect or simulation route a non-ROS single-arm manipulation task here (Gazebo and Isaac Sim are the ROS / GPU-photoreal alternatives). Defers to MuJoCo's own docs and mujoco_menagerie for the physics/model API; embeds the macOS-arm64 platform and grasp/IK gotchas hard-won from a real SO-101 build. Not for: ROS-integrated sim (gazebo), GPU-parallel RL or photoreal synthetic data (isaac-sim / isaac-lab), or LeRobot policy training/eval mechanics (lerobot). - Skill: `robium-ai/mujoco-3` (Agent Skill) - Install (CLI): `npx skillmds@latest add robium-ai/mujoco-3` - Raw SKILL.md: https://api.skillmd.com/api/skills/robium-ai/mujoco-3/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: robium-ai (https://skillmd.com/u/robium-ai) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/robium-ai/mujoco-3 --- # mujoco MuJoCo 3.x for robium's lightweight, non-ROS manipulation side: a fast contact-rich physics engine with an offscreen renderer, well suited to hand-building a single-arm grasping/pick-and-place environment when the full ROS + Gazebo stack (or a GPU-gated Isaac stack) is more than the task needs. This skill stays deliberately thin on the MuJoCo API itself — the model format (MJCF), the Python bindings, and the robot assets in mujoco_menagerie are all documented upstream and move release to release. What it embeds instead is the set of hard-won gotchas from the vla-trial SO-101 build (macOS arm64, MuJoCo 3.x, observed 2026-07-13/14): the macOS headless-render backend, version-pin traps, offscreen-render determinism, and the IK / grasp-calibration failure modes that cost the most time and give no error message when they bite. ## When to use this skill - Simulating a low-DOF arm (grasping, pick-and-place, contact-rich manipulation) where ROS is not required and a GPU-photoreal simulator is overkill — MuJoCo runs natively and fast on Apple Silicon. - Rendering camera observations headless for a policy/dataset pipeline, and needing that rendering to be deterministic/reproducible. - Writing an inverse-kinematics reach + grasp routine by hand (damped least-squares), and debugging why the arm won't reach or knocks the object over. - Loading a robot from mujoco_menagerie (e.g. the SO-101 / SO-ARM100 arm) and adapting its scene into a manipulation task. - The trigger phrases in the description: 'MuJoCo', 'MJCF', 'mjpython', 'MUJOCO_GL', 'menagerie', 'SO-101', 'offscreen render', 'IK', 'grasp'. - Cross-references — go to the sibling skill instead when the question is: - A ROS 2-integrated simulation (ros_gz bridge, SDF worlds) → the `gazebo` skill. - Photoreal rendering, synthetic data at scale, or GPU-parallel RL → the `isaac-sim` / `isaac-lab` skills (GPU-gated). - Which simulator to pick at all → the `simulation` umbrella (routes here for lightweight non-ROS manipulation). - LeRobot dataset/policy training/eval mechanics → the `lerobot` skill; MuJoCo here is only the sim that produces observations/actions. - The whole-stack decision this feeds into → the `architect` skill. ## Key directives - **Delegation posture: embed + links.** The robium-specific, hard-won gotchas (macOS render backend, version pins, offscreen-render determinism, IK/grasp failure modes) live here because they are not written down anywhere upstream. For the physics/model API itself — MJCF syntax, the `mujoco` Python bindings, `mj_step`/`mj_forward`, the `Renderer` class, and robot assets — point upstream to MuJoCo's docs and the mujoco_menagerie repo (see References); do not restate the API from memory, it changes across 3.x releases. - **On macOS, `MUJOCO_GL=cgl` is the only headless render backend.** osmesa and egl are Linux-only (mujoco#2164). Set it before importing `mujoco`. - **Watch the version-pin trap.** Menagerie's SO-101 needs MuJoCo ≥3.1.3, but common gym wrappers pin it *down* (gym-hil pins `mujoco<3.9`, gym-xarm pins `<3.0`) — depending on one of those can silently downgrade MuJoCo below the model's requirement. Pin MuJoCo explicitly and verify the installed version. - **Hand-written IK fails silently.** A damped-least-squares solver does not raise on an unreachable target or on a bad grasp orientation — it returns a locally-optimal pose short of the goal. Probe reachability and calibrate grasps empirically (see Usage patterns); never assume a returned qpos means success. - **Verify the asset matches the robot's intended work envelope** before blaming IK. Menagerie scenes are generic and can put the floor, object spawn, or camera in the wrong place for the arm (see Platform gotchas). ## Quick start **1. Install (pin MuJoCo explicitly, above the menagerie floor):** ```bash uv add "mujoco>=3.1.3" # current 3.10 as of 2026-07; SO-101 needs >=3.1.3 ``` Do not let a gym wrapper (gym-hil, gym-xarm) choose the version transitively — it can pin MuJoCo below what the model needs. **2. Get the robot asset from mujoco_menagerie, not upstream MJCF.** Clone [mujoco_menagerie](https://github.com/google-deepmind/mujoco_menagerie) and use its SO-101 model (the robotstudio_so101 directory, which ships a pick-and-place scene, manipulation-tuned collision geometry, and a camera mount). Do NOT use the upstream TheRobotStudio/SO-ARM100 MJCF — its own README admits the gripper linear-joint mapping is not reflected in the model. **3. Headless offscreen render (macOS):** ```python import os os.environ["MUJOCO_GL"] = "cgl" # macOS headless backend; must precede import import mujoco model = mujoco.MjModel.from_xml_path("scene.xml") data = mujoco.MjData(model) renderer = mujoco.Renderer(model, height=256, width=256) # warm-up: one throwaway full reset+render (see determinism gotcha below) mujoco.mj_resetData(model, data) mujoco.mj_forward(model, data) renderer.update_scene(data, camera="fixed_cam") _ = renderer.render() ``` **4. Interactive viewer needs `mjpython`.** `mujoco.viewer.launch_passive` requires launching with `mjpython`, not `python` (and is buggy under `uv`, mujoco#1923). Offscreen rendering (step 3) has no such requirement. For MJCF syntax, the bindings, and API signatures, go to MuJoCo's own docs (References) before hand-writing model or physics code. ## Usage patterns **Headless render is not the bottleneck on Apple Silicon.** Measured on the vla-trial build: ~84 fps at 256x256 via `MUJOCO_GL=cgl`, against a 60 fps floor; `mj_step` ~0.012 ms vs ~11.8 ms per render. This kills the common upstream worry that "offscreen render is extremely slow" — for a low-DOF arm scene it is comfortably real-time. Render, don't skip frames to save time. **Deterministic offscreen rendering** (needed for reproducible datasets/evals): - **Use a static xyaxes look-at camera, not `mode="targetbody"`.** A target-tracking camera re-runs its smoothing filter on every `update_scene()`, making the rendered pose call-history-dependent (observed up to 85 LSB drift) even when the physics is bit-identical. - **Warm up the renderer with one throwaway full reset + render in `__init__`.** A fresh `mujoco.Renderer`'s first reset→`mj_forward`→render cycle is not repeatable against later renders (~6 LSB). A raw pre-`mj_forward` warm-up does NOT consume the cold cycle — it must be a full reset+render. - Ruled out as a cause: MSAA offsamples (a red herring here). **Reachability probing (IK).** Because the DLS solver returns silently on an unreachable target, probe the full task footprint × every lift height *independently* before trusting the env. Menagerie's default box spawn (0.5, 0, 0.03) is unreachable by the SO-101 mount, for example. **Empirical grasp calibration.** The end-effector site is not the grasp point. Calibrate the gripper-local grasp offset by experiment: teleport the object to a grid, close, lift, and keep the offsets that actually lift. Key facts: - Calibration is a function of **pose**, not a gripper property — recalibrate after any wrist-orientation change. - It is millimeter-sensitive and non-monotonic: a 3 mm offset change cost 6/10 success in one sweep. Sweep the grasp offset *and* the pedestal height end-to-end rather than tuning one in isolation. **Wrist-roll orientation is the silent half of position-only IK.** On a 5-DOF arm, position-only IK leaves wrist roll free. At roll≈0 the pinch axis was 91% vertical, so a 4.2 cm aperture could not span a 6 cm cube (closed on air, 0/10). Diagnostic: print the world-frame pinch axis and check it is perpendicular to the grasped dimension. Fix: solve wrist roll as a 1-D root find (`pinch_z(roll)` is a smooth sinusoid), then re-solve position with roll pinned via jnt_range — adding roll to the DLS objective *diverges*. This took the oracle grasp from 0/10 to 8/10. (Position-only IK also gives no signal about the orientation it chose; a level-finger constraint may be genuinely infeasible on a 5-DOF arm.) **Debugging "arm won't reach / knocked the object over":** - **Bisect IK vs obstruction** by deleting/banishing the object and re-running the same pose. Cube present: arm_err=0.18, saturated_joints=[1 2]; cube banished: arm_err=0.0006, saturated=[]. Instantly tells you whether IK or a collision is the problem. - **Check `qfrc_actuator` vs `forcerange`** — saturated joints mean BLOCKED, not slow. Raising settle steps (e.g. 12→60) is a dead end when joints are saturated. - **The first waypoint can sweep the arm through the object** — a position servo takes an arbitrary joint path to the target. Raising the initial approach waypoint (0.16→0.20 m) fixed a 2/10 → 10/10 grasp. **Resolve gripper geoms by body, not name.** Menagerie leaves mesh geoms UNNAMED, so a name-based gripper-geom list silently misses the fixed-jaw collision mesh — a false-success risk in contact checks. Collect gripper geoms by `model.geom_bodyid` membership instead. **Determine gripper polarity empirically.** Drive the gripper joint to each end of its `actuator_ctrlrange` and measure the fingertip gap. SO-101 is LOW=closed / HIGH=open; the upstream SO-ARM100 docs claim the opposite and even admit their own MJCF disagrees — so measure, don't trust the docs. ## Platform gotchas - **macOS headless render backend is `MUJOCO_GL=cgl` only** (osmesa/egl are Linux-only, mujoco#2164). Set it before `import mujoco`. - **The macOS CGL renderer is thread-affine — a cross-thread first render is a silent deadlock.** A `mujoco.Renderer` created on one thread hangs forever (no error, no timeout, no exception) the first time a *different* thread renders on it — the stack sits in cgl `make_current`. This bites frameworks that run handlers on worker threads (e.g. Gradio): a first `env.reset()` called from a worker freezes the stream. Fix: construct the env/renderer in the same thread that runs the episode (a per-run env in the handler), not once at import on the main thread. (observed 2026-07-15, vla-trial) - **Interactive viewer needs `mjpython`**, not `python`, and is buggy under `uv` (mujoco#1923). Offscreen rendering does not need `mjpython`. - **No off-the-shelf SO-101 Gymnasium env exists.** gym-hil is Franka-only; LIBERO is Linux-only. A language-conditioned SO-101 sim env must be hand-built — budget for it rather than expecting to import one. - **Menagerie assets can be geometrically wrong for the robot.** The SO-101 scene put the floor at the arm's base level; a real rig has base + objects on a raised surface. A 6 cm pedestal restored the intended work envelope (fixing a forced ~32° down-pitch). Check the asset world matches the robot's intended envelope before blaming the IK solver. ## Customization - **Different arm from menagerie:** the failure modes above (silent IK, unnamed geoms, empirical gripper polarity, static-camera determinism) are general MuJoCo/menagerie traps, not SO-101-specific. Re-run the empirical calibration (grasp offset grid, gripper-polarity sweep, reachability probe) for the new model rather than porting the SO-101 numbers. - **Different task envelope:** re-check the scene geometry (floor height, object spawn, camera pose) against the robot's reach before writing task logic — a pedestal or a re-placed spawn is often the fix, not the IK. - **Feeding a LeRobot pipeline:** MuJoCo here produces the observations and executes actions; the dataset format, recording, and policy training/eval belong to the `lerobot` skill — keep the sim env and the dataset layer separate. ## References - Upstream (primary sources — check before writing model or physics code): [MuJoCo documentation](https://mujoco.readthedocs.io/) (MJCF, Python bindings, the `Renderer` API), [google-deepmind/mujoco GitHub repo](https://github.com/google-deepmind/mujoco) (releases and the issues cited above: #2164 render backends, #1923 viewer-under-uv), [mujoco_menagerie](https://github.com/google-deepmind/mujoco_menagerie) (robot assets, incl. the SO-101 model — prefer this over upstream TheRobotStudio/SO-ARM100 MJCF). All facts above observed on MuJoCo 3.x, macOS arm64, 2026-07-13/14 (vla-trial SO-101 build); re-verify version-tied claims against current releases. - Sibling skills: the `simulation` umbrella (routes here for lightweight non-ROS manipulation; owns the Gazebo-vs-Isaac selection), `gazebo` (ROS 2 simulation), `isaac-sim` / `isaac-lab` (GPU-gated photoreal / RL sim), `lerobot` (policy training/eval that consumes this sim's observations), `architect` (routes the whole stack decision here). ## Changelog - 1.0.1 (2026-08-01): anchor IDs added to claim-bearing items (learning-engine Phase 1); no content changes. - 1.0.0 (2026-07-31): created — captured from the vla-trial SO-101 manipulation build (2026-07-13/14).