Mujoco

mujoco

MuJoCo 3.x for robium's lightweight, non-ROS manipulation side: a fast contact-rich physics engine with an offscreen renderer, well suited to hand-building a single-arm grasping/pick-and-place environment when the full ROS + Gazebo stack (or a GPU-gated Isaac stack) is more than the task needs. This skill stays deliberately thin on the MuJoCo API itself — the model format (MJCF), the Python bindings, and the robot assets in mujoco_menagerie are all documented upstream and move release to release. What it embeds instead is the set of hard-won gotchas from the vla-trial SO-101 build (macOS arm64, MuJoCo 3.x, observed 2026-07-13/14): the macOS headless-render backend, version-pin traps, offscreen-render determinism, and the IK / grasp-calibration failure modes that cost the most time and give no error message when they bite.

When to use this skill

• Simulating a low-DOF arm (grasping, pick-and-place, contact-rich manipulation) where ROS is not required and a GPU-photoreal simulator is overkill — MuJoCo runs natively and fast on Apple Silicon.
• Rendering camera observations headless for a policy/dataset pipeline, and needing that rendering to be deterministic/reproducible.
• Writing an inverse-kinematics reach + grasp routine by hand (damped least-squares), and debugging why the arm won't reach or knocks the object over.
• Loading a robot from mujoco_menagerie (e.g. the SO-101 / SO-ARM100 arm) and adapting its scene into a manipulation task.
• The trigger phrases in the description: 'MuJoCo', 'MJCF', 'mjpython', 'MUJOCO_GL', 'menagerie', 'SO-101', 'offscreen render', 'IK', 'grasp'.
• Cross-references — go to the sibling skill instead when the question is:
  • A ROS 2-integrated simulation (ros_gz bridge, SDF worlds) → the gazebo skill.
  • Photoreal rendering, synthetic data at scale, or GPU-parallel RL → the isaac-sim / isaac-lab skills (GPU-gated).
  • Which simulator to pick at all → the simulation umbrella (routes here for lightweight non-ROS manipulation).
  • LeRobot dataset/policy training/eval mechanics → the lerobot skill; MuJoCo here is only the sim that produces observations/actions.
  • The whole-stack decision this feeds into → the architect skill.

Key directives

• Delegation posture: embed + links. The robium-specific, hard-won gotchas (macOS render backend, version pins, offscreen-render determinism, IK/grasp failure modes) live here because they are not written down anywhere upstream. For the physics/model API itself — MJCF syntax, the mujoco Python bindings, mj_step/mj_forward, the Renderer class, and robot assets — point upstream to MuJoCo's docs and the mujoco_menagerie repo (see References); do not restate the API from memory, it changes across 3.x releases.
• On macOS, MUJOCO_GL=cgl is the only headless render backend. osmesa and egl are Linux-only (mujoco#2164). Set it before importing mujoco.
• Watch the version-pin trap. Menagerie's SO-101 needs MuJoCo ≥3.1.3, but common gym wrappers pin it down (gym-hil pins mujoco<3.9, gym-xarm pins <3.0) — depending on one of those can silently downgrade MuJoCo below the model's requirement. Pin MuJoCo explicitly and verify the installed version.
• Hand-written IK fails silently. A damped-least-squares solver does not raise on an unreachable target or on a bad grasp orientation — it returns a locally-optimal pose short of the goal. Probe reachability and calibrate grasps empirically (see Usage patterns); never assume a returned qpos means success.
• Verify the asset matches the robot's intended work envelope before blaming IK. Menagerie scenes are generic and can put the floor, object spawn, or camera in the wrong place for the arm (see Platform gotchas).

Quick start

1. Install (pin MuJoCo explicitly, above the menagerie floor):

uv add "mujoco>=3.1.3"   # current 3.10 as of 2026-07; SO-101 needs >=3.1.3

Do not let a gym wrapper (gym-hil, gym-xarm) choose the version transitively — it can pin MuJoCo below what the model needs.

2. Get the robot asset from mujoco_menagerie, not upstream MJCF. Clone mujoco_menagerie and use its SO-101 model (the robotstudio_so101 directory, which ships a pick-and-place scene, manipulation-tuned collision geometry, and a camera mount). Do NOT use the upstream TheRobotStudio/SO-ARM100 MJCF — its own README admits the gripper linear-joint mapping is not reflected in the model.

3. Headless offscreen render (macOS):

import os
os.environ["MUJOCO_GL"] = "cgl"   # macOS headless backend; must precede import
import mujoco

model = mujoco.MjModel.from_xml_path("scene.xml")
data = mujoco.MjData(model)
renderer = mujoco.Renderer(model, height=256, width=256)
# warm-up: one throwaway full reset+render (see determinism gotcha below)
mujoco.mj_resetData(model, data)
mujoco.mj_forward(model, data)
renderer.update_scene(data, camera="fixed_cam")
_ = renderer.render()

4. Interactive viewer needs mjpython. mujoco.viewer.launch_passive requires launching with mjpython, not python (and is buggy under uv, mujoco#1923). Offscreen rendering (step 3) has no such requirement.

For MJCF syntax, the bindings, and API signatures, go to MuJoCo's own docs (References) before hand-writing model or physics code.

Usage patterns

Headless render is not the bottleneck on Apple Silicon. Measured on the vla-trial build: ~84 fps at 256x256 via MUJOCO_GL=cgl, against a 60 fps floor; mj_step ~0.012 ms vs ~11.8 ms per render. This kills the common upstream worry that "offscreen render is extremely slow" — for a low-DOF arm scene it is comfortably real-time. Render, don't skip frames to save time.

Deterministic offscreen rendering (needed for reproducible datasets/evals):

• Use a static xyaxes look-at camera, not mode="targetbody". A target-tracking camera re-runs its smoothing filter on every update_scene(), making the rendered pose call-history-dependent (observed up to 85 LSB drift) even when the physics is bit-identical.
• Warm up the renderer with one throwaway full reset + render in __init__. A fresh mujoco.Renderer's first reset→mj_forward→render cycle is not repeatable against later renders (~6 LSB). A raw pre-mj_forward warm-up does NOT consume the cold cycle — it must be a full reset+render.
• Ruled out as a cause: MSAA offsamples (a red herring here).

Reachability probing (IK). Because the DLS solver returns silently on an unreachable target, probe the full task footprint × every lift height independently before trusting the env. Menagerie's default box spawn (0.5, 0, 0.03) is unreachable by the SO-101 mount, for example.

Empirical grasp calibration. The end-effector site is not the grasp point. Calibrate the gripper-local grasp offset by experiment: teleport the object to a grid, close, lift, and keep the offsets that actually lift. Key facts:

• Calibration is a function of pose, not a gripper property — recalibrate after any wrist-orientation change.
• It is millimeter-sensitive and non-monotonic: a 3 mm offset change cost 6/10 success in one sweep. Sweep the grasp offset and the pedestal height end-to-end rather than tuning one in isolation.

Wrist-roll orientation is the silent half of position-only IK. On a 5-DOF arm, position-only IK leaves wrist roll free. At roll≈0 the pinch axis was 91% vertical, so a 4.2 cm aperture could not span a 6 cm cube (closed on air, 0/10). Diagnostic: print the world-frame pinch axis and check it is perpendicular to the grasped dimension. Fix: solve wrist roll as a 1-D root find (pinch_z(roll) is a smooth sinusoid), then re-solve position with roll pinned via jnt_range — adding roll to the DLS objective diverges. This took the oracle grasp from 0/10 to 8/10. (Position-only IK also gives no signal about the orientation it chose; a level-finger constraint may be genuinely infeasible on a 5-DOF arm.)

Debugging "arm won't reach / knocked the object over":

• Bisect IK vs obstruction by deleting/banishing the object and re-running the same pose. Cube present: arm_err=0.18, saturated_joints=[1 2]; cube banished: arm_err=0.0006, saturated=[]. Instantly tells you whether IK or a collision is the problem.
• Check qfrc_actuator vs forcerange — saturated joints mean BLOCKED, not slow. Raising settle steps (e.g. 12→60) is a dead end when joints are saturated.
• The first waypoint can sweep the arm through the object — a position servo takes an arbitrary joint path to the target. Raising the initial approach waypoint (0.16→0.20 m) fixed a 2/10 → 10/10 grasp.

Resolve gripper geoms by body, not name. Menagerie leaves mesh geoms UNNAMED, so a name-based gripper-geom list silently misses the fixed-jaw collision mesh — a false-success risk in contact checks. Collect gripper geoms by model.geom_bodyid membership instead.

Determine gripper polarity empirically. Drive the gripper joint to each end of its actuator_ctrlrange and measure the fingertip gap. SO-101 is LOW=closed / HIGH=open; the upstream SO-ARM100 docs claim the opposite and even admit their own MJCF disagrees — so measure, don't trust the docs.

Platform gotchas

• macOS headless render backend is MUJOCO_GL=cgl only (osmesa/egl are Linux-only, mujoco#2164). Set it before import mujoco.
• The macOS CGL renderer is thread-affine — a cross-thread first render is a silent deadlock. A mujoco.Renderer created on one thread hangs forever (no error, no timeout, no exception) the first time a different thread renders on it — the stack sits in cgl make_current. This bites frameworks that run handlers on worker threads (e.g. Gradio): a first env.reset() called from a worker freezes the stream. Fix: construct the env/renderer in the same thread that runs the episode (a per-run env in the handler), not once at import on the main thread. (observed 2026-07-15, vla-trial)
• Interactive viewer needs mjpython, not python, and is buggy under uv (mujoco#1923). Offscreen rendering does not need mjpython.
• No off-the-shelf SO-101 Gymnasium env exists. gym-hil is Franka-only; LIBERO is Linux-only. A language-conditioned SO-101 sim env must be hand-built — budget for it rather than expecting to import one.
• Menagerie assets can be geometrically wrong for the robot. The SO-101 scene put the floor at the arm's base level; a real rig has base + objects on a raised surface. A 6 cm pedestal restored the intended work envelope (fixing a forced ~32° down-pitch). Check the asset world matches the robot's intended envelope before blaming the IK solver.

Customization

• Different arm from menagerie: the failure modes above (silent IK, unnamed geoms, empirical gripper polarity, static-camera determinism) are general MuJoCo/menagerie traps, not SO-101-specific. Re-run the empirical calibration (grasp offset grid, gripper-polarity sweep, reachability probe) for the new model rather than porting the SO-101 numbers.
• Different task envelope: re-check the scene geometry (floor height, object spawn, camera pose) against the robot's reach before writing task logic — a pedestal or a re-placed spawn is often the fix, not the IK.
• Feeding a LeRobot pipeline: MuJoCo here produces the observations and executes actions; the dataset format, recording, and policy training/eval belong to the lerobot skill — keep the sim env and the dataset layer separate.

References

• Upstream (primary sources — check before writing model or physics code): MuJoCo documentation (MJCF, Python bindings, the Renderer API), google-deepmind/mujoco GitHub repo (releases and the issues cited above: #2164 render backends, #1923 viewer-under-uv), mujoco_menagerie (robot assets, incl. the SO-101 model — prefer this over upstream TheRobotStudio/SO-ARM100 MJCF). All facts above observed on MuJoCo 3.x, macOS arm64, 2026-07-13/14 (vla-trial SO-101 build); re-verify version-tied claims against current releases.
• Sibling skills: the simulation umbrella (routes here for lightweight non-ROS manipulation; owns the Gazebo-vs-Isaac selection), gazebo (ROS 2 simulation), isaac-sim / isaac-lab (GPU-gated photoreal / RL sim), lerobot (policy training/eval that consumes this sim's observations), architect (routes the whole stack decision here).

Changelog

• 1.0.0 (2026-07-31): created — captured from the vla-trial SO-101 manipulation build (2026-07-13/14).