Calhacks Agenticrobot

ManiSkill + XLeRobot Setup Guide

This guide sets up MERLIN to control XLeRobot in ManiSkill simulation for sim2real validation.

Quick Setup (15 minutes)

1. Create ManiSkill Environment

# Create separate conda env for ManiSkill
conda create -y -n merlin-sim python=3.11
conda activate merlin-sim

# Install ManiSkill and dependencies
pip install mani-skill pygame rerun-sdk

# Download ReplicaCAD scene dataset
python -m mani_skill.utils.download_asset "ReplicaCAD"

2. Install XLeRobot Files in ManiSkill

1. Download XLeRobot simulation files:

   • Clone: https://github.com/xle-robots/xle-robot
   • Navigate to: sim/maniskill/ folder

2. Copy to ManiSkill package:

   # Find your ManiSkill installation
   MANI_PATH=$(python -c "import mani_skill; print(mani_skill.__path__[0])")
   
   # Copy XLeRobot files
   cp -r xle-robot/sim/maniskill/agents/xlerobot/ $MANI_PATH/agents/robots/
   cp -r xle-robot/sim/maniskill/assets/xlerobot/ $MANI_PATH/assets/robots/
   cp -r xle-robot/sim/maniskill/envs/scenes/* $MANI_PATH/envs/scenes/
   

3. Register XLeRobot in ManiSkill:

   # Edit: $MANI_PATH/agents/robots/__init__.py
   # Add: from .xlerobot import *
   

3. Test ManiSkill Standalone

# Test ManiSkill works with XLeRobot
python -m mani_skill.examples.demo_ctrl_action \
  -e "ReplicaCAD_SceneManipulation-v1" \
  -r "xlerobot_single" \
  --render-mode="human" \
  --shader="default" \
  -c "pd_joint_delta_pos"

Expected: XLeRobot appears in 3D scene, can accept keyboard commands.

4. Run MERLIN with ManiSkill

# Activate MERLIN environment (with ManiSkill installed)
conda activate merlin-sim

# Set PYTHONPATH to MERLIN
export PYTHONPATH=/path/to/calhacks:$PYTHONPATH

# Run agent mission in ManiSkill
python examples/test_maniskill.py

Expected:

• ManiSkill environment loads
• Agent executes pick-and-place in sim
• Results printed to console

5. Full Integration Test

# Run MERLIN with ManiSkill backend
python main.py --backend maniskill --agent simple --mission "Pick and place demo"

Expected Output:

{
  "ok": true,
  "backend": "maniskill",
  "agent": "SimpleAgent",
  "mission": "Pick and place demo",
  "result": "Mission complete:\nInitial status: battery=95.0%\nNavigated to object at [...]\n..."
}

Architecture

┌─────────────────────────────────────┐
│        SimpleAgent                  │
│   (hardcoded pick-and-place)        │
└──────────────┬──────────────────────┘
               │ agent.run_mission()
               ▼
┌──────────────────────────────────────┐
│     MCP Server (5 tools)             │
│  navigate_to, grasp, release, etc    │
└──────────────┬──────────────────────┘
               │ execute_tool()
               ▼
┌──────────────────────────────────────┐
│    StateMachine (60 Hz loop)         │
│  IDLE → NAVIGATING → MANIPULATING    │
└──────────────┬──────────────────────┘
               │ state-specific logic
               ▼
┌──────────────────────────────────────┐
│  ManiSkillController                 │
│  (XLeRobot in ReplicaCAD scene)      │
└──────────────┬──────────────────────┘
               │ gymnasium API
               ▼
    XLeRobot Robot (Physics Sim)
     - 7 DOF arm
     - gripper
     - sensors

Next Steps

Immediate (sim validation)

1. Run test_maniskill.py and verify agent controls robot
2. Modify mission prompts to test different behaviors
3. Add custom control policies in _build_action() method

Before Real Hardware

1. Record trajectories from successful sim missions
2. Test transfer learning (sim → real)
3. Validate gripper force/torque limits
4. Test emergency stop procedures

Real Hardware (XLE Robot)

1. Create XLEController backend (rest API or ROS2)
2. Test low-level commands match sim
3. Validate sensor readings
4. Deploy trained agents

Troubleshooting

ManiSkill import error

Error: cannot import mani_skill.envs
Solution: pip install --upgrade mani-skill

XLeRobot files not found

Error: Robot xlerobot_single not registered
Solution: Verify XLeRobot files copied to ManiSkill package
         Check /agents/robots/__init__.py has xlerobot import

Slow simulation

Solution 1: Use --shader="default" (faster than ray-tracing)
Solution 2: Reduce render resolution or use headless mode
Solution 3: Upgrade GPU drivers

Observation extraction failing

Solution: Print env.observation_space to see observation keys
         Modify _extract_pose() in maniskill.py to match actual keys

Performance

Files Modified

• merlin/hardware/maniskill.py - ManiSkillController (new)
• merlin/hardware/__init__.py - Factory support (updated)
• examples/test_maniskill.py - Integration test (new)
• SETUP_MANISKILL.md - This guide (new)

Resources

• XLeRobot Documentation
• ManiSkill Documentation
• ManiSkill XLeRobot Setup
• MERLIN Architecture

Ready to validate your robot missions in simulation before real hardware! 🚀