Reward Penalty Engineering

Purpose

This skill teaches the methodology of reward/penalty exploration — how to discover, test, evaluate, and archive reward signals. It is a process guide, not a recipe book.

• How to identify what reward/penalty to try next
• How to formulate a hypothesis and test it
• How to evaluate whether a reward change helped
• How to archive findings in the reward library for reuse

IMPORTANT:

• The reward function lives in starter_kit/navigation*/vbot/vbot_*_np.py → _compute_reward().
• Reward weights are in starter_kit/navigation*/vbot/cfg.py → RewardConfig.scales dict.
• Current reward component details, default values, and search ranges are documented in starter_kit_docs/{task-name}/Task_Reference.md.
• Anti-laziness mechanisms (conditional alive_bonus, time_decay, successful truncation) are active. Do NOT remove them.
• Do NOT re-implement the reward function from scratch. Always check the existing implementation and documentation before making changes. Modify weights or add new terms incrementally with clear hypotheses, testing, and evaluation and archiving.

This skill does NOT contain reward component examples or scale tables. Those live in their respective locations:

What	Where
Component reference & scale ranges	starter_kit_schedule/templates/reward_config_template.yaml
Archived reward/penalty instances	starter_kit_schedule/reward_library/
Terrain strategies & reward code	quadruped-competition-tutor skill
Stage-specific reward overrides	curriculum-learning skill
Reward weight search spaces	hyperparameter-optimization skill
Visual reward debugging	subagent-copilot-cli skill

When to Use This Skill

The Exploration Cycle

Reward engineering is iterative. Every change follows this cycle:

    ┌──────────────┐
    │   DIAGNOSE   │ ← What behavior is wrong?
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │  HYPOTHESIZE  │ ← What reward signal could fix it?
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │   IMPLEMENT   │ ← Minimal change, one variable at a time
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │     TEST      │ ← Short run (1-2M steps), multiple seeds
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │   EVALUATE    │ ← Did the hypothesis hold?
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │   ARCHIVE     │ ← Record result in reward library
    └──────┬───────┘
           │
           ▼
      Next cycle

Rule: Never change more than one reward dimension per cycle. If you change both the termination penalty AND add a new gait reward, you cannot attribute outcomes.

Phase 1: Diagnose

Behavioral Signals

Before touching rewards, identify what behavior is wrong. Not "the reward is too low" but a concrete observable:

Diagnostic Commands

# 1. Watch the policy — ALWAYS start here before looking at numbers
uv run scripts/play.py --env <env-name>

# 2. Train with rendering to see behavior in real time
uv run scripts/train.py --env <env-name> --render

# 3. TensorBoard for reward curves
uv run tensorboard --logdir runs/<env-name>

Visual Diagnosis

Use subagent-copilot-cli to analyze simulation frames and training curves:

# Describe what you see, ask what reward signal is missing
copilot --model gpt-4.1 --allow-all -p "Watch this simulation frame. The robot is <describe behavior>. What reward signal might cause this?" -s

Key insight: A reward signal is "missing" if the agent has no gradient pointing toward the desired behavior in its current state. The fix may be a new reward, a penalty, or reshaping an existing one.

Phase 2: Hypothesize

Formulating a Good Hypothesis

A testable reward hypothesis has three parts:

1. Behavior target: What the robot should do differently
2. Signal mechanism: What mathematical signal encodes that behavior
3. Expected side effect: What might go wrong

Template:

"If I add/modify <signal> with weight <w>, the robot should <desired behavior>, but might also <risk>."

Discovery Strategies

When you don't know what to try, use these strategies to generate candidates:

Strategy 1: Inversion

Take the undesired behavior and directly penalize it.

Robot bouncing → penalize vertical velocity Robot spinning → penalize angular velocity Robot retreating → penalize backward displacement

Strategy 2: Shaping the Gradient

If the robot is stuck, the reward surface is flat in its current region. Add a signal that creates local gradient:

Robot stuck far from goal → Add distance-based shaping (sigmoid, exponential) Robot stuck near goal → Add fine-grained proximity bonus Robot stuck on terrain edge → Add progress checkpoints

Strategy 3: Proxy Decomposition

Break the competition score into component sub-goals and create a signal for each:

Final score = traversal + bonus zones + time bonus → Create separate signals for: forward progress, zone proximity, speed

Strategy 4: Biomimetic Analogy

What would a real quadruped "want" in this situation?

Stairs → lift knees higher Uneven ground → keep center of mass low Obstacles → slow down, increase awareness

Strategy 5: Ablation Discovery

Temporarily remove one existing reward and see what degrades:

# Remove component to see its effect
python scripts/train.py --env <env> --seed 42 --cfg-override "reward_config.scales.<component>=0.0"

If removing a component doesn't change behavior, it was irrelevant. If behavior collapses, it was critical.

Strategy 6: Competition-Score Alignment

Compare training reward to competition scoring rules. Gaps indicate missing signals:

Competition awards points for stopping in smiley zones → but training reward only rewards forward velocity → mismatch: need a "stop in zone" signal

Refer to quadruped-competition-tutor skill for competition scoring rules.

Strategy 7: Browse the Library

Check previously tried components in the reward library before inventing new ones:

# Browse archived reward components
Get-ChildItem starter_kit_schedule/reward_library/components/ | Select-Object Name
# Read a specific component's notes
Get-Content starter_kit_schedule/reward_library/components/<name>.yaml

Phase 3: Implement

Principles

1. One variable at a time — Change a single reward component per experiment
2. Minimal change — Prefer adjusting a weight before adding new code
3. Use existing infrastructure — Check reward_config_template.yaml for components that can be enabled/disabled before writing new code

Where to Make Changes

Change Magnitude Guidelines

When adjusting weights, use multiplicative steps not additive:

• Small adjustment: ×0.5 or ×2 (halve or double)
• Medium adjustment: ×0.1 or ×10
• Large adjustment: ×0.01 or ×100

For new components, start with a weight that produces reward magnitude comparable to existing dominant terms (check reward_breakdown logs).

Phase 4: Test

🔴 AutoML-First Testing (MANDATORY)

NEVER iterate manually with train.py, changing one reward weight, running, reading TensorBoard, killing, repeating. This is manual one-at-a-time search — slow, error-prone, and wasteful. ALWAYS use automl.py for batch reward hypothesis testing.

The correct workflow:

1. Add your reward hypothesis as a search range in REWARD_SEARCH_SPACE (in automl.py)
2. Run automl.py --hp-trials 8+ to test multiple configurations in one batch
3. Read the structured comparison in starter_kit_log/automl_*/report.md
4. Archive results in the reward library

Example: Testing near_target_speed activation radius

# In automl.py REWARD_SEARCH_SPACE:
"near_target_speed": {"type": "uniform", "low": -2.0, "high": -0.1},
"near_target_activation": {"type": "choice", "values": [0.3, 0.5, 1.0, 2.0]},

Then run: uv run starter_kit_schedule/scripts/automl.py --mode stage --hp-trials 8

Why AutoML is better:

• Tests N configurations in parallel with identical conditions
• Produces structured comparison table (reward, reached%, distance, ep_len)
• Bayesian suggestion improves with each trial
• No manual TensorBoard reading or experiment-killing needed

Exception: train.py is acceptable for testing ONLY when:

• Smoke test — --max-env-steps 200000 to verify new reward code compiles
• Visual debugging — --render to watch behavior qualitatively
• The reward requires NEW CODE in _compute_reward() (test compilation first, then use automl)

Experiment Protocol

# Record this BEFORE running the experiment
experiment:
  id: "RPE_YYYYMMDD_NNN"         # Auto-incrementing
  hypothesis: "<one sentence>"
  change: "<what exactly changed>"
  baseline: "<what to compare against>"
  environment: "<env id>"
  tool: "automl.py"               # MUST be automl.py, not train.py
  hp_trials: 8                    # Minimum 8 for meaningful comparison
  steps_per_trial: 5_000_000      # Each trial
  metrics_to_watch:
    - episode_reward_mean
    - reached_fraction
    - <behavior-specific metric>

Running Tests

# CORRECT: Batch search with AutoML (use this!)
uv run starter_kit_schedule/scripts/automl.py `
    --mode stage `
    --budget-hours 4 `
    --hp-trials 8

# Read results
Get-Content starter_kit_log/automl_*/report.md

# WRONG: Manual iteration (do NOT do this!)
# uv run scripts/train.py --env vbot_navigation_section001  # ← NEVER for parameter search

What Counts as "Enough" Testing

Phase 5: Evaluate

Comparison Checklist

After a test run, answer these questions:

1. Did the target behavior improve? (Watch the policy, not just numbers)
2. Did any other behavior degrade? (Side effects)
3. Is the improvement consistent across seeds? (Not lucky variance)
4. Is the reward curve healthy? (Monotonic improvement, no collapse)
5. Did training become slower? (Some signals slow convergence)

Decision Matrix

Quantitative Evaluation

Use starter_kit_schedule/scripts/analyze.py for systematic comparison:

# Compare experiments by reward metric
uv run starter_kit_schedule/scripts/analyze.py `
    --metric episode_reward_mean `
    --group-by reward_config

# Visual comparison via subagent
copilot --model gpt-4.1 --allow-all -p "Compare reward curves in runs/<exp_a>/ vs runs/<exp_b>/. Which converged faster? Which is more stable?" -s

Phase 6: Archive

Why Archive Everything

Every experiment result — positive or negative — is valuable. Negative results prevent re-trying failed ideas. Positive results enable reuse and combination.

Reward Library Location

All reward/penalty instances are archived in:

starter_kit_schedule/reward_library/
├── README.md                      # Library index and conventions
├── components/                    # Individual reward/penalty definitions
│   └── <component_name>.yaml      # One per reward idea
├── configs/                       # Complete reward configurations
│   └── <config_name>.yaml         # Tested combinations that work
└── rejected/                      # Ideas that didn't work
    └── <idea_name>.yaml           # With notes on why

Component Archive Format

# starter_kit_schedule/reward_library/components/<name>.yaml
name: "<descriptive name>"
type: "reward" | "penalty"
category: "navigation" | "stability" | "efficiency" | "terrain" | "gait" | "safety"
status: "tested" | "promising" | "rejected" | "untested"

description: "<what this signal does>"
hypothesis: "<why it should help>"
mechanism: "<mathematical formulation or pseudocode>"

weight_range:
  tested: [<min>, <max>]
  recommended: <value>

terrain_applicability:
  - flat
  - stairs
  - waves
  - obstacles

experiments:
  - id: "RPE_20260206_001"
    result: "improved" | "degraded" | "neutral"
    notes: "<brief finding>"

implementation:
  file: "<path to _compute_reward if custom code needed>"
  config_key: "<key in reward_scales dict>"
  requires_code_change: true | false

Config Archive Format

# starter_kit_schedule/reward_library/configs/<name>.yaml
name: "<config name>"
description: "<what this config is optimized for>"
environment: "<env id>"
date_tested: "YYYY-MM-DD"

reward_scales:
  position_tracking: 2.0
  heading_tracking: 1.0
  # ... all weights
  termination: -200.0

performance:
  episode_reward_mean: <value>
  success_rate: <value>
  notes: "<qualitative assessment>"

based_on: "<parent config name, if iterative>"

Archiving Commands

# After an experiment, archive the finding
# (manual for now — create the YAML by hand or instruct Copilot)

# List what's in the library
Get-ChildItem starter_kit_schedule/reward_library/components/ -Name
Get-ChildItem starter_kit_schedule/reward_library/configs/ -Name
Get-ChildItem starter_kit_schedule/reward_library/rejected/ -Name

Anti-Patterns to Avoid

Optimal Reward Calculation Principle

For each reward/penalty design, first read the workspace REPORT_NAV*.md files to locate or add representative test cases (for example: robot standing still, robot randomly walking, robot sprinting to the target and staying there safely). For each test case estimate the expected cumulative reward magnitudes (order-of-magnitude) before running long experiments — this helps set sensible search ranges and avoids reward hacking and reward-budget imbalances that lead to exploits.

🔴 Case Study: The Lazy Robot (Reward Hacking)

This case study documents a critical reward hacking failure discovered during long training, and the three-pronged fix. The specific numbers below are illustrative — see starter_kit_docs/{task-name}/Task_Reference.md for exact values.

Timeline

1. Short AutoML trials: Best trial achieved good reward, moderate reached%. Looked promising.
2. Long full training: Early metrics excellent — distance decreasing, high reach rate.
3. At ~30 min mark: Robot became lazy. Distance went UP, reached% dropped, episode length near max.
4. Root cause: alive_bonus × ~max_steps dwarfed arrival_bonus. The robot discovered standing still in a safe spot was the optimal strategy.

Detection Signals

The Fix: Anti-Laziness Trifecta

1. Conditional alive_bonus

# BEFORE (exploitable):
alive_bonus = scales["alive_bonus"]  # 0.5 every step, unconditionally

# AFTER (fixed):
alive_bonus = np.where(ever_reached, 0.0, scales["alive_bonus"])
# Once robot reaches target, alive bonus stops → no "stand around" exploit

2. Time decay on navigation rewards

time_decay = np.clip(1.0 - 0.5 * env_steps / max_episode_steps, 0.5, 1.0)
# Step 0: time_decay = 1.0 (full reward)
# Step 2000 (half episode): time_decay = 0.75
# Step 4000 (max): time_decay = 0.5
# Creates URGENCY — reach the goal early for maximum reward
# NOTE: Penalties are NOT multiplied by time_decay (stay full strength)

3. Massive arrival_bonus

# BEFORE: arrival_bonus too small (trivially beaten by alive_bonus accumulation)
# AFTER: arrival_bonus large enough to dominate
# Rule of thumb: arrival_bonus > alive_bonus × max_steps / 4

Lesson

Always audit the reward budget. Compute: what is the maximum reward achievable by the desired behavior vs. by the degenerate behavior? If they're close, the agent WILL find the exploit given enough training time.

Summary: Exploration Philosophy

1. Diagnose before prescribing — Watch the policy, identify the behavioral gap
2. One variable per experiment — Isolate cause and effect
3. Test short, test often — 2M step runs across 3 seeds reveal more than one 20M run
4. Archive everything — Positive results reuse, negative results prevent repetition
5. Methodology over recipes — The process of finding good rewards matters more than any specific reward
6. Library over memory — Store tried components in reward_library/, not in your head
7. Competition ≠ training — Verify that training rewards actually improve competition score

Converted and distributed by TomeVault — claim your Tome and manage your conversions.