Zhangdanyang Amd Lumen Rl Lumenrl Debug

LumenRL Upstream Debug Guide

Core Principle

LumenRL depends on three upstream libraries. When a bug is found during RL training or inference, first identify which library owns the buggy code, then fix it in the correct location:

Never patch upstream bugs directly in lumenrl/ with workarounds. Always fix at the source.

Identifying Bug Origin

Use this decision tree:

1. Stack trace contains lumen.quantize, lumen.models, lumen.ops → Lumen bug → fix in third_party/Lumen/
2. Stack trace contains atom., atom.model_engine, atom.model_ops → ATOM bug → fix in third_party/ATOM/
3. Stack trace contains aiter., aiter_fused_, or ASM/CK kernel names → AITER bug → fix in AITER submodule
4. Stack trace contains mori., mori.ccl, mori.ep → MORI bug → fix in MORI submodule
5. Stack trace only contains lumenrl/ → LumenRL bug → fix in lumenrl/

For ambiguous cases (e.g., wrong config passed from LumenRL to Lumen), fix the caller in lumenrl/.

Workflow

Step 1: Reproduce and log

# Document in .cursor/tmp-rl-bugs.md
echo "## [$(date)] Bug: <title>" >> .cursor/tmp-rl-bugs.md
echo "- Symptom: ..." >> .cursor/tmp-rl-bugs.md
echo "- Stack trace: ..." >> .cursor/tmp-rl-bugs.md
echo "- Origin: Lumen | ATOM | AITER | MORI | LumenRL" >> .cursor/tmp-rl-bugs.md

Step 2: Fix in the correct location

For Lumen bugs:

cd third_party/Lumen
# make the fix
git add -A && git commit -m "fix: <description>"
git push origin dev/RL
cd ../..
git add third_party/Lumen
git commit -m "bump Lumen: fix <description>"

For ATOM bugs:

cd third_party/ATOM
# make the fix
git add -A && git commit -m "fix: <description>"
git push origin <branch>
cd ../..
git add third_party/ATOM
git commit -m "bump ATOM: fix <description>"

For AITER bugs:

# AITER lives at: https://github.com/ZhangDanyang-AMD/aiter.git (lumen/triton_kernels branch)
cd <aiter-submodule-path>
# make the fix
git add -A && git commit -m "fix: <description>"
git push origin lumen/triton_kernels
cd <back-to-lumenrl>
git add <aiter-submodule-path>
git commit -m "bump AITER: fix <description>"

For MORI bugs:

# MORI lives at: https://github.com/ZhangDanyang-AMD/mori.git (sdma-new branch)
cd <mori-submodule-path>
# make the fix
git add -A && git commit -m "fix: <description>"
git push origin sdma-new
cd <back-to-lumenrl>
git add <mori-submodule-path>
git commit -m "bump MORI: fix <description>"

Step 3: Verify

After fixing, re-run the failing test or training step to confirm the fix. Update .cursor/tmp-rl-bugs.md with the resolution.

Hard Rules

• Never monkey-patch upstream code from lumenrl/ (e.g., lumen.quantize.some_fn = patched_fn)
• Never copy upstream files into lumenrl/ to work around a bug
• Always commit the fix in the upstream submodule first, then bump the submodule ref in Lumen-RL
• If a fix requires coordinated changes across multiple repos (e.g., Lumen API change + LumenRL caller update), commit the upstream change first, then the LumenRL adaptation
• Document every upstream bug and fix in .cursor/tmp-rl-bugs.md

Common Upstream Bug Patterns

Pairing

Pair with lumenrl-coding for code changes in lumenrl/. Pair with lumenrl-training for RL training stability issues.

Source: ZhangDanyang-AMD/Lumen-RL — distributed by TomeVault.