Reinforcement Learning Training Skill
You are an expert in reinforcement learning, specifically Deep Q-Networks (DQN) for game AI.
Project Context
This CTF-AI project uses:
- DQN Agent with Double DQN, Prioritized Experience Replay, Huber loss
- 19-dimensional state vector: player(5) + target(6) + enemy(4) + global(4)
- 3 actions: DEFENCE(0), SCORING(1), SAVING(2)
- Gymnasium environment for training interface
- PyTorch for neural network
Key Files
backend/lib/reinforcement_learning/agent.py- DQNAgent classbackend/lib/reinforcement_learning/network.py- Neural network architecturebackend/lib/reinforcement_learning/reward_calculator.py- Reward functionbackend/lib/reinforcement_learning/state_extractor.py- Feature extractionbackend/lib/reinforcement_learning/training/train_gym.py- Training script
When Asked to Train
# Offline training
python3 -m lib.reinforcement_learning.training.train_gym 8080 --algorithm CustomDQN --train-offline
# Online training (requires game server)
python3 -m lib.reinforcement_learning.training.train_gym 34712 --algorithm CustomDQN
Hyperparameter Guidelines
| Parameter | Default | Tuning Advice |
|---|---|---|
| Learning rate | 0.0005 | Lower (0.0001) if unstable, higher (0.001) if slow |
| Epsilon decay | 0.998 | Slower (0.999) for more exploration |
| Gamma | 0.99 | Lower (0.95) for shorter-term focus |
| Batch size | 32 | Larger (64, 128) for more stable gradients |
| Target update | 50 | More frequent (20) for faster learning |
Debugging Training Issues
- Reward not improving: Check reward_calculator.py, ensure rewards are balanced
- Q-values exploding: Enable gradient clipping, reduce learning rate
- Agent stuck in one action: Increase exploration (epsilon), check state features
- Training unstable: Enable Double DQN, use Huber loss, increase buffer size
Reward Tuning
Current reward structure:
- Score flag: +150 (primary objective)
- Pick up flag: +10
- Lose flag: -40
- Get captured: -25
- Step penalty: -0.02
Adjust in reward_calculator.py if agent behavior is suboptimal.
Source: sdd330/ctf-ai — distributed by TomeVault.