# G2rl Gradient Guided

> Guide LLM exploration through the model's own gradient geometry rather than external signals. Extract sequence-level gradient features measuring how tokens would reshape output distributions. Reward responses introducing novel gradient directions while deemphasizing redundant ones. Achieve orthogonal gradient directions and improved accuracy.

- Skill: `adu2021/g2rl-gradient-guided` (Agent Skill)
- Install (CLI): `npx skillmds@latest add adu2021/g2rl-gradient-guided`
- Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/g2rl-gradient-guided/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: adu2021 (https://skillmd.com/u/adu2021)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/adu2021/g2rl-gradient-guided

---


## Skill Summary

G2RL introduces gradient-guided reinforcement learning that leverages the model's own gradient geometry for exploration guidance. Rather than external signals like entropy or semantic embeddings, the method computes sequence-level features from final-layer gradient sensitivity, comparing responses by how differently they would update policy parameters. Correct but geometrically redundant answers are slightly downweighted; correct answers with novel gradient directions receive bonuses. The approach achieves "self-guided" exploration: the model learns which variations meaningfully reshape its own parameters.

## When To Use

- Training language models with exploration that reflects actual policy update dynamics
- Scenarios where external exploration signals misalign with meaningful parameter updates
- Projects exploring policy gradient interpretability and update geometry
- Research on self-guided exploration in reinforcement learning

## When NOT To Use

- Scenarios where external exploration signals (entropy, diversity) already work well
- Real-time applications where gradient feature computation adds overhead
- Domains where exploration doesn't benefit from update-direction alignment
- Models with strict computational budgets prohibiting additional calculations

## Core Technique

Three key components enable gradient-guided exploration:

**1. Gradient Feature Extraction**
For each response token, extract "sequence-level feature" measuring how that token would reshape the model's output distribution. Compute the token's first-order sensitivity—how strongly the token would affect final predictions through gradient updates.

**2. Exploration Scoring**
Compare responses within a group based on how differently they would update policy parameters. Those introducing novel gradient directions receive reward bonuses, while redundant ones are deemphasized. This measures orthogonality of gradient directions.

**3. Reward Shaping**
Apply bounded multiplicative factor adjusting rewards asymmetrically:
- Correct but geometrically redundant answers: slightly downweighted
- Correct answers with novel gradient directions: boosted

The asymmetry encourages exploration of underrepresented gradient regions without discouraging correctness.

## Key Insight

"Exploration often becomes diffuse, misaligned, or fragile" when driven by external signals that don't reflect the policy's actual update dynamics. G2RL achieves "self-guided" exploration where the model learns which variations meaningfully reshape its own parameters, improving accuracy across math and reasoning benchmarks.

## Implementation Notes

Compute gradient features via final-layer gradient sensitivity during forward passes. Group responses by prompt. Compute pairwise gradient direction similarity/orthogonality. Score based on novel gradient directions. Apply asymmetric reward shaping. Fine-tune model with shaped rewards via policy gradient.

## References

- Original paper: Can LLMs Guide Their Own Exploration? (Dec 2025)
- Policy gradient interpretation and geometry
- Exploration in reinforcement learning