Paper Rl Dqc

Decoupled Q-Chunking (DQC)

Decoupled Q-Chunking Qiyang Li, Seohong Park, Sergey Levine (UC Berkeley) arXiv 2025 | Paper | Code | Website

TL;DR

DQC improves upon Q-Chunking (QC) by decoupling the critic chunk size from the policy chunk size. Long critic chunks (h) accelerate value learning; short policy chunks (h_a) improve policy learnability and reactivity. A distilled partial-chunk critic bridges the two via expectile/quantile regression.

Problem

• TD methods suffer from slow value propagation in long-horizon sparse-reward tasks
• N-step returns speed up propagation but introduce off-policy bias
• QC groups actions into chunks for the critic, but forces the policy to output the same long chunk — hard to learn and reduces closed-loop reactivity
• QC assumes open-loop consistency (OLC) in the data, which real data rarely satisfies

Key Contributions

1. Theoretical analysis: QC is near-optimal under OLC; DQC is near-optimal under the weaker BOV (Bounded Optimality Variability) condition
2. Decoupled architecture: critic uses chunk size h (e.g. 25), policy uses chunk size h_a (e.g. 5 or 1)
3. Distilled partial critic: bridges the gap between full-chunk critic and short-chunk policy via implicit maximization

Method Overview

Architecture (5 networks, jointly trained)

All Q/V outputs are in logit space (sigmoid maps to [0,1] probabilities).

Training Pipeline

Step 1 — Chunk critic loss (only when use_chunk_critic=True):

target = reward + gamma^h * mask * sigmoid(V(s_{t+h}, g))
L_chunk = BCE(Q_chunk_logit(s, a_{0:h}, g), clip(target, 0, 1))

Step 2 — Action critic (distillation) loss:

# Target comes from chunk_critic (if DQC) or n-step return (if QC/NS)
target_v = sigmoid(Q_chunk(s, a_{0:h}, g))   # DQC
# OR: target_v = reward + gamma^h * mask * sigmoid(V(s_{t+h}, g))  # QC/NS

# Expectile regression to distill into partial-chunk critic
weight = kappa_d if target_v >= q else (1 - kappa_d)
L_distill = weight * BCE(Q^P_logit, target_v)

Step 3 — Value loss (implicit value backup):

ex_q = agg(sigmoid(target_Q^P(s, a_{0:h_a}, g)))  # min or mean over ensemble
weight = kappa_b if ex_q >= v else (1 - kappa_b)
L_value = weight * BCE(V_logit, ex_q)  # or quantile loss on logits

Step 4 — Actor loss (conditional flow matching):

x_0 ~ N(0, I),  x_1 = a_{0:h_a}  (ground-truth partial chunk)
x_t = (1-t)*x_0 + t*x_1
L_actor = ||f_theta(s, x_t, t) - (x_1 - x_0)||^2

Step 5 — Target update: EMA on action_critic with tau=0.005.

Inference (Best-of-N)

1. Sample N action chunks from flow policy (10 Euler steps)
2. Score each with action_critic (min or mean over ensemble)
3. Select highest-scoring chunk
4. Execute first action_chunk_eval_size actions, then re-plan

Paper-Code Mapping

Key Hyperparameters

Per-Environment Tuned Parameters

Baselines & Variants

Dependencies

ogbench
jax >= 0.4.26
flax >= 0.8.4
distrax >= 0.1.5
ml_collections
matplotlib
moviepy
wandb

Built on top of horizon-reduction codebase. Uses OGBench environments.

Relationship to Q-Chunking (QC)

DQC is the direct follow-up to QC (same first author). Key differences:

Gotchas & Tips

• All Q/V values are in logit space — always apply sigmoid() before interpreting as probabilities
• The valids mask is critical for action chunks near trajectory boundaries
• MUJOCO_GL=egl is required for headless rendering on GPU servers
• Large datasets (100m, 1b) must be downloaded separately following these instructions
• For humanoidmaze-giant and puzzle-4x5, set --dataset_dir=None (uses default OGBench download)
• Wandb logging is set to offline mode by default
• 10 seeds (100001–1000010) are used for paper results