Emergent Temporal Abstraction

Overview

This technique enables autoregressive models to learn hierarchical behaviors through discovery of temporal abstractions, dramatically accelerating learning on sparse-reward tasks.

Core Technique

Internal RL with Discovered Abstractions:

class HierarchicalARModel:
    def __init__(self):
        self.base_ar_model = PretrainedAutoregressive()
        self.metacontroller = MetacontrollerPolicy()
        self.abstract_controllers = nn.ModuleList()

    def forward_hierarchical(self, state):
        # Metacontroller generates abstract action sequence
        abstract_actions = self.metacontroller.sample_actions(state)

        # Each abstract action is a sequence of residual stream modifications
        output = self.base_ar_model.initial_forward(state)

        for t, abstract_action in enumerate(abstract_actions):
            # Apply abstract action via residual stream modification
            controller_output = self.abstract_controllers[abstract_action](output)
            output = output + controller_output  # Residual addition
            
            # Check switching condition
            if self.should_switch(output, t):
                break  # Move to next abstract action

        return output

Switching Gates and Temporal Patterns:

def switching_gate_mechanism(features, temperature=1.0):
    """
    Binary switching via gating, creating sparse temporal patterns.
    """
    gate_logits = nn.Linear(hidden_dim, 1)(features)
    gate_prob = sigmoid(gate_logits / temperature)
    
    # Gumbel-softmax for differentiable sampling
    gate_sample = gumbel_softmax(gate_prob)
    
    return gate_sample

RL in Abstract Space:

def abstract_space_rl(model, env):
    for episode in range(num_episodes):
        state = env.reset()
        abstract_actions = model.metacontroller.sample_actions(state)
        
        # Accumulate token-level transitions
        tokens = []
        for abstract_action in abstract_actions:
            token_sequence = model.forward_with_controller(abstract_action, state)
            tokens.extend(token_sequence)
            state = env.step(tokens)
        
        # RL update on abstract actions, not tokens
        reward = env.get_reward()
        log_prob = model.metacontroller.log_prob(abstract_actions)
        loss = -reward * log_prob
        loss.backward()

When to Use

Use when: Sparse-reward tasks, token-level RL too slow, hierarchical structure evident.

References

• Metacontroller for abstract action selection
• Residual stream modification via controllers
• Switching gates for temporal abstraction
• RL in abstract action space