Understanding how reinforcement learning (RL) agents behave when optimizing for rewards across multiple episodes—rather than just within a single episode—is a critical gap in AI safety research. Many alignment techniques assume specific optimization horizons, and unexpected exploitation of cross-episodic rewards could undermine these assumptions. One way to explore this would be to empirically test how RL agents respond to side channels that allow them to boost rewards in future episodes.

Testing Cross-Episodic Reward Exploitation

The project could involve training RL agents in environments where a side channel enables cross-episodic rewards. For example:

• A grid world where agents can leave "traps" in one episode to increase rewards in the next.
• Population-Based Training (PBT) to see if agents naturally discover and exploit these channels.
• Explicit cues, like visual indicators, to test whether simpler planning algorithms can also exploit cross-episodic rewards.

Key metrics would include the frequency of side-channel use, the reward delta between exploiters and non-exploiters, and generalization performance on held-out tasks. This could help distinguish strategic planning from overfitting.

Why This Matters

AI safety researchers could use these insights to refine alignment techniques, while RL practitioners might design more robust training environments. Theoretical researchers could also improve models of agent planning. The project’s novelty lies in its focus on cross-episodic side channels—a largely unexplored area in safety research.

Execution Strategy

An MVP could start with a simple grid-world environment and PBT, then scale to more complex setups if results are promising. Challenges like myopic agent behavior could be addressed by gradually increasing side-channel rewards. Compared to existing work like RL² or Meta-RL, this project isolates cross-episodic optimization as a safety-specific variable, offering unique insights.

While primarily a research project, potential applications could include consulting for AI companies or licensing tailored training frameworks for safety testing.