Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

Key points

• Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency.1
• We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome.1
• We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return.1
• We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender.1

Sources (1)

• [1]Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:30 PM

  Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision.

  Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency.

Extractive summary: sentences quoted from the sources.