Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models

Key points

• We argue that agents can learn more effectively by prospectively comparing the consequences of candidate actions rather than diagnosing failures only from realized returns.1
• We introduce CASTLE (Counterfactual Action-conditioned Semantic Tokens for Local Execution in Decentralized MARL), an offline-training, online-in-context guidance framework with two complementary world models.1
• A Local Dynamics World Model, offline pre-trained over agents' local trajectories, summarizes the agent's local trajectory dynamics and partial observability, while a Semantic-Social World Model predicts compact short-horizon task and social consequences for each candidate ego action.1
• The latter is trained from counterfactual simulator rollouts that expose plausible teammate and opponent responses to alternative actions taken from the same logged rollout state.1

Sources (1)

• [1]Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 03:53 AM

  Fully decentralized multi-agent reinforcement learning (MARL), also referred to as independent learning, requires each agent to learn and act using only its local information and experience, without a centralized critic or inter-agent communication.

  We argue that agents can learn more effectively by prospectively comparing the consequences of candidate actions rather than diagnosing failures only from realized returns.

Extractive summary: sentences quoted from the sources.