Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation

Key points

• Reinforcement learning is commonly used to train language agents in interactive environments, but cannot be directly applied when rewards are unavailable.1
• SELF learns to predict environmental responses while distilling guidance from a feedback-conditioned self-teacher into the policy.1
• With Qwen3-8B, SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in $τ$-bench success rate, and by 10.71 and 3.57 percentage points in AppWorld task goal completion, respectively.1
• These results show that SELF uses environmental feedback more effectively within agentic self-distillation, improving agent capabilities.1

Sources (1)

• [1]Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 07:16 AM

  Given that environmental feedback contains rich supervision for modeling how the environment responds to agent actions, we introduce agentic SElf-distilLation with environmental Feedback modeling (SELF), a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation.

  Reinforcement learning is commonly used to train language agents in interactive environments, but cannot be directly applied when rewards are unavailable.

Extractive summary: sentences quoted from the sources.