Speaking the Navigator's Language: Trajectory-Grounded Instruction Translation for Frozen Aerial VLN Agents

Key points

• Aerial vision-and-language navigation (VLN) agents are typically trained on detail-rich, trajectory-aligned commands, whereas users issue short, intent-driven instructions; on a frozen OpenFly navigator, this instruction gap drops success rate (SR) from $31.03%$ to $11.33%$.1
• To scale translator training, we prompt a language model with human-written style examples to convert original commands into paired, intent-centered Weak commands, which yield $15.27%$ SR.1
• The resulting Weak-trained translator raises Weak-input SR to $37.93%$ and transfers zero-shot to real human instructions ($11.33%{\rightarrow}32.51%$); it also improves held-out OpenFly ($4.95%{\rightarrow}20.79%$) and yields recovery on CityNav and AirVLN.1

Sources (1)

• [1]Speaking the Navigator's Language: Trajectory-Grounded Instruction Translation for Frozen Aerial VLN Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 01:52 PM

  We introduce the Trajectory-Grounded Instruction Translator (TGIT), a front-end that keeps the navigator frozen and translates Weak inputs into agent-executable commands by learning from its trajectory outcomes.

  Aerial vision-and-language navigation (VLN) agents are typically trained on detail-rich, trajectory-aligned commands, whereas users issue short, intent-driven instructions; on a frozen OpenFly navigator, this instruction gap drops success rate (SR) from $31.03%$ to $11.33%$.

Extractive summary: sentences quoted from the sources.