Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?
Visual foundation models (VFMs) are increasingly integrated into end-to-end autonomous driving for their powerful representations, yet it remains unclear when these representations improve driving performance.
ProofPaper ↗
Key points
- To investigate this question, we introduce ViRA, a planner-agnostic visual representation alignment framework that keeps the planner architecture and inference cost unchanged.
- Our study reveals three findings: (1) VFM-guided visual representations consistently improve driving performance across diverse end-to-end planners, with gains extending to zero-shot closed-loop evaluation.
- Guided by these findings, we develop ViRA-Diffusion, a diffusion-based planner trained without auxiliary perception supervision, which achieves 92.3 EPDMS on NAVSIM v2 navtest, outperforming recent methods in our comparison by at least 1.9 points.
- The results motivate jointly considering target selection and planner supervision when integrating VFMs into end-to-end autonomous driving.
Sources (1)
- [1]Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 08:54 AM
Visual foundation models (VFMs) are increasingly integrated into end-to-end autonomous driving for their powerful representations, yet it remains unclear when these representations improve driving performance.
To investigate this question, we introduce ViRA, a planner-agnostic visual representation alignment framework that keeps the planner architecture and inference cost unchanged.
Extractive summary: sentences quoted from the sources.