RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection

Key points

• Beyond vocabulary scaling, zero-shot generalization may benefit from reusable visual--semantic cues learned from seen data, including attributes, actions, states, and contextual relations.1
• We propose RT-DETR-World, a compact DETR-style detector that transfers the rich semantics conveyed by descriptions during training while retaining lightweight query--text matching at inference.1
• We construct GroundingCapv2 with three levels of supervision: category names for standard OVD, object descriptions conveying instance-level semantics, and image descriptions conveying object relations and scene context.1
• To help the compact detector absorb these semantics, we propose Dual-Path Description Alignment (DDA), combining a deployment-consistent MiniLM pathway with a training-only LLM teacher.1

Sources (1)

• [1]RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 06:01 AM

  Open-vocabulary detection (OVD) recognizes categories unseen during training through textual category queries, yet achieving strong generalization with real-time efficiency remains challenging.

  Beyond vocabulary scaling, zero-shot generalization may benefit from reusable visual--semantic cues learned from seen data, including attributes, actions, states, and contextual relations.

Extractive summary: sentences quoted from the sources.