Sensor-Language-Action Models
We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model.
ProofPaper ↗
Key points
- Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces.
- SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence.
- We build a large-scale SLA benchmark consisting of datasets that span more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns user context, sensor dynamics, and action evidence.
- Building on this framework, we present OpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation.
Sources (1)
- [1]Sensor-Language-Action ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 12:26 PM
We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model.
Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces.
Extractive summary: sentences quoted from the sources.