Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents

Key points

• Small language models (SLMs) are attractive as local agent controllers because they reduce remote inference, latency, and deployment footprint, yet structured tool errors can cause an agent step to fail.1
• Existing routers typically select a model once per query.1
• In a separate multi-turn evaluation, STEPGATE achieves 69.0% trajectory success and 84.0% action success using only 30.0% cloud actions, compared with 48.0%/70.5% local-only, 60.0%/78.2% random escalation, and 57.0%/77.1% query-level routing (strong-only achieves 82.0% trajectory success at 100% cloud actions).1
• These results suggest that step-level escalation recovers a large share of the performance gap to the stronger Qwen2.5-7B backend at a matched cloud-action rate while transmitting fewer tokens remotely.1

Sources (1)

• [1]Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 06:09 AM

  We propose STEPGATE, an uncertainty-aware handoff framework that scores each local SLM action and selectively escalates challenging steps to a stronger model.

  Small language models (SLMs) are attractive as local agent controllers because they reduce remote inference, latency, and deployment footprint, yet structured tool errors can cause an agent step to fail.

Extractive summary: sentences quoted from the sources.