Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?

Key points

• Small vision-language models may be able to read external evidence yet struggle to obtain it.1
• Across seven visual question-answering settings with students of at most 9B parameters, HC improves scores over bare students by 9.9-23.9 points, averaged over three independent builds per setting.1
• Interventions on five runtime decision types (invocation, selection, argument generation, evidence integration and abstention) show why this allocation matters: requesting evidence and generating open queries can be costly, whereas bounded choices and reading supplied text can remain useful student work.1
• Recompilation for a new student model helps when the transferred interface no longer fits the student.1

Sources (1)

• [1]Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 04:31 AM

  We introduce Harness Compilation (HC), an offline procedure that adapts the division of work between a frozen small VLM and its external harness.

  Small vision-language models may be able to read external evidence yet struggle to obtain it.

Extractive summary: sentences quoted from the sources.