Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization

Key points

• Multimodal large language models offer a unified interface, but extending vision-language models (VLMs) to raw I/Q signals requires tokenization that balances fidelity against a strict budget.1
• Thus, we propose BATok, a budget-adaptive signal tokenizer that adjusts token capacity to the input length while allocating that capacity according to the signal content.1
• BATok constructs candidate representations from signal-derived features using lightweight multi-resolution branches, then combines a local energy prior with learnable queries to resample these representations into compact signal tokens.1
• The resulting tokens are projected into the language embedding space of VLMs. We further introduce EMSpec-Instruct, a multimodal instruction dataset aligning raw I/Q signals, waterfall images, and language supervision for modulation recognition, structured detection, and language-conditioned signal grounding.1

Sources (1)

• [1]Instruction-Conditioned Electromagnetic Spectrum Understanding via Budget-Adaptive Signal Tokenization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 03:28 PM

  Electromagnetic spectrum monitoring increasingly requires flexible analysis beyond task-specific recognition and detection.

  Multimodal large language models offer a unified interface, but extending vision-language models (VLMs) to raw I/Q signals requires tokenization that balances fidelity against a strict budget.

Extractive summary: sentences quoted from the sources.