Breaking the Space Barrier and its Application to Language Model Inference
Language models are more and more often asked for structured output: JSON that follows a schema, or a tool call with typed arguments.
Key points
- A small machine, an automaton, enforces the format by forbidding the tokens that would break it.
- Graphs in which only a few paths join any two points are a classical object of complexity theory, and our theoretical result settles an open question about them: one can decide whether such a graph connects two points while verifying that it really has few paths, with very little memory.
- The constructions behind the proofs become an inference engine: text the format forces is written without running the model, the mask is recomputed on the GPU without any table, recursive formats use a small stack, every output stays valid under a token limit, and independent fields are decoded in parallel and verified.
- On one 16 GB Apple M2 Pro with Qwen3.5-2B and 4B, against MLX with llguidance, the standard setup for this hardware, schema-constrained extraction finishes 1.2- 1.3x sooner with the same answers, a grammar costs 3 MB instead of up to 1.5 GB, one server holds sixteen grammars where tables run out of memory, and sixteen tool-calling agents finish 2.5x sooner.
Sources (1)
- [1]Breaking the Space Barrier and its Application to Language Model InferencearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 09:29 PM
Language models are more and more often asked for structured output: JSON that follows a schema, or a tool call with typed arguments.
A small machine, an automaton, enforces the format by forbidding the tokens that would break it.
Extractive summary: sentences quoted from the sources.