Taking Tokenization off the Host for Agentic LLM Serving
We present GraphTok, a GPU tokenizer that runs the entire encoding pipeline, from pre-tokenization through byte-pair encoding (BPE) merging to token-ID assembly, as one replayable CUDA graph.
ProofPaper ↗
Key points
- Agentic workloads resend long contexts that barely change between calls.
- Tokenization, a CPU stage long treated as negligible, becomes the largest part of time-to-first-token (TTFT), and a frontend that tokenizes for a whole GPU fleet limits how many GPUs it can feed.
- GraphTok reproduces the reference token IDs bit-for-bit on GPT-2, Llama 3, Qwen 2.5, DeepSeek-V3, and Gemma 3, hands short inputs to a CPU engine where the GPU cannot win, and plugs into unmodified vLLM and NVIDIA Dynamo.
- GraphTok encodes one million GPT-2 tokens in 1.7-3.2 ms including data transfers, and on Llama 3 it is 4.7-6.0x faster than gigatoken, the strongest CPU tokenizer.
Sources (1)
- [1]Taking Tokenization off the Host for Agentic LLM ServingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 10:05 AM
We present GraphTok, a GPU tokenizer that runs the entire encoding pipeline, from pre-tokenization through byte-pair encoding (BPE) merging to token-ID assembly, as one replayable CUDA graph.
Agentic workloads resend long contexts that barely change between calls.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 8, 2026unslothai/unsloth v0.1.905-beta: Sandboxing is here!
- Oct 8, 2026The State of AI Report 2026
- Oct 8, 2026ConwayResearch/Underdog-Saluki-27B-1.0
- Oct 8, 2026A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization
- Oct 7, 2026Evaluating Rubric Generation with Interventional Transfer
- Oct 7, 2026unslothai/unsloth v0.1.904-beta: Train your own Decision model