Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.

Key points

• Upstream vLLM does not start this model on that machine at all: the model refuses --pipeline-parallel-size > 1, TP has to be 1/2/4/8 so six cards are useless for it, and a 24 GB card cannot hold it at TP=1.1
• Putting the 102 GB n-gram (PLE) table on SSD has just showed up as PR on upstream today (keeping it in RAM is the typical setup), and fp8 KV crashes on Ampere.1
• workload ctx llama.cpp-multigpu vLLM, this branch (int8 KV)1
• So: vLLM for prefill, single-user speed and seat count; llama.cpp if you want every user to see the same speed (I posted a similar set of fixes for llama earlier).1

Sources (1)

• [1]Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.
  r/LocalLLaMA (top, daily) · Oct 12, 08:37 AM

  This shows vLLM fixes and setup that let you put PLE on SSD and save 108GB of system RAM, use int8 kv and squeeze more users in your VRAM without the performace hit of fp8.

  Upstream vLLM does not start this model on that machine at all: the model refuses --pipeline-parallel-size > 1, TP has to be 1/2/4/8 so six cards are useless for it, and a 24 GB card cannot hold it at TP=1.

Extractive summary: sentences quoted from the sources.