Improve token per second without touching quant
Spent the past month tweaking and experimenting with many different numbers to achieve 30tps.
Key points
- -window11 (no wsl switching since im not used to the environment)
- Im looking for any improvements to achieve maybe 40tps? without touching quant at all, q8 and q4kxl remains.
- I've experiment with threads at 6 is the best out of (4,8,12) Other than that im not sure on what to improve to achieve higher tps, any advices? appreciate it. ctx remains 100,000
Sources (1)
- [1]Improve token per second without touching quantr/LocalLLaMA (top, daily) · Oct 10, 12:05 PM
Spent the past month tweaking and experimenting with many different numbers to achieve 30tps.
-window11 (no wsl switching since im not used to the environment)
Extractive summary: sentences quoted from the sources.