Gemma4 E4B EXL3 - Alternative to Llama.cpp on Jetson Orin
My Jetson Orin–optimized engine, little-gemma V1.0, substantially outperforms llama.cpp.
Proof1 community thread
Key points
- Even after exhausting every practical GGUF option, however, it still falls short of the ideal performance level for Gemma E4B on the Jetson Orin Nano Super 8GB.
- I forked exllamav3 and ported key code from little-gemma V1.0.
- Although the decoding rate took a hit, overall voice-chat latency is lower.
- EXL3 on the Jetson Orin Nano isn’t for everyone, but if you need the superior prefill rate, this is it.
Sources (1)
- [1]Gemma4 E4B EXL3 - Alternative to Llama.cpp on Jetson Orinr/LocalLLaMA (top, daily) · Oct 11, 07:47 PM
My Jetson Orin–optimized engine, little-gemma V1.0, substantially outperforms llama.cpp.
Even after exhausting every practical GGUF option, however, it still falls short of the ideal performance level for Gemma E4B on the Jetson Orin Nano Super 8GB.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 11, 2026OpenMed 3.0 is out: Apache-2.0 clinical AI that runs fully local and never falls back to the cloud. 422 open issues if you want in on 3.1
- Oct 10, 2026Nace AI Open-Sources Drex 1.5: A 9B Decision Model That Scores Options, Not Text
- Oct 9, 2026Qwen/Qwen-Image-2.1-Turbo
- Oct 9, 2026AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis
- Oct 8, 2026Microsoft Joins the Local AI Push
- Oct 8, 2026ConwayResearch/Underdog-Saluki-27B-1.0