Exciting Update: llama.cpp Now Supports Gemma 4 MTP!
The AI and machine learning community has been eagerly awaiting this development, and we’re thrilled to announce that support for Gemma 4 MTP has finally been integrated into llama.cpp. This enhancement unlocks new possibilities for deploying high-performance language models with improved efficiency and versatility.
Getting Started with Gemma 4 MTP in llama.cpp
To demonstrate the capabilities, I successfully configured llama.cpp using a QAT (quantization-aware training) GGUF model with the Gemma 4 31B assistant model. Here’s a step-by-step overview of the process:
1. Obtain the Model
The assistant model is available for download from Hugging Face:
Download Gemma 4 31B Assistant Model
2. Launch the Server
Using the provided CLI command, you can run the model efficiently. Feel free to customize the parameters to suit your hardware setup and application needs:
bash
llama-server.exe --ctx-size 64000 -fa on --jinja --reasoning on --host 0.0.0.0 --port 8502 --fit on --fit-ctx 64000 -kvu --no-mmap -ctk q8_0 -ctv q8_0 -np 1 --dry-multiplier 0.8 --repeat-last-n 1024 --repeat-penalty 1.2 --seed -1 --samplers "temperature;top_k;top_p;min_p;penalties;dry" -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp --spec-draft-n-max 2 --model-draft gemma-4-31B-it-qat-q4_0-unquantized-assistant-q8_0.gguf
3. Performance Benchmarks
In my testing on a high-end Nvidia RTX 5090 setup, I observed a remarkable performance increase — from approximately 55-60 tokens per second to over 100 tokens per second. This nearly doubles the throughput, demonstrating the efficiency gains achievable with Gemma 4 MTP support.
Conclusion
The integration of Gemma 4 MTP into llama.cpp marks a significant milestone for AI practitioners seeking robust, efficient language models. Whether for research, development, or deployment, this update provides enhanced performance and flexibility.
Stay tuned for more updates as the community continues to optimize and innovate with these powerful tools. Happy coding!
