Gemma 4 E2B Instruct — W4A16 Quantized AutoRound

This repository hosts W4A16 INT4-quantized versions of google/gemma-4-E2B-it, a multimodal mixture-of-experts model supporting text, vision, and audio inputs. Two quantized variants are available:

Note: Only the language model (LM) layers are quantized to INT4. The vision tower, audio tower, and multimodal projectors are kept at full precision (BF16) to preserve multimodal quality.

Quantization Details

Model Architecture

Gemma 4 E2B is a multimodal MoE model (Gemma4ForConditionalGeneration) with:

Usage

vLLM Inference

The recommended way to serve this model is via the official vllm/vllm-openai:gemma4 Docker image, which ships vLLM v0.19.1 with the latest Transformers patches required for Gemma 4.

Serve with Docker (recommended)

docker run --gpus all --rm -p 8000:8000 \
  vllm/vllm-openai:gemma4 \
  vllm serve Vishva007/gemma-4-E2B-it-W4A16-AutoRound \
    --served-model-name Gemma-4-E2B-it \
    --quantization autoround \
    --kv-cache-dtype auto \
    --max-num-batched-tokens 16384 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --dtype bfloat16 \
    --max-model-len 18432 \
    --enable-auto-tool-choice \
    --tool-call-parser gemma4 \
    --port 8000 \
    --default-chat-template-kwargs '{"enable_thinking": false}' \
    --mm-processor-kwargs '{"max_soft_tokens": 560}'

Direct `vllm serve` (vLLM ≥ 0.19.0)

vllm serve Vishva007/gemma-4-E2B-it-W4A16-AutoRound \
  --served-model-name Gemma-4-E2B-it \
  --quantization autoround \
  --kv-cache-dtype auto \
  --max-num-batched-tokens 16384 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --dtype bfloat16 \
  --max-model-len 18432 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --port 8000 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --mm-processor-kwargs '{"max_soft_tokens": 560}'

`max_soft_tokens` — Image Token Budget

The max_soft_tokens parameter controls how many visual tokens are allocated per image. Higher values give richer image representations at the cost of context length and throughput.

Pass it via --mm-processor-kwargs '{"max_soft_tokens": <value>}'.

OpenAI-compatible API call

Once the server is running, query it like any OpenAI-compatible endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see."},
                {"type": "image_url", "image_url": {"url": "https://..."}},
            ],
        }
    ],
    max_tokens=512,
)
print(response.choices.message.content)

The quantized model was then exported in both AutoRound and GPTQ formats and pushed to Hugging Face Hub.

Limitations & Notes

• RTN mode (iters=0) is used instead of full AutoRound optimization due to Gemma 4's architecture constraints.
• Some layers with shapes not divisible by 32 are skipped during quantization (minor precision impact).
• Multimodal (vision/audio) capabilities are fully preserved as those towers are not quantized.

Acknowledgements

Big thanks to OLAF-OSS and the contributors of gemma4-vllm — a fantastic resource for running Gemma 4 with vLLM. Their detailed QUANTIZE.md guide was instrumental in figuring out the correct quantization setup for gemma-4-E2B-it, and this work is directly inspired by ciocan/gemma-4-E2B-it-W4A16.

If you're looking to quantize or serve Gemma 4 yourself, their repo is the best starting point. 🙏

The full quantization process used to produce these models is documented here: 📓 auto_round_Gemma4-E2B.ipynb

License

This quantized model is derived from google/gemma-4-E2B-it and is subject to the Gemma Terms of Use.

Citation

If you use this quantized model, please also cite the original Gemma 4 work:

@misc{gemma4_2026,
  title  = {Gemma 4},
  author = {Google DeepMind},
  year   = {2026},
  url    = {https://huggingface.co/google/gemma-4-E2B-it}
}

Quantized by Vishva007