1.58bit DeepSeek R1 - 131GB Dynamic GGUF

Hey r/LocalLLaMA! I managed to dynamically quantize the full DeepSeek R1 671B MoE to 1.58bits in GGUF format. The trick is not to quantize all layers, but quantize only the MoE layers to 1.5bit, and leave attention and other layers in 4 or 6bit.

You can get 140 tokens / s for throughput and 14 tokens /s for single user inference on 2x H100 80GB GPUs with all layers offloaded. A 24GB GPU like RTX 4090 should be able to get at least 1 to 3 tokens / s.

If we naively quantize all layers to 1.5bit (-1, 0, 1), the model will fail dramatically, since it'll produce gibberish and infinite repetitions. I selectively leave all attention layers in 4/6bit, and leave the first 3 transformer dense layers in 4/6bit. The MoE layers take up 88% of all space, so we can leave them in 1.5bit. We get in total a weighted sum of 1.58bits!

I asked it the 1.58bit model to create Flappy Bird with 10 conditions (like random colors, a best score etc), and it did pretty well! Using a generic non dynamically quantized model will fail miserably - there will be no output at all!

Flappy Bird game made by 1.58bit R1

There's more details in the blog here: https://unsloth.ai/blog/deepseekr1-dynamic The link to the 1.58bit GGUF is here: https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-IQ1_S You should be able to run it in your favorite inference tool if it supports i matrix quants. No need to re-update llama.cpp.

A reminder on DeepSeek's chat template (for distilled versions as well) - it auto adds a BOS - do not add it manually!

<｜begin▁of▁sentence｜><｜User｜>What is 1+1?<｜Assistant｜>It's 2.<｜end▁of▁sentence｜><｜User｜>Explain more!<｜Assistant｜>

To know how many layers to offload to the GPU, I approximately calculated it as below:

All other GGUFs for R1 are here: https://huggingface.co/unsloth/DeepSeek-R1-GGUF There's also GGUFs and dynamic 4bit bitsandbytes quants and others for all other distilled versions (Qwen, Llama etc) at https://huggingface.co/collections/unsloth/deepseek-r1-all-versions-678e1c48f5d2fce87892ace5