Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses - Quesma BlogSkip to main contentHow it worksBenchmarksBlogAbout usToken economicsTalk to the founder

How it worksBenchmarksBlogAbout usToken economicsTalk to the founderBenchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapsesPiotr Migdał·26 August 2026

Share / QR codeOpen on another device.

Download PNGHow much GPU RAM do you actually need to run Qwen3.8 27B without sacrificing quality?
The full BF16 model weighs 55 GB, putting it beyond most consumer hardware.
Yet the 17 GB Q4_K_M matches the full model on a popular agentic coding benchmark, Terminal-Bench 2.1. It fits on a 24 GB card such as RTX 4090, still leaving room for about 64k tokens of context.
Compression eventually hits a cliff. At 1 bit, the model performs around random chance on GPQA Diamond, and longer reasoning makes it worse.
Background
Qwen3.8 27B GGUF quantizations available from Unsloth on Hugging
Face. So much to choose from! I will check 8-bit Q8_0 (29 GB), 4-bit Q4_K_M (17 GB), 2-bit UD-Q2_K_XL (10.7 GB), and the smallest one possible, 1-bit UD-IQ1_S (6.2 GB).
Previously, I investigated the Qwen3.6 27B model, which was good at generating SVG pelicans even at 12GB, and maintained most of its knowledge up to 16GB.
At the same time, in Reddit threads, many complain that all quantizations, even the 8-bit ones, give worse results - with people asking why your local LLM feels dumber than it is. Are these complaints grounded?
Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks. Some noise might be irrelevant for solving tasks, as (say) a quantized model generates an answer of precisely the same quality, paraphrased a bit. In other cases, a single different token might be a logical error, or even abruptly end the output.
Download as PNG

70%80%90%100%610203050 GBmodel size on disksame top-1 token as BF16UD-IQ1_SUD-IQ1_MUD-Q2_K_XLQ4_K_MQ8_0BF16
So, I focus on directly measuring results on popular benchmarks - GPQA Diamond, instruction-following IFBench, programming Terminal-Bench 2.1.
First, to replicate official results of the full model BF16, and then to see how quantization affects results.
I burned around $3,000 on Modal GPUs when I ran models with llama.cpp using a build from 16 August 2026 as earlier builds do not work for this model. I could have run it on my own laptop, in principle, but (unlike pelican-generation), these are time-consuming benchmarks.
Note that I use F16 KV-cache regardless of model quantization, weighing around 2.3 GB per 32k tokens.
I used Unsloth quantizations: v2 for the 2-, 4-, and 8-bit models, and v3 for the 1-bit models. Unsloth replaced the v2 files on 19 August 2026, so the exact files used for most tests are no longer available.
In short, if you go with a 4-bit quantization Q4_K_M (17GB), you won’t notice a difference on these benchmarks.
At the same time, the effort setting matters a lot (note that the default is xhigh) - and it is a tricky choice, as it can overthink.
One-shot tests
The easiest ones are one-shot tests: in this case, graduate-level science GPQA Diamond and instruction-following IFBench.
I run each at three reasoning efforts: low, medium, and the default xhigh.
GPQA Diamond
Download as PNG

70%75%80%85%90%95%100%10203050 GBmodel size on diskGPQA Diamond scorexhigh (default)reported by QwenlowmediumUD-Q2_K_XLQ4_K_MQ8_0BF16
First and foremost, I was happy I replicated the official results. Running benchmarks is hard; there are many hidden settings or assumptions that can change the results drastically. Here, on the first go, results were as reported by Qwen.
Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower.
At the same time, thinking level changed the score dractically. The best results, for xhigh, needed around 8k reasoning tokens.
IFBench
Download as PNG

60%80%10203050 GBmodel size on diskIFBench followed (strict)xhigh (default)reported by QwenmediumlowUD-Q2_K_XLQ4_K_MQ8_0BF16
Here, to my great surprise, there is no change between models, down to a decent 2-bit one, weighing less than 11 GB.
Yet, context is even lower, around 4k tokens.
Agentic coding and Terminal-Bench 2.1
How does it work for programming? Terminal-Bench 2.1 is a standard agentic benchmark, with 89 tasks.
Here I use 3h timeout, xhigh effort. I reserve 98k context.
Download as PNG

reported by Qwen55%60%65%70%75%80%1020304050 GBmodel size on disk (log scale)Terminal-Bench 2.1 passedUD-Q2_K_XLQ4_K_MBF16
Not only does my measurement of BF16 replicate the stated result, but, to my surprise, Q4_K_M does as well.
I accidentally skipped running Q8_0; yet, in this case, I can safely interpolate between 4-bit and the full model’s values. Running it would be both costly and unnecessary (and would exceed an informal blog post’s budget).
Only at 2-bit UD-Q2_K_XL things break a bit. A noticeable fall, but still the level of Opus 4.7 or Gemini 3.1 Pro. Again, far from frontier, but also - far from useless.
Results are one thing, but what about the process? Do smaller models need more turns, tokens or time to get the result?
Download as PNG

same as BF160.8x1.0x1.2x1.4x1.6x1.8xoutput tokens vs BF16, same solved tasksUD-Q2_K_XLTerminal-Bench 2.1turnsGPQA DiamondIFBenchQ4_K_MTerminal-Bench 2.1turnsGPQA DiamondIFBenchQ8_0GPQA DiamondIFBench
On the same solved tasks, UD-Q2_K_XL takes as many turns as BF16 but writes about a quarter more tokens.
The number of turns stay roughly the same.
The 1-bit cliff
Quality drops off a cliff at 1-bit. As with knowledge, quantization damage is nonlinear: first there is no measurable change, then a small decline, and finally a collapse.
While 2-bit quantizations work to some extent, even the best 1-bit model is useless for these benchmarks:
Download as PNG

0%20%40%60%80%100%lowmediumxhighreasoning effort (xhigh is the model default)GPQA Diamond scoreQ4_K_Mreported by QwenUD-Q2_K_XLUD-IQ1_Mrandom guessUD-IQ1_S
As you may see, the scores are around the random guessing level, with the smallest model being below that threshold.
And longer reasoning makes it worse: at xhigh, scores drop below low, as the model more often reasons until the token budget runs out and returns an empty answer.
Sure, Unsloth boasts that:

We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller.

But in this case, these remaining 28% matter a lot. And this matches another user’s experience, vide Qwen3.8 27b 1bit brain damage quant on r/LocalLLaMA.
Costs
Running these benchmarks isn’t cheap.
Running benchmarks via API is costly, as I know from my previous benchmarks. Running on rented GPU is much costlier.
Download as PNG

BF16$804task containers$759Q4_K_M$502UD-Q2_K_XL$243UD-IQ1_S$167UD-IQ1_M$143Q4_K_M$120UD-Q2_K_XL$80BF16$80Q8_0$73Terminal-Bench 2.1 $2,308GPQA + IFBench $663
I used Modal, as it is easy to run it from the CLI, including from agents. Other setups may have different pricing. Obviously, this calculation changes if you have your own devices.
It takes some testing to find the optimal way to run models. Usually, instead of using Multi-Token Prediction (MTP), which works well for a single stream, I use a few parallel streams. The key constraint is whether the GPU has enough memory for both the model and the required KV caches.
I used NVIDIA L40S (the same Ada Lovelace chip as the RTX 4090 but twice as much memory: 48 GB), H100 (80 GB) and H200 (141 GB). I would like to share costs to give you a ballpark estimate if you want to run benchmarks yourself.
Download as PNG

$3$5$1020305080 tok/sspeed per streamprice per 1M output tokensUD-Q2_K_XLQ4_K_MQ8_0BF16L40SH100H2008 parallel streams1 stream, MTP draftOpenRouter
For comparison, DeepSeek V4 Flash 0731, a 284B model, costs around $0.1/Mtok for output from the cheapest providers on OpenRouter. I am not sure how much of this difference comes from the efficiency of running models at scale, pricing strategy, or popularity.
Conclusion
If you run experiments locally, usually pick the best model that fits in your GPU memory together with the required context.
For most tasks Unsloth’s Q4_K_M should be good enough, without any noticeable difference; for some simpler tasks UD-Q2_K_XL should be more than fine.
Since people report that KV-caches are more susceptible to quantization, I may test it as well.
But in general, I believe that quantization should be embraced, rather than feared.
And what is your experience? Join the discussion on r/LocalLLaMA, Hacker News, or LinkedIn.More articles.Email addressSubscribe

RSSAll articles

PreviousGemini 3.7 Flash, Grok 4.6, GLM-5.3 and DeepSeek V4 Pro joined the frontierAugust 2026 on Baba Is Bench: Gemini 3.7 Flash, Grok 4.6 and DeepSeek V4 Pro 0813 each beat their predecessor while costing 3-20x less. For open-weight GLM-5.3 and Qwen3.8 progress is gradual.Piotr Migdał20 Aug 2026Read more

NextOpenAI Codex pricing: the $270 PR a $200/month sub covers dailyA buyer’s guide to OpenAI Codex for individuals, teams, and enterprises. Includes comparisons with Claude Code.Jacek Migdal28 Aug 2026Read more

RelatedDo Qwen3.6 27B quantizations break the pelican?We tested Qwen3.6 27B quantizations by Unsloth on Hugging Face, with pelicans on bikes, gears, Terminal-Bench 2.1, and AIME-120.Piotr Migdał27 Jul 2026Read more

Understand what your coding agents actually do.[email protected]

© 2026Quesma Inc.ExploreHomeBenchmarksToken economicsPrivacy PolicyCompanyBlogAbout usContactMedia KitConnectGitHubGitHub

LinkedInXx.com

The investigation involved benchmarking various quantization levels of the Qwen3.8 27B model to determine the impact of compression on performance across established benchmarks, specifically focusing on task-solving capabilities rather than mere token prediction accuracy. The full BF16 version of the model requires 55 GB of memory. The study explored different Unsloth quantizations, including the 8-bit Q8_0, the widely used 4-bit Q4_K_M, the 2-bit UD-Q2_K_XL, and the most extreme quantization, the 1-bit UD-IQ1_S.

The methodology centered on measuring performance on popular agentic coding benchmark Terminal-Bench 2.1, instruction-following IFBench, and knowledge retrieval GPQA Diamond. The authors sought to replicate official full model results while analyzing how quantization affected these outcomes. While initial token prediction metrics like KL-divergence are easy to measure, the research emphasizes that true performance is determined by a model's ability to solve complex tasks, as small differences in single tokens might be irrelevant to the final result quality.

For agentic coding tasks on Terminal-Bench 2.1, the 4-bit Q4_K_M quantization successfully matched the results reported for the full BF16 model, suggesting that this level of compression holds up effectively for programming benchmarks without noticeable degradation. Similarly, one-shot tests on GPQA Diamond and IFBench showed minimal performance differences between models when evaluated under default reasoning settings. However, optimizing the reasoning effort proved crucial; achieving optimal scores required adjusting the number of reasoning tokens based on model defaults, indicating that thinking level significantly impacted final assessment.

In contrast, while 2-bit quantizations like UD-Q2_K_XL maintained some functional ability, they resulted in a noticeable decline compared to the full model. The critical finding emerged when assessing the performance of the 1-bit quantization: quality collapses entirely at this level. For tasks demanding reasoning, models quantized to 1-bit exhibited performance near random guessing, and extending reasoning efforts further caused scores to deteriorate significantly, resulting in empty or poor answers rather than logical progression. Although some smaller 1-bit variants retained a percentage of accuracy, the overall functional utility for complex benchmarks was severely compromised by this non-linear degradation.

The study concluded that while high-quality quantization methods like Q4_K_M are often sufficient for most tasks, it is essential to recognize the nonlinearity of performance loss, particularly as context and reasoning depth increase. The research suggests that perceived performance differences in quantized models may be exacerbated by factors such as the sensitivity of KV-caches to quantization and the context management overhead during extended inference. Therefore, the authors advocate for embracing quantization rather than fearing it, emphasizing that optimal local model deployment involves selecting a model size that fits memory constraints while balancing accuracy and computational feasibility.