TL;DR
The same model can produce different outputs and even fail at tool calls depending on the attention backend, quantization level, and other runtime settings—it's not the model being dumber, but the implementation.
Someone raves that a model is amazing, but locally it feels 'meh' — the issue may not be the model but how it's run. A researcher compared the same model under different settings and found striking differences.
1. Changing the attention backend changes outputs
When an LLM calculates the next token, it can use different low-level implementations, such as FlashAttention 2, Flash Inference, and Triton. In tests, just switching this option changed the model's 'preference' for token choices in a long context.
On the same GPU and prompt, switching backends changed the most likely token in about 40% of sampled positions (over long contexts). The divergence was not random but correlated with prompt content.
Worse, this divergence broke tool calls: instead of 'show arp', the model ran 'show run'. A simple configuration choice made the model 'mess up'.
2. More quantization, more divergence
Quantization compresses a model to save memory, but reduces precision. The test compared BF16 reference, FP8, INT8, NVFP4, and AWQ. The results: INT8 stayed closest to BF16, while NVFP4 had the highest 'token flip' rate at nearly 50%—almost every other token differed.
Heavy quantization also caused more severe issues: beyond ~40k tokens, the model's 'memory' faltered, leading to repetition. In tool calling, both NVFP4 and AWQ could execute wrong commands and fail tasks.
3. Quantization isn't all bad, but 'out-of-the-box' is risky
The author found not all quantizations are equally bad. Among 'abliterated' (uncensored) models, some carefully designed ones (like WildCard) preserved tool-calling ability while removing refusals. But NVFP4-type quants hurt both precision and tool use.
So, when choosing a model, don't just look at benchmark scores—test it on your own tasks. If you rely on tools or long context, prefer high-precision (e.g., INT8) over very low-bit quantizations.
In short, the model isn't dumb—your setup makes it act so. Tiny config changes can lead to wildly different results.
Curated from high-quality sources, with concise summaries and key takeaways.
