Study · Quantisation
Does running a model in fewer bits hurt its Salesforce work?
Quantisation stores a model's weights in fewer bits so it needs less memory and runs cheaper. It is how most teams run open models privately. The question is what it costs in quality. The formats run here are Native FP8, FP4 experts, EXL3 2.9bpw, EXL3 4.0bpw experts, FP8 dense, EXL3 4.0bpw experts, 4-bit dense, MLX 8-bit, MLX 4-bit, Splash 4-bit, AWQ-INT4, INT4-AutoRound, 5 of 10 experts and NVFP4. There are no BF16 runs.
Each chart below is one model at one effort setting, run at different quantisations. The prompts, tasks and sampling settings stay fixed, but each quantisation needs an engine that supports it, so quantisation and engine usually change together: a difference can't then be put down to quantisation alone. Each chart says which.
Every run is compared with a reference run of the same model: the one with the most bits per weight, by Forcebench’s reckoning of each format (where two have the same, the first by engine name). The reference is not full precision. Differences are paired: the two runs answered the same tasks, so the difference comes with its own 95% interval from resampling those tasks (see the methodology).
Show the numbers
| Quantisation | Engine | pass@1 | 95% CI | vs reference (paired) | 95% CI of the difference | No answer | Tokens: mean | median | Latency: median |
|---|---|---|---|---|---|---|---|---|---|
| Native FP8, FP4 experts | SGLang | 57% | 52–62 | reference | — | 8% | 9.6k | 6.1k | 253 s |
| EXL3 2.9bpw | vLLM + ExLlamaV3 | 59% | 54–64 | +2 pts | −3 pts to +7 pts | 6% | 7.4k | 3.5k | 150 s |
Quantisation and engine change together in every comparison here (SGLang and vLLM + ExLlamaV3), so a difference can't be put down to quantisation alone.
EXL3 2.9bpw · vLLM + ExLlamaV3 against the reference: no clear difference (+2 pts, 95% CI −3 pts to +7 pts); this data can't rule out a change of up to 7 points.
Show the numbers
| Quantisation | Engine | pass@1 | 95% CI | vs reference (paired) | 95% CI of the difference | No answer | Tokens: mean | median | Latency: median |
|---|---|---|---|---|---|---|---|---|---|
| EXL3 4.0bpw experts, FP8 dense | vLLM + ExLlamaV3 | 43% | 38–48 | reference | — | 2% | 3.4k | 1.1k | 64 s |
| EXL3 4.0bpw experts, 4-bit dense | TensorFold | 37% | 32–42 | −6 pts | −12 pts to −0.7 pts | 2% | 4k | 1.2k | 45 s |
Quantisation and engine change together in every comparison here (vLLM + ExLlamaV3 and TensorFold), so a difference can't be put down to quantisation alone.
EXL3 4.0bpw experts, 4-bit dense · TensorFold against the reference: clearly lower (−6 pts, 95% CI −12 pts to −0.7 pts).
Show the numbers
| Quantisation | Engine | pass@1 | 95% CI | vs reference (paired) | 95% CI of the difference | No answer | Tokens: mean | median | Latency: median |
|---|---|---|---|---|---|---|---|---|---|
| MLX 8-bit | MTPLX | 28% | 24–33 | reference | — | 0% | 3.1k | 2k | 54 s |
| MLX 4-bit | MTPLX | 26% | 21–31 | −2 pts | −7 pts to +2 pts | 0% | 3.4k | 2.2k | 45 s |
| Splash 4-bit | Splash | 22% | 18–27 | −6 pts | −10 pts to −2 pts | 0% | 3.4k | 2.3k | 54 s |
| AWQ-INT4 | vLLM | 24% | 19–28 | −4 pts | −9 pts to −0.5 pts | 1% | 3.3k | 2.2k | 15 s |
Quantisation and engine change together in most comparisons here; MLX 4-bit shares the reference's engine (MTPLX).
MLX 4-bit · MTPLX against the reference: no clear difference (−2 pts, 95% CI −7 pts to +2 pts); this data can't rule out a change of up to 7 points.
Splash 4-bit · Splash against the reference: clearly lower (−6 pts, 95% CI −10 pts to −2 pts).
AWQ-INT4 · vLLM against the reference: clearly lower (−4 pts, 95% CI −9 pts to −0.5 pts).
All 3 scored below the reference (−2 pts, −6 pts and −4 pts). Splash 4-bit and AWQ-INT4 are clear on their own.
Show the numbers
| Quantisation | Engine | pass@1 | 95% CI | vs reference (paired) | 95% CI of the difference | No answer | Tokens: mean | median | Latency: median |
|---|---|---|---|---|---|---|---|---|---|
| MLX 4-bit | MTPLX | 37% | 31–42 | reference | — | 12% | 9.8k | 4.4k | 58 s |
| MLX 4-bit | TensorFold | 34% | 29–39 | −3 pts | −8 pts to +3 pts | 15% | 12k | 5.7k | 215 s |
| INT4-AutoRound, 5 of 10 experts | TensorFold (Zig) | 29% | 24–34 | −8 pts | −13 pts to −3 pts | 26% | 13k | 5.9k | 130 s |
| NVFP4 | vLLM | 40% | 34–45 | +3 pts | −2 pts to +8 pts | 2% | 5.1k | 2.9k | 75 s |
Quantisation and engine change together in every comparison here (MTPLX, TensorFold, TensorFold (Zig) and vLLM), so a difference can't be put down to quantisation alone.
MLX 4-bit · TensorFold against the reference: no clear difference (−3 pts, 95% CI −8 pts to +3 pts); this data can't rule out a change of up to 8 points.
INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) against the reference: clearly lower (−8 pts, 95% CI −13 pts to −3 pts).
NVFP4 · vLLM against the reference: no clear difference (+3 pts, 95% CI −2 pts to +8 pts); this data can't rule out a change of up to 8 points.
No-answer rates differ here (MLX 4-bit 12%, MLX 4-bit 15%, INT4-AutoRound, 5 of 10 experts 26% and NVFP4 2%): answers cut off at the 32,768-token output budget fail, so part of any gap may come from the engine or from truncation rather than from quantisation.
“Clearly higher” or “clearly lower” means the paired 95% interval of the difference excludes zero. Otherwise there is no clear difference, and the interval shows how large a change the data can't rule out. Latency depends on the engine, the hardware and how many requests ran at once, which differ between these runs, so it is shown but not compared. See how the intervals work.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au