Models / qwen3-8-27b
Qwen3.8 27B
Qwen's model, with open weights. Forcebench has run it in 6 configurations on the Local board. API is models reached through a hosted service; Local, models served on Forcebench’s own hardware.
On the Local board
Models served on Forcebench’s own hardware. Qwen3.8 27B is ranked by the highest-scoring of its 6 configurations: the eighth-highest score of 9 models, not clearly different from 3 other models. Scores are within 3 attempts (c@3) with 95% intervals; each configuration is compared with the highest-scoring one by whether their 95% intervals overlap. Every configuration on the board →
- 95% interval
Show the numbers
| Configuration | Rank all configs | Within 3 attempts c@3 · 95% CI | Against the highest by 95% interval overlap | Tokens mean · median · p90 | No answer | Latency median per task, every attempt counted |
|---|---|---|---|---|---|---|
| MLX 8-bit · MTPLX · low · Apple M5 Max (40-core GPU, 128 GB)highest score | 15 | 37%95% CI 33–42 | — | 3.1k2k · 5.4k | 0% | 73 s |
| AWQ-INT4 · vLLM · low · 1× NVIDIA GeForce RTX 5090 (32 GB) | 15 | 37%95% CI 32–42 | not clearly different | 3.3k2.2k · 8.4k | 1% | 28 s |
| Splash 4-bit · Splash · low · Apple M5 Max (40-core GPU, 128 GB) | 17 | 36%95% CI 31–41 | not clearly different | 3.4k2.3k · 7.1k | 0% | 99 s |
| MLX 4-bit · MTPLX · low · Apple M5 Max (40-core GPU, 128 GB) | 18 | 35%95% CI 30–41 | not clearly different | 3.4k2.2k · 6.4k | 0% | 59 s |
| AWQ-INT4 · vLLM · xhigh · 1× NVIDIA GeForce RTX 5090 (32 GB) | 18 | 35%95% CI 30–40 | not clearly different | 18k18k · 33k | 25% | 265 s |
| AWQ-INT4 · vLLM · medium · 1× NVIDIA GeForce RTX 5090 (32 GB) | 21 | 33%95% CI 29–38 | not clearly different | 4.7k2.8k · 9k | 2% | 31 s |
Latency depends on the hardware: compare it only between configurations on the same hardware.
- 95% interval
- Local board median
Show the numbers
| Suite | c@3 95% CI | Local median | Tasks |
|---|---|---|---|
| Apex | 70%95% CI 48–85 | 70% | 20 |
| Salesforce APIs | 44%95% CI 25–66 | 44% | 18 |
| CI/CD | 29%95% CI 13–53 | 53% | 17 |
| Salesforce CLI | 30%95% CI 15–52 | 45% | 20 |
| Salesforce docs | 0%95% CI 0–18 | 12% | 17 |
| fflib Enterprise Patterns | 0%95% CI 0–18 | 17% | 18 |
| Flow | 17%95% CI 6–39 | 17% | 18 |
| Governor limits & pushback | 63%95% CI 41–81 | 84% | 19 |
| Lightning Web Components | 39%95% CI 20–61 | 89% | 18 |
| Nonprofit Success Pack | 56%95% CI 34–75 | 61% | 18 |
| Packaging | 22%95% CI 9–45 | 39% | 18 |
| Permissions & access | 60%95% CI 36–80 | 60% | 15 |
| Scratch org definitions | 6%95% CI 1–26 | 17% | 18 |
| SOQL | 80%95% CI 58–92 | 90% | 20 |
| Trigger Actions Framework | 44%95% CI 25–66 | 44% | 18 |
By reasoning effort
The same configuration at several efforts, each against its lowest reasoning effort on the same tasks (a paired 95% interval), and charted against the effort tier and against the output tokens it costs. These scores are pass@1, the first answer alone: a paired comparison needs both efforts' answers to the same tasks, and not every effort has had its retries. Lite-subset sweeps (4 tasks per suite) are never plotted with the full benchmark. The effort study has every model.
| Configuration · effort · full benchmark | pass@1 95% CI | Against the lowest reasoning effort same tasks, paired | Tokens |
|---|---|---|---|
| AWQ-INT4 · vLLM · lowbaseline | 24%95% CI 19–28 | — | 3.3k |
| AWQ-INT4 · vLLM · medium | 24%95% CI 19–28 | no clear difference (±0 pts, 95% CI −3 pts to +3 pts); this data can't rule out a change of up to 3 points | 4.7k |
| AWQ-INT4 · vLLM · xhigh | 21%95% CI 17–26 | no clear difference (−3 pts, 95% CI −6 pts to +2 pts); this data can't rule out a change of up to 6 points | 18k |
Show the numbers
| Model | Effort (tier) | pass@1 | 95% CI | Tasks | No answer | Tokens: mean | median | Reasoning tokens | Latency: median per answer |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8 27B · AWQ-INT4 | low | 24% | 19–28 | 272 | 1% | 3.3k | 2.2k | 2.6k | 15 s |
| Qwen3.8 27B · AWQ-INT4 | medium | 24% | 19–28 | 272 | 2% | 4.7k | 2.8k | 3.4k | 20 s |
| Qwen3.8 27B · AWQ-INT4 | xhigh (max) · vendor default | 21% | 17–26 | 272 | 25% | 18k | 18k | 10k | 141 s |
By quantisation
The same effort in different quantisations, each against the highest-precision one on the same tasks (a paired 95% interval), on pass@1, the first answer alone. Where the engine changes too, a difference can't be put down to quantisation alone. The quantisation study has every model.
- 95% interval
Show the numbers
| Build | pass@1 | 95% CI | Tokens |
|---|---|---|---|
| MLX 8-bit · MTPLX · Apple M5 Max (40-core GPU, 128 GB) | 28% | 24–33 | 3.1k |
| MLX 4-bit · MTPLX · Apple M5 Max (40-core GPU, 128 GB) | 26% | 21–31 | 3.4k |
| Splash 4-bit · Splash · Apple M5 Max (40-core GPU, 128 GB) | 22% | 18–27 | 3.4k |
| AWQ-INT4 · vLLM · 1× NVIDIA GeForce RTX 5090 (32 GB) | 24% | 19–28 | 3.3k |
| Quantisation · engine · low | pass@1 95% CI | Against MLX 8-bit same tasks, paired | Tokens |
|---|---|---|---|
| MLX 8-bit · MTPLX · Apple M5 Max (40-core GPU, 128 GB)reference | 28%95% CI 24–33 | — | 3.1k |
| MLX 4-bit · MTPLX · Apple M5 Max (40-core GPU, 128 GB) | 26%95% CI 21–31 | no clear difference (−2 pts, 95% CI −7 pts to +2 pts); this data can't rule out a change of up to 7 points | 3.4k |
| Splash 4-bit · Splash · Apple M5 Max (40-core GPU, 128 GB) | 22%95% CI 18–27 | clearly lower (−6 pts, 95% CI −10 pts to −2 pts) | 3.4k |
| AWQ-INT4 · vLLM · 1× NVIDIA GeForce RTX 5090 (32 GB) | 24%95% CI 19–28 | clearly lower (−4 pts, 95% CI −9 pts to −0.5 pts) | 3.3k |
In the studies
- Reasoning effort: its score against effort and against output tokens.
- Max effort: what its max effort did against the effort below it.
- Quantisation: its quantisations side by side.
- Local vs API: where it lands among the configurations of both boards.
- Pushback: whether it pushed back on requests that break at scale.
- Coding agent: the same tasks answered by a coding agent, with and without skills.
- Harness: one task in three coding agent harnesses.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au