Models / qwen3-8-flash-next
Qwen3.8 Flash-Next
Qwen's model, with open weights. Forcebench has run it in 5 configurations on the API and Local boards. API is models reached through a hosted service; Local, models served on Forcebench’s own hardware.
On the API board
Models reached through a hosted service. Qwen3.8 Flash-Next is ranked by its one configuration: the 23rd-highest score of 26 models, not clearly different from 3 other models. Scores are within 3 attempts (c@3) with 95% intervals; each configuration is compared with the highest-scoring one by whether their 95% intervals overlap. Every configuration on the board → Cost and speed →
| Configuration | Rank all configs | Within 3 attempts c@3 · 95% CI | Tokens mean · median · p90 | No answer | Latency median per task, every attempt counted | Cost per task | Cost per solved task |
|---|---|---|---|---|---|---|---|
| OpenRouter · thinking onhighest score | 37 | 44%95% CI 39–49 | 21k22k · 33k | 34% | 610 s | $0.021 | $0.048 |
- 95% interval
- API board median
Show the numbers
| Suite | c@3 95% CI | API median | Tasks |
|---|---|---|---|
| Apex | 30%95% CI 15–52 | 90% | 20 |
| Salesforce APIs | 33%95% CI 16–56 | 72% | 18 |
| CI/CD | 59%95% CI 36–78 | 88% | 17 |
| Salesforce CLI | 40%95% CI 22–61 | 90% | 20 |
| Salesforce docs | 0%95% CI 0–18 | 29% | 17 |
| fflib Enterprise Patterns | 17%95% CI 6–39 | 61% | 18 |
| Flow | 11%95% CI 3–33 | 69% | 18 |
| Governor limits & pushback | 58%95% CI 36–77 | 89% | 19 |
| Lightning Web Components | 78%95% CI 55–91 | 100% | 18 |
| Nonprofit Success Pack | 44%95% CI 25–66 | 81% | 18 |
| Packaging | 44%95% CI 25–66 | 69% | 18 |
| Permissions & access | 60%95% CI 36–80 | 87% | 15 |
| Scratch org definitions | 11%95% CI 3–33 | 50% | 18 |
| SOQL | 95%95% CI 76–99 | 100% | 20 |
| Trigger Actions Framework | 83%95% CI 61–94 | 86% | 18 |
On the Local board
Models served on Forcebench’s own hardware. Qwen3.8 Flash-Next is ranked by the highest-scoring of its 4 configurations: the third-highest score of 9 models, not clearly different from 3 other models. Scores are within 3 attempts (c@3) with 95% intervals; each configuration is compared with the highest-scoring one by whether their 95% intervals overlap. Every configuration on the board →
- 95% interval
Show the numbers
| Configuration | Rank all configs | Within 3 attempts c@3 · 95% CI | Against the highest by 95% interval overlap | Tokens mean · median · p90 | No answer | Latency median per task, every attempt counted |
|---|---|---|---|---|---|---|
| NVFP4 · vLLM · medium · 2× NVIDIA DGX Spark (GB10, 128 GB each)highest score | 5 | 58%95% CI 53–62 | — | 5.1k2.9k · 11k | 2% | 103 s |
| MLX 4-bit · TensorFold · medium · 1× Gigabyte AI TOP ATOM (NVIDIA GB10, 128 GB) | 8 | 51%95% CI 46–56 | not clearly different | 12k5.7k · 33k | 15% | 331 s |
| MLX 4-bit · MTPLX · medium · Apple M5 Max (40-core GPU, 128 GB) | 10 | 50%95% CI 45–55 | not clearly different | 9.8k4.4k · 33k | 12% | 72 s |
| INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · medium · 2× NVIDIA DGX Spark (GB10, 128 GB each) | 12 | 49%95% CI 43–54 | not clearly different | 13k5.9k · 33k | 26% | 291 s |
Latency depends on the hardware: compare it only between configurations on the same hardware.
- 95% interval
- Local board median
Show the numbers
| Suite | c@3 95% CI | Local median | Tasks |
|---|---|---|---|
| Apex | 80%95% CI 58–92 | 70% | 20 |
| Salesforce APIs | 44%95% CI 25–66 | 44% | 18 |
| CI/CD | 71%95% CI 47–87 | 53% | 17 |
| Salesforce CLI | 55%95% CI 34–74 | 45% | 20 |
| Salesforce docs | 24%95% CI 10–47 | 12% | 17 |
| fflib Enterprise Patterns | 17%95% CI 6–39 | 17% | 18 |
| Flow | 28%95% CI 13–51 | 17% | 18 |
| Governor limits & pushback | 68%95% CI 46–85 | 84% | 19 |
| Lightning Web Components | 100%95% CI 82–100 | 89% | 18 |
| Nonprofit Success Pack | 61%95% CI 39–80 | 61% | 18 |
| Packaging | 44%95% CI 25–66 | 39% | 18 |
| Permissions & access | 60%95% CI 36–80 | 60% | 15 |
| Scratch org definitions | 17%95% CI 6–39 | 17% | 18 |
| SOQL | 100%95% CI 84–100 | 90% | 20 |
| Trigger Actions Framework | 94%95% CI 74–99 | 44% | 18 |
By quantisation
The same effort in different quantisations, each against the highest-precision one on the same tasks (a paired 95% interval), on pass@1, the first answer alone. Where the engine changes too, a difference can't be put down to quantisation alone. The quantisation study has every model.
- 95% interval
Show the numbers
| Build | pass@1 | 95% CI | Tokens |
|---|---|---|---|
| MLX 4-bit · MTPLX · Apple M5 Max (40-core GPU, 128 GB) | 37% | 31–42 | 9.8k |
| MLX 4-bit · TensorFold · 1× Gigabyte AI TOP ATOM (NVIDIA GB10, 128 GB) | 34% | 29–39 | 12k |
| INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · 2× NVIDIA DGX Spark (GB10, 128 GB each) | 29% | 24–34 | 13k |
| NVFP4 · vLLM · 2× NVIDIA DGX Spark (GB10, 128 GB each) | 40% | 34–45 | 5.1k |
| Quantisation · engine · medium | pass@1 95% CI | Against MLX 4-bit same tasks, paired | Tokens |
|---|---|---|---|
| MLX 4-bit · MTPLX · Apple M5 Max (40-core GPU, 128 GB)reference | 37%95% CI 31–42 | — | 9.8k |
| MLX 4-bit · TensorFold · 1× Gigabyte AI TOP ATOM (NVIDIA GB10, 128 GB) | 34%95% CI 29–39 | no clear difference (−3 pts, 95% CI −8 pts to +3 pts); this data can't rule out a change of up to 8 points | 12k |
| INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · 2× NVIDIA DGX Spark (GB10, 128 GB each) | 29%95% CI 24–34 | clearly lower (−8 pts, 95% CI −13 pts to −3 pts) | 13k |
| NVFP4 · vLLM · 2× NVIDIA DGX Spark (GB10, 128 GB each) | 40%95% CI 34–45 | no clear difference (+3 pts, 95% CI −2 pts to +8 pts); this data can't rule out a change of up to 8 points | 5.1k |
In the studies
- Quantisation: its quantisations side by side.
- Local vs API: where it lands among the configurations of both boards.
- Pushback: whether it pushed back on requests that break at scale.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au