Models / gemma-4-26b-a4b
Gemma 4 26B-A4B
Google's model, with open weights. Forcebench has run it in 1 configuration on the Local board, plus 1 on the lite subset. API is models reached through a hosted service; Local, models served on Forcebench’s own hardware.
On the Local board
Models served on Forcebench’s own hardware. Gemma 4 26B-A4B is ranked by its one configuration: the seventh-highest score of 9 models, not clearly different from 3 other models. Scores are within 3 attempts (c@3) with 95% intervals; each configuration is compared with the highest-scoring one by whether their 95% intervals overlap. Every configuration on the board →
| Configuration | Rank all configs | Within 3 attempts c@3 · 95% CI | Tokens mean · median · p90 | No answer | Latency median per task, every attempt counted |
|---|---|---|---|---|---|
| NVFP4 · vLLM · thinking on · 1× NVIDIA GeForce RTX 5090 (32 GB)highest score | 14 | 40%95% CI 35–44 | 5.8k5.6k · 10k | 0% | 53 s |
Latency depends on the hardware: compare it only between configurations on the same hardware.
- 95% interval
- Local board median
Show the numbers
| Suite | c@3 95% CI | Local median | Tasks |
|---|---|---|---|
| Apex | 70%95% CI 48–85 | 70% | 20 |
| Salesforce APIs | 39%95% CI 20–61 | 44% | 18 |
| CI/CD | 35%95% CI 17–59 | 53% | 17 |
| Salesforce CLI | 15%95% CI 5–36 | 45% | 20 |
| Salesforce docs | 0%95% CI 0–18 | 12% | 17 |
| fflib Enterprise Patterns | 6%95% CI 1–26 | 17% | 18 |
| Flow | 17%95% CI 6–39 | 17% | 18 |
| Governor limits & pushback | 89%95% CI 69–97 | 84% | 19 |
| Lightning Web Components | 83%95% CI 61–94 | 89% | 18 |
| Nonprofit Success Pack | 61%95% CI 39–80 | 61% | 18 |
| Packaging | 11%95% CI 3–33 | 39% | 18 |
| Permissions & access | 40%95% CI 20–64 | 60% | 15 |
| Scratch org definitions | 11%95% CI 3–33 | 17% | 18 |
| SOQL | 90%95% CI 70–97 | 90% | 20 |
| Trigger Actions Framework | 28%95% CI 13–51 | 44% | 18 |
In the studies
- Local vs API: where it lands among the configurations of both boards.
- Pushback: whether it pushed back on requests that break at scale.
- Coding agent: the same tasks answered by a coding agent, with and without skills.
- Harness: one task in three coding agent harnesses.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au