Open benchmark v0.2.0
How good are AI models at real Salesforce work?
Not trivia. Apex, Flow and LWC that have to pass hidden tests, and SOQL that has to return the right rows.
Forcebench gives models real Salesforce engineering tasks, runs what they build (a check-only deploy with hidden tests in a clean scratch org, or Jest) and reports how often it works, with honest error bars. Open source and versioned, with every final answer on the public tasks published.
$forcebench leaderboard --api --top 3
$forcebench leaderboard --local --top 3
Leaderboard
Top models, API and Local
The score within three attempts (c@3), each retry shown what the deploy and tests reported, each model at its highest-scoring configuration (as the leaderboard opens), on two boards ranked separately: API, models reached through a hosted service, and Local, models served on Forcebench’s own hardware. Where two bars overlap, those models are not clearly different; where they don't, one is clearly above the other. A configuration still waiting for its retries isn’t ranked yet; it’s added once they finish.
API
| # | Model and best config | Within 3 attempts c@3 · 95% CI |
|---|---|---|
| 1 | Claude Opus 5.5effort low | 96%95% CI 93–98 |
| 2 | GPT-6 Astraeffort medium | 95%95% CI 92–97 |
| 3 | GPT-6.1 Soleffort high | 93%95% CI 90–95 |
| 3 | Claude Sonnet 5.5effort high | 93%95% CI 90–96 |
| 5 | Grok 4.7effort high | 92%95% CI 89–95 |
Local
| # | Model and best config | Within 3 attempts c@3 · 95% CI |
|---|---|---|
| 1 | DeepSeek V4.1 FlashEXL3 2.9bpw · effort high | 73%95% CI 68–78 |
| 2 | GLM-5.3 FlashEXL3 4.0bpw experts, FP8 dense · effort high | 64%95% CI 59–68 |
| 3 | Qwen3.8 Flash-NextNVFP4 · effort medium | 58%95% CI 53–62 |
| 4 | DeepSeek V4 Flash Vision experimentalFP8 + FP4 experts · effort high | 52%95% CI 47–57 |
| 5 | MiMo V2.6 FlashFP8 + MXFP4 experts · thinking on | 51%95% CI 46–56 |
How the best local configuration compares with the API ones, score, tokens and time: the local vs API study.
Findings
What the numbers say
Worked out from the latest results every time the site builds, not written by hand. Each comes from one board and compares configurations on it only, except local vs API. Each names its metric, the same as the chart or table its link opens: c@3 for rankings, pass@1 for comparisons paired task by task on first answers.
Claude Opus 5.5 leads at 96% within three attempts
But 4 other API models, each at its best configuration, overlap its 95% interval of 93–98%, so it is not clearly different from them. Nemotron 3 Super isn’t ranked yet: its retries haven’t finished.
DeepSeek V4.1 Flash leads at 73% within three attempts
But 1 other local model, at its best configuration, overlaps its 95% interval of 68–78%, so it is not clearly different from that one.
Best local: DeepSeek V4.1 Flash at 73% within three attempts
10 of 26 API models, each at its best configuration, are clearly above it. The best API model, Claude Opus 5.5 (effort low, reasoning on), scores 96%: clearly above it.
EXL3 4.0bpw experts, 4-bit dense vs EXL3 4.0bpw experts, FP8 dense: −6 pts pass@1
GLM-5.3 Flash at EXL3 4.0bpw experts, 4-bit dense, against its EXL3 4.0bpw experts, FP8 dense reference run, pass@1 on the same tasks: clearly lower (−6 pts, 95% CI −12 pts to −0.7 pts). The engine changed too (vLLM + ExLlamaV3 to TensorFold), so not all of it can be put down to quantisation.
More thinking: clearly higher for Claude Sonnet 5
Going from effort low to high took Claude Sonnet 5 from 55% to 65% pass@1, compared on the same tasks: clearly higher (+10 pts, 95% CI +5 pts to +14 pts). Mean output tokens went from 730 to 2.4k.
Salesforce docs is the hardest suite
Of the API models, each at its highest-scoring configuration, the best there is Claude Opus 5.5 (effort low), at only 82% within three attempts.
Sovereign AI
Models you own, tested on your work
Open-weight models are models a team can own, so the data stays on infrastructure you control. If you are choosing one, Azul Labs can benchmark your shortlist on your kind of Salesforce work for a fixed fee and report back.
How it works
Graded the way a deploy would grade it
- 1
Real tasks
Fifteen suites of the work Salesforce developers actually do, including Apex, Lightning Web Components, Permissions & access, Packaging and CI/CD. Not multiple-choice trivia.
- 2
Run, not judged
Answers are deployed as a check-only validation to a clean scratch org with hidden tests, run under Jest, or checked by deterministic validators. No model grades another model's work.
- 3
Honest numbers
Ranked within three attempts (c@3): after a failed answer the model sees what the deploy, tests or tools reported and tries again, the way a developer works, with 95% confidence intervals and the exact quantisation, engine and effort of every configuration. Final answers, grades and token counts for public tasks are published in the repo; reasoning traces are available on request through the gated Hugging Face dataset; nothing from the private pool is published except approved aggregate scores. Read the methodology.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au