Suites / api
Salesforce APIs
Integration work against Salesforce's APIs: REST sObject, query, composite, composite graph and sObject Collections resources, Bulk API 2.0 ingest and query jobs, Tooling API, User Interface API, platform events, custom Apex REST endpoints and OAuth 2.0 token requests. Answers are raw HTTP requests, checked deterministically for method, path, query, headers and body (JSON, form-encoded or CSV), plus a few questions on API behaviour and choice.
How it's graded
Answers are raw HTTP requests, checked deterministically for method, path, query, headers and body (JSON, form-encoded or CSV). A few questions cover API behaviour and choosing the right API.
Difficulty mix
Leaderboard
Who's best at Salesforce APIs
Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 18 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →
| # | Model and config | Score c@3 · 95% CI | Tasks |
|---|---|---|---|
| 1 | GPT-6 Astra effort high | 100%95% CI 82–100 | 18 |
| 2 | Claude Opus 4.8 effort medium | 94%95% CI 74–99 | 18 |
| 2 | Claude Opus 4.8 effort low | 94%95% CI 74–99 | 18 |
| 2 | Claude Opus 5.5 effort medium | 94%95% CI 74–99 | 18 |
| 5 | Claude Haiku 5.5 effort medium | 89%95% CI 67–97 | 18 |
| 5 | Claude Haiku 5.5 effort high | 89%95% CI 67–97 | 18 |
| 5 | Claude Opus 4.8 thinking off | 89%95% CI 67–97 | 18 |
| 5 | Claude Sonnet 5 effort high | 89%95% CI 67–97 | 18 |
| 5 | Claude Sonnet 5.5 effort high | 89%95% CI 67–97 | 18 |
| 5 | Claude Sonnet 5.5 effort low | 89%95% CI 67–97 | 18 |
| 5 | Claude Sonnet 5.5 effort medium | 89%95% CI 67–97 | 18 |
| 5 | DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high | 89%95% CI 67–97 | 18 |
| 5 | Gemini 3.8 Flash effort medium | 89%95% CI 67–97 | 18 |
| 5 | GPT-6 Astra effort medium | 89%95% CI 67–97 | 18 |
| 5 | GPT-6.1 Sol effort high | 89%95% CI 67–97 | 18 |
| 5 | Grok 4.7 effort high | 89%95% CI 67–97 | 18 |
| 17 | Claude Opus 4.8 effort high | 83%95% CI 61–94 | 18 |
| 17 | Claude Opus 5.5 effort high | 83%95% CI 61–94 | 18 |
| 17 | Claude Opus 5.5 effort low | 83%95% CI 61–94 | 18 |
| 17 | Claude Sonnet 5 effort medium | 83%95% CI 61–94 | 18 |
| 17 | DeepSeek V4.1 Flash FP8 · effort high | 83%95% CI 61–94 | 18 |
| 22 | Claude Haiku 5.5 effort low | 78%95% CI 55–91 | 18 |
| 22 | Claude Sonnet 5 thinking off | 78%95% CI 55–91 | 18 |
| 22 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 78%95% CI 55–91 | 18 |
| 22 | GPT-5.6 Sol effort medium | 78%95% CI 55–91 | 18 |
| 22 | Grok 4.6 effort high | 78%95% CI 55–91 | 18 |
| 27 | Claude Sonnet 5 effort low | 72%95% CI 49–88 | 18 |
| 27 | GLM-5.3 Flash FP8 · effort high | 72%95% CI 49–88 | 18 |
| 27 | GPT-6 Luna effort medium | 72%95% CI 49–88 | 18 |
| 27 | MiMo V2.6 Pro FP8 · thinking on | 72%95% CI 49–88 | 18 |
| 31 | DeepSeek V4 Pro FP8 · effort high | 67%95% CI 44–84 | 18 |
| 32 | Nemotron 3 Ultra thinking on | 61%95% CI 39–80 | 18 |
| 33 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high | 56%95% CI 34–75 | 18 |
| 33 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off | 56%95% CI 34–75 | 18 |
| 33 | Inkling Small FP8 · effort high | 56%95% CI 34–75 | 18 |
| 33 | Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium | 56%95% CI 34–75 | 18 |
| 37 | DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high | 50%95% CI 29–71 | 18 |
| 37 | Gemma 4 31B QAT W4A16 · vLLM · thinking on | 50%95% CI 29–71 | 18 |
| 39 | Claude Haiku 4.5 thinking on | 44%95% CI 25–66 | 18 |
| 39 | Claude Haiku 4.5 thinking off | 44%95% CI 25–66 | 18 |
| 39 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low | 44%95% CI 25–66 | 18 |
| 39 | MiMo V2.6 Flash FP8 · thinking on | 44%95% CI 25–66 | 18 |
| 39 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 44%95% CI 25–66 | 18 |
| 39 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 44%95% CI 25–66 | 18 |
| 39 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 44%95% CI 25–66 | 18 |
| 39 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 44%95% CI 25–66 | 18 |
| 47 | Claude Fable 5.1 effort high | 39%95% CI 20–61 | 18 |
| 47 | Gemma 4 26B-A4B NVFP4 · vLLM · thinking on | 39%95% CI 20–61 | 18 |
| 47 | MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on | 39%95% CI 20–61 | 18 |
| 47 | Mistral Large 4 effort high | 39%95% CI 20–61 | 18 |
| 47 | Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on | 39%95% CI 20–61 | 18 |
| 47 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 39%95% CI 20–61 | 18 |
| 47 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 39%95% CI 20–61 | 18 |
| 47 | Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium | 39%95% CI 20–61 | 18 |
| 55 | GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high | 33%95% CI 16–56 | 18 |
| 55 | Laguna S 2.1 FP4 · thinking on | 33%95% CI 16–56 | 18 |
| 55 | Ling 3.0 Flash BF16 · thinking on | 33%95% CI 16–56 | 18 |
| 55 | Nemotron 3.5 Lightning NVFP4 · thinking on | 33%95% CI 16–56 | 18 |
| 55 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 33%95% CI 16–56 | 18 |
| 55 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 33%95% CI 16–56 | 18 |
| 55 | Qwen3.8 Flash-Next thinking on | 33%95% CI 16–56 | 18 |
Tasks
What's in the suite
Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.
| Task | Difficulty | From results | Solve rate |
|---|---|---|---|
| Close an opportunity and delete a duplicate contactapi-sobject-update-delete | easy | easy | 95% |
| Create a Case with the REST APIapi-sobject-create-case | easy | easy | 97% |
| Pick the API for a nightly 5-million-row loadapi-choose-bulk-for-large-load | easy | easy | 100% |
| Pre-sync checks with describe and limitsapi-describe-and-limits | easy | easy | 90% |
| Publish a platform event from an external systemapi-platform-event-publish | easy | easy | 89% |
| Query Apex code coverage with the Tooling APIapi-tooling-coverage-query | easy | medium | 61% |
| Call a managed-package Apex REST endpointapi-apexrest-namespaced-post | medium | easy | 69% |
| Create and update records with sObject Collectionsapi-collections-create-update | medium | medium | 55% |
| Fetch a record with UI API fields and optionalFieldsapi-uiapi-record-optional-fields | medium | medium | 48% |
| Page through a delta query that includes deleted recordsapi-query-paging-queryall | medium | medium | 48% |
| Request a token with the OAuth 2.0 client credentials flowapi-oauth-client-credentials | medium | easy | 90% |
| Run anonymous Apex through the Tooling APIapi-tooling-execute-anonymous | medium | medium | 35% |
| Upsert an account by external ID with its parentapi-upsert-external-id | medium | easy | 67% |
| Bulk API 2.0 export including deleted recordsapi-bulk2-query-export | hard | medium | 55% |
| Bulk API 2.0 upsert with a parent external IDapi-bulk2-ingest-upsert | hard | medium | 45% |
| Composite request with an implicit dependencyapi-composite-implicit-dependency | hard | hard | 8% |
| Independent all-or-nothing orders in one callapi-composite-graph-orders | hard | medium | 42% |
| Interpret external-ID upsert responsesapi-upsert-response-semantics | hard | medium | 34% |
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au