Suites / soql
SOQL
Text-to-SOQL on a realistic B2B sales and service org: filters, relationship queries in both directions, semi- and anti-joins, aggregates, date and fiscal functions, polymorphic relationships, multi-select picklists and null handling. Each answer is executed in a seeded scratch org (orgs/base) and its result set must equal the gold query's (execution accuracy).
How it's graded
Each query runs against a scratch org seeded with fixed data, and its result set must equal the gold query’s (execution accuracy, as in BIRD and Spider). A query that errors or returns different rows fails.
Difficulty mix
Leaderboard
Who's best at SOQL
Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 20 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →
| # | Model and config | Score c@3 · 95% CI | Tasks |
|---|---|---|---|
| 1 | Claude Haiku 5.5 effort high | 100%95% CI 84–100 | 20 |
| 1 | Claude Haiku 5.5 effort low | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 4.8 effort high | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 4.8 effort medium | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 4.8 effort low | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 4.8 thinking off | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 5.5 effort high | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 5.5 effort medium | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 5.5 effort low | 100%95% CI 84–100 | 20 |
| 1 | Claude Sonnet 5 effort high | 100%95% CI 84–100 | 20 |
| 1 | Claude Sonnet 5 effort medium | 100%95% CI 84–100 | 20 |
| 1 | Claude Sonnet 5 thinking off | 100%95% CI 84–100 | 20 |
| 1 | Claude Sonnet 5 effort low | 100%95% CI 84–100 | 20 |
| 1 | Claude Sonnet 5.5 effort high | 100%95% CI 84–100 | 20 |
| 1 | Claude Sonnet 5.5 effort low | 100%95% CI 84–100 | 20 |
| 1 | Claude Sonnet 5.5 effort medium | 100%95% CI 84–100 | 20 |
| 1 | DeepSeek V4 Pro FP8 · effort high | 100%95% CI 84–100 | 20 |
| 1 | Gemini 3.8 Flash effort medium | 100%95% CI 84–100 | 20 |
| 1 | GPT-5.6 Sol effort medium | 100%95% CI 84–100 | 20 |
| 1 | GPT-6 Astra effort medium | 100%95% CI 84–100 | 20 |
| 1 | GPT-6 Astra effort high | 100%95% CI 84–100 | 20 |
| 1 | GPT-6 Luna effort medium | 100%95% CI 84–100 | 20 |
| 1 | GPT-6.1 Sol effort high | 100%95% CI 84–100 | 20 |
| 1 | Grok 4.6 effort high | 100%95% CI 84–100 | 20 |
| 1 | Grok 4.7 effort high | 100%95% CI 84–100 | 20 |
| 1 | MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on | 100%95% CI 84–100 | 20 |
| 1 | MiMo V2.6 Flash FP8 · thinking on | 100%95% CI 84–100 | 20 |
| 1 | MiMo V2.6 Pro FP8 · thinking on | 100%95% CI 84–100 | 20 |
| 1 | Mistral Large 4 effort high | 100%95% CI 84–100 | 20 |
| 1 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 100%95% CI 84–100 | 20 |
| 31 | Claude Fable 5.1 effort high | 95%95% CI 76–99 | 20 |
| 31 | Claude Haiku 5.5 effort medium | 95%95% CI 76–99 | 20 |
| 31 | DeepSeek V4.1 Flash FP8 · effort high | 95%95% CI 76–99 | 20 |
| 31 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 95%95% CI 76–99 | 20 |
| 31 | DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high | 95%95% CI 76–99 | 20 |
| 31 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high | 95%95% CI 76–99 | 20 |
| 31 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off | 95%95% CI 76–99 | 20 |
| 31 | Inkling Small FP8 · effort high | 95%95% CI 76–99 | 20 |
| 31 | Nemotron 3 Ultra thinking on | 95%95% CI 76–99 | 20 |
| 31 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 95%95% CI 76–99 | 20 |
| 31 | Qwen3.8 Flash-Next thinking on | 95%95% CI 76–99 | 20 |
| 42 | DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high | 90%95% CI 70–97 | 20 |
| 42 | Gemma 4 26B-A4B NVFP4 · vLLM · thinking on | 90%95% CI 70–97 | 20 |
| 42 | GLM-5.3 Flash FP8 · effort high | 90%95% CI 70–97 | 20 |
| 42 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low | 90%95% CI 70–97 | 20 |
| 42 | Laguna S 2.1 FP4 · thinking on | 90%95% CI 70–97 | 20 |
| 42 | Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium | 90%95% CI 70–97 | 20 |
| 48 | GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high | 85%95% CI 64–95 | 20 |
| 48 | Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium | 85%95% CI 64–95 | 20 |
| 50 | Claude Haiku 4.5 thinking off | 80%95% CI 58–92 | 20 |
| 50 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 80%95% CI 58–92 | 20 |
| 52 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 78%95% CI 56–90 | 20 |
| 53 | Gemma 4 31B QAT W4A16 · vLLM · thinking on | 75%95% CI 53–89 | 20 |
| 53 | Ling 3.0 Flash BF16 · thinking on | 75%95% CI 53–89 | 20 |
| 55 | Claude Haiku 4.5 thinking on | 70%95% CI 48–85 | 20 |
| 55 | Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on | 70%95% CI 48–85 | 20 |
| 55 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 70%95% CI 48–85 | 20 |
| 58 | Nemotron 3.5 Lightning NVFP4 · thinking on | 65%95% CI 43–82 | 20 |
| 58 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 65%95% CI 43–82 | 20 |
| 58 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 65%95% CI 43–82 | 20 |
| 58 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 65%95% CI 43–82 | 20 |
Tasks
What's in the suite
Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.
| Task | Difficulty | From results | Solve rate |
|---|---|---|---|
| Accounts missing billing country or industrysoql-accounts-missing-country-or-industry | easy | easy | 97% |
| Count open German web leads for a dashboard tilesoql-open-german-web-leads-count | easy | easy | 74% |
| Energy-sector wins in calendar 2025soql-energy-wins-2025 | easy | easy | 94% |
| Financial-services accounts in Germany and the UKsoql-banking-insurance-de-uk | easy | easy | 100% |
| In-transit shipments with their account namesoql-in-transit-shipments | easy | easy | 91% |
| Managers at healthcare accountssoql-healthcare-manager-contacts | easy | easy | 97% |
| 2025 closed-won business by industrysoql-won-2025-by-industry | medium | easy | 69% |
| Accounts with open high-priority casessoql-accounts-open-high-cases | medium | easy | 98% |
| Customers with no wins in 2025soql-customers-without-2025-wins | medium | easy | 90% |
| Five largest customers by annual revenuesoql-top-customers-by-revenue | medium | easy | 100% |
| German customers with their open shipments nestedsoql-german-customers-open-shipments | medium | medium | 39% |
| Open tasks related to opportunitiessoql-open-tasks-on-opportunities | medium | easy | 97% |
| Products whose code starts with SVC_soql-service-product-codes | medium | easy | 73% |
| Responders to an event campaignsoql-summit-responders | medium | easy | 72% |
| Activity log with type-specific related-record fieldssoql-march-2025-activity-typeof | hard | easy | 68% |
| Bookings per fiscal quarter with a non-calendar fiscal yearsoql-fy2026-bookings-by-quarter | hard | easy | 68% |
| Contacts anywhere in an account hierarchysoql-globex-group-contacts | hard | easy | 74% |
| Hardware deals without services attachedsoql-hardware-without-services-opps | hard | easy | 99% |
| Open shipments needing special handling (multi-select picklist)soql-open-shipments-handling | hard | easy | 73% |
| Parent accounts of German subsidiariessoql-parents-of-german-accounts | hard | hard | 21% |
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au