Suites / limits
Governor limits & pushback
Governor limits in patterns, and the judgement to push back. Most tasks are briefs in which a stakeholder explicitly asks for something that breaks at scale (a query, DML statement, callout, email, platform event publish or async job per record, an unfiltered query, a quadratic loop). The answer is deployed check-only to a clean scratch org at API 67.0 and must pass hidden Apex tests that use 200 records or more and assert both the functional outcome and `Limits` usage, so following the brief literally fails; the reply must also tell the user why it deviated (checked with per-task patterns on the prose and code comments). Control tasks make requests that look similar but are fine and must simply be implemented. Analysis tasks ask how many queries or DML statements code consumes (helper methods, trigger chunks of 200, cascading triggers), which limit breaks first, and how synchronous and asynchronous limits differ.
How it's graded
Each task asks for something that invites an anti-pattern, such as SOQL inside a loop. The code is deployed as a check-only validation with hidden bulk tests that assert on governor limits, and the reply must also flag the problem instead of quietly doing what was asked.
Difficulty mix
Leaderboard
Who's best at Governor limits & pushback
Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 19 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →
| # | Model and config | Score c@3 · 95% CI | Tasks |
|---|---|---|---|
| 1 | Claude Fable 5.1 effort high | 100%95% CI 83–100 | 19 |
| 1 | Claude Opus 5.5 effort high | 100%95% CI 83–100 | 19 |
| 1 | Claude Opus 5.5 effort medium | 100%95% CI 83–100 | 19 |
| 1 | Claude Opus 5.5 effort low | 100%95% CI 83–100 | 19 |
| 1 | GPT-6 Astra effort medium | 100%95% CI 83–100 | 19 |
| 1 | GPT-6 Astra effort high | 100%95% CI 83–100 | 19 |
| 1 | GPT-6.1 Sol effort high | 100%95% CI 83–100 | 19 |
| 8 | Claude Haiku 5.5 effort medium | 95%95% CI 75–99 | 19 |
| 8 | Claude Haiku 5.5 effort high | 95%95% CI 75–99 | 19 |
| 8 | Claude Opus 4.8 effort high | 95%95% CI 75–99 | 19 |
| 8 | Claude Opus 4.8 effort medium | 95%95% CI 75–99 | 19 |
| 8 | Claude Opus 4.8 effort low | 95%95% CI 75–99 | 19 |
| 8 | Claude Sonnet 5.5 effort high | 95%95% CI 75–99 | 19 |
| 8 | Claude Sonnet 5.5 effort low | 95%95% CI 75–99 | 19 |
| 8 | Claude Sonnet 5.5 effort medium | 95%95% CI 75–99 | 19 |
| 8 | Gemini 3.8 Flash effort medium | 95%95% CI 75–99 | 19 |
| 8 | GPT-5.6 Sol effort medium | 95%95% CI 75–99 | 19 |
| 8 | Grok 4.7 effort high | 95%95% CI 75–99 | 19 |
| 8 | MiMo V2.6 Pro FP8 · thinking on | 95%95% CI 75–99 | 19 |
| 20 | Claude Haiku 5.5 effort low | 89%95% CI 69–97 | 19 |
| 20 | Claude Sonnet 5 effort medium | 89%95% CI 69–97 | 19 |
| 20 | DeepSeek V4 Pro FP8 · effort high | 89%95% CI 69–97 | 19 |
| 20 | DeepSeek V4.1 Flash FP8 · effort high | 89%95% CI 69–97 | 19 |
| 20 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 89%95% CI 69–97 | 19 |
| 20 | Gemma 4 26B-A4B NVFP4 · vLLM · thinking on | 89%95% CI 69–97 | 19 |
| 20 | Gemma 4 31B QAT W4A16 · vLLM · thinking on | 89%95% CI 69–97 | 19 |
| 20 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high | 89%95% CI 69–97 | 19 |
| 20 | Grok 4.6 effort high | 89%95% CI 69–97 | 19 |
| 29 | Claude Sonnet 5 thinking off | 84%95% CI 62–94 | 19 |
| 29 | DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high | 84%95% CI 62–94 | 19 |
| 29 | DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high | 84%95% CI 62–94 | 19 |
| 29 | GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high | 84%95% CI 62–94 | 19 |
| 29 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low | 84%95% CI 62–94 | 19 |
| 29 | GPT-6 Luna effort medium | 84%95% CI 62–94 | 19 |
| 29 | Inkling Small FP8 · effort high | 84%95% CI 62–94 | 19 |
| 36 | Claude Haiku 4.5 thinking on | 79%95% CI 57–91 | 19 |
| 36 | Claude Opus 4.8 thinking off | 79%95% CI 57–91 | 19 |
| 36 | Claude Sonnet 5 effort high | 79%95% CI 57–91 | 19 |
| 36 | GLM-5.3 Flash FP8 · effort high | 79%95% CI 57–91 | 19 |
| 36 | Nemotron 3 Ultra thinking on | 79%95% CI 57–91 | 19 |
| 36 | Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium | 79%95% CI 57–91 | 19 |
| 42 | Claude Sonnet 5 effort low | 74%95% CI 51–88 | 19 |
| 42 | Ling 3.0 Flash BF16 · thinking on | 74%95% CI 51–88 | 19 |
| 42 | MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on | 74%95% CI 51–88 | 19 |
| 42 | Mistral Large 4 effort high | 74%95% CI 51–88 | 19 |
| 46 | MiMo V2.6 Flash FP8 · thinking on | 68%95% CI 46–85 | 19 |
| 46 | Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on | 68%95% CI 46–85 | 19 |
| 46 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 68%95% CI 46–85 | 19 |
| 46 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 68%95% CI 46–85 | 19 |
| 46 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 68%95% CI 46–85 | 19 |
| 51 | Claude Haiku 4.5 thinking off | 63%95% CI 41–81 | 19 |
| 51 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off | 63%95% CI 41–81 | 19 |
| 51 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 63%95% CI 41–81 | 19 |
| 51 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 63%95% CI 41–81 | 19 |
| 51 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 63%95% CI 41–81 | 19 |
| 56 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 60%95% CI 38–78 | 19 |
| 57 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 58%95% CI 36–77 | 19 |
| 57 | Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium | 58%95% CI 36–77 | 19 |
| 57 | Qwen3.8 Flash-Next thinking on | 58%95% CI 36–77 | 19 |
| 60 | Laguna S 2.1 FP4 · thinking on | 53%95% CI 32–73 | 19 |
| 61 | Nemotron 3.5 Lightning NVFP4 · thinking on | 47%95% CI 27–68 | 19 |
Tasks
What's in the suite
Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.
| Task | Difficulty | From results | Solve rate |
|---|---|---|---|
| Account rating from open pipeline, asked for a future call per opportunitylimits-pushback-future-per-opportunity | easy | easy | 71% |
| Case priority from the account, asked for a query in the looplimits-pushback-case-priority-query-in-loop | easy | easy | 90% |
| Clean up contact phones with one partial-success update (control, comply)limits-control-contact-phone-partial-update | easy | easy | 90% |
| Close stale deals with a SOQL for loop (control, comply)limits-control-soql-for-loop-stale-deals | easy | easy | 89% |
| Count the SOQL queries of a trigger with a querying helperlimits-count-queries-helper-per-lead | easy | easy | 98% |
| Which per-transaction limits are higher in a Queueablelimits-async-vs-sync-limits | easy | easy | 69% |
| Contact country sync, asked to load every contact into a maplimits-pushback-unfiltered-contact-query | medium | easy | 81% |
| Follow-up tasks with partial success, asked to insert each one in a try/catchlimits-pushback-followup-tasks-dml-per-record | medium | easy | 68% |
| Pipeline totals with one query per fixed stage (control, comply)limits-control-pipeline-query-per-stage | medium | easy | 74% |
| Renewal reminders, asked for a Messaging.sendEmail call per opportunitylimits-pushback-send-email-per-opportunity | medium | medium | 37% |
| Survey tasks for closed cases, asked to enqueue a Queueable per caselimits-pushback-queueable-per-closed-case | medium | hard | 31% |
| Trigger chunks of 200 in one Apex insert versus Data Loader batcheslimits-trigger-chunks-apex-vs-data-loader | medium | medium | 58% |
| Which governor limit breaks first in a looplimits-first-limit-exceeded | medium | easy | 98% |
| Count SOQL queries across cascading and re-entrant triggerslimits-count-cascading-trigger-queries | hard | easy | 94% |
| Default line discounts, asked to call a querying helper per line itemlimits-pushback-helper-query-per-line-item | hard | medium | 56% |
| Match imported leads to contacts, asked for a nested looplimits-pushback-nested-loop-lead-matching | hard | medium | 52% |
| Push new accounts to the ERP, asked for a callout per record from the triggerlimits-pushback-erp-callout-per-account | hard | hard | 23% |
| Stage-change events, asked to call EventBus.publish per opportunitylimits-pushback-eventbus-publish-per-record | hard | medium | 43% |
| Which trigger snippets fail on a 200-record insertlimits-spot-risky-trigger-snippets | hard | hard | 29% |
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au