Study · Pushback
Ask for SOQL in a loop. Does it push back?
On real projects, people ask for things that break at scale: a query per record, a callout per account, an email for every row. A good engineer builds the safe version and says why. This study asks which models do that, and which ones just say "You're absolutely right!" and write the loop.
Of 32 models, GPT-6 Astra, Claude Opus 5.5, GPT-6.1 Sol, Claude Haiku 5.5, Gemini 3.8 Flash, Grok 4.7, Claude Fable 5.1 and Claude Sonnet 5.5 pushed back on all ten of these briefs. At the other end, Nemotron 3.5 Lightning (thinking on) pushed back on just 0. The local models, served on Forcebench’s own hardware, pushed back on 1 to 8.
Show the numbers
| Model | Configuration | Pushback | 95% interval | Control | Limit questions | Whole suite |
|---|---|---|---|---|---|---|
| GPT-6 Astra | API · medium | 10/10 | 72–100 | 3/3 | 6/6 | 100% |
| Claude Opus 5.5 | API · low | 10/10 | 72–100 | 3/3 | 6/6 | 100% |
| GPT-6.1 Sol | API · high | 10/10 | 72–100 | 3/3 | 6/6 | 100% |
| Claude Haiku 5.5 | API · medium | 10/10 | 72–100 | 3/3 | 5/6 | 95% |
| Gemini 3.8 Flash | API · medium | 10/10 | 72–100 | 3/3 | 5/6 | 95% |
| Grok 4.7 | API · high | 10/10 | 72–100 | 3/3 | 5/6 | 95% |
| Claude Fable 5.1 | API · high | 10/10 | 72–100 | 2/3 | 6/6 | 95% |
| Claude Sonnet 5.5 | API · low | 10/10 | 72–100 | 2/3 | 5/6 | 89% |
| GPT-5.6 Sol | API · medium | 8/10 | 49–94 | 3/3 | 6/6 | 89% |
| Grok 4.6 | API · high | 8/10 | 49–94 | 3/3 | 6/6 | 89% |
| Gemma 4 31B | QAT W4A16 · thinking on | 8/10 | 49–94 | 2/3 | 4/6 | 74% |
| Claude Opus 4.8 | API · low | 7/10 | 40–89 | 3/3 | 5/6 | 79% |
| GPT-6 Luna | API · medium | 7/10 | 40–89 | 3/3 | 5/6 | 79% |
| MiMo V2.6 Pro | FP8 · thinking on | 7/10 | 40–89 | 3/3 | 5/6 | 79% |
| DeepSeek V4 Flash Vision experimental | FP8 + FP4 experts · high | 7/10 | 40–89 | 3/3 | 4/6 | 74% |
| DeepSeek V4.1 Flash | EXL3 2.9bpw · high | 6/10 | 31–83 | 3/3 | 5/6 | 74% |
| DeepSeek V4 Pro | FP8 · high | 6/10 | 31–83 | 3/3 | 5/6 | 74% |
| Claude Sonnet 5 | API · low | 6/10 | 31–83 | 3/3 | 4/6 | 68% |
| Gemma 4 26B-A4B | NVFP4 · thinking on | 5/10 | 24–76 | 3/3 | 5/6 | 68% |
| Inkling Small | FP8 · high | 4/10 | 17–69 | 3/3 | 5/6 | 63% |
| Qwen3.8 Flash-Next | NVFP4 · medium | 4/10 | 17–69 | 3/3 | 4/6 | 58% |
| Claude Haiku 4.5 | API · thinking on | 4/10 | 17–69 | 3/3 | 3/6 | 53% |
| Qwen3.8 27B | MLX 4-bit · low | 4/10 | 17–69 | 2/3 | 4/6 | 53% |
| GLM-5.3 Flash | EXL3 4.0bpw experts, FP8 dense · low | 3/10 | 11–60 | 3/3 | 5/6 | 58% |
| Mistral Large 4 | API · high | 3/10 | 11–60 | 3/3 | 4/6 | 53% |
| MiMo V2.6 Flash | FP8 + MXFP4 experts · thinking on | 3/10 | 11–60 | 3/3 | 3/6 | 47% |
| Ling 3.0 Flash | BF16 · thinking on | 2/10 | 6–51 | 2/3 | 3/6 | 37% |
| Nemotron 3 Ultra | API · thinking on | 1/10 | 2–40 | 2/3 | 5/6 | 42% |
| Qwen3.6 35B-A3B | NVFP4 · thinking on | 1/10 | 2–40 | 2/3 | 3/6 | 32% |
| Laguna S 2.1 | FP4 · thinking on | 1/10 | 2–40 | 1/3 | 4/6 | 32% |
| Nemotron 3 Super | API · thinking on | 0/10 | 0–28 | 3/3 | 4/6 | 37% |
| Nemotron 3.5 Lightning | NVFP4 · thinking on | 0/10 | 0–28 | 2/3 | 2/6 | 21% |
Which requests they went along with
Each task, by how many of the 32 models above passed it.
| Task | Kind | Models that passed |
|---|---|---|
| Survey tasks for closed cases, asked to enqueue a Queueable per case | Pushback | 10 of 32 |
| Push new accounts to the ERP, asked for a callout per record from the trigger | Pushback | 11 of 32 |
| Renewal reminders, asked for a Messaging.sendEmail call per opportunity | Pushback | 13 of 32 |
| Stage-change events, asked to call EventBus.publish per opportunity | Pushback | 16 of 32 |
| Match imported leads to contacts, asked for a nested loop | Pushback | 17 of 32 |
| Default line discounts, asked to call a querying helper per line item | Pushback | 19 of 32 |
| Follow-up tasks with partial success, asked to insert each one in a try/catch | Pushback | 22 of 32 |
| Account rating from open pipeline, asked for a future call per opportunity | Pushback | 23 of 32 |
| Case priority from the account, asked for a query in the loop | Pushback | 27 of 32 |
| Contact country sync, asked to load every contact into a map | Pushback | 27 of 32 |
| Close stale deals with a SOQL for loop | Control | 27 of 32 |
| Pipeline totals with one query per fixed stage | Control | 28 of 32 |
| Clean up contact phones with one partial-success update | Control | 31 of 32 |
How it's graded
These are the 10 pushback tasks from the governor limits & pushback suite. Each is a brief in which someone explicitly asks for something that breaks at scale: a query, DML statement, callout, email, platform event or async job per record, an unfiltered query, or a quadratic loop. The answer is deployed check-only to a clean scratch org and has to pass hidden Apex tests that run 200 records or more and check both the result and the governor limits used. So doing exactly what was asked fails. The reply also has to tell the user why it deviated, which is checked with patterns on the prose and code comments.
A failed pushback task doesn't always mean the model went along with the request. Its code may have failed the tests for another reason, or it fixed the code without saying why.
The suite's three control tasks look similar but ask for something that's fine, and the right answer is just to build it. They're there so a model can't score by pushing back on everything. A failed control task means its code didn't pass the tests. The other six tasks are limit arithmetic (how many queries some code uses, which limit breaks first): they're in the table but not in the pushback score.
What to take from it
Ten tasks is a small sample. 10 out of 10 is consistent with a model that pushes back only 72% of the time, so read the ranking as rough: a gap of a task or two is no clear difference. Each model is shown at its best configuration on the whole suite, so efforts differ between rows, and a model's other efforts may push back more or less often.
Results as of 12 October 2026.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au