Suites / apex
Apex
General Apex engineering: collections and aggregate SOQL, partial-success DML, triggers and handlers (bulkification, recursion), batch, queueable (chaining, finalizers) and schedulable jobs, platform events, callouts through Named Credentials, JSON, dynamic SOQL, Schema describe, and security at API 67.0 (user mode by default). Each answer is deployed check-only to a clean scratch org at API 67.0 and must pass hidden Apex tests, including 200-record bulk tests. Test-writing tasks are graded by mutation testing: the model's tests must pass on the given implementation and fail on every hidden buggy mutant.
How it's graded
Each answer is deployed as a check-only validation to a clean scratch org together with hidden Apex tests, including 200-record bulk tests and governor-limit assertions. It passes only if it compiles and every test passes. Test-writing tasks are graded by mutation testing: the model’s tests must pass against correct code and fail against seeded bugs.
Difficulty mix
Leaderboard
Who's best at Apex
Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 20 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →
| # | Model and config | Score c@3 · 95% CI | Tasks |
|---|---|---|---|
| 1 | Claude Fable 5.1 effort high | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 4.8 effort medium | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 5.5 effort high | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 5.5 effort medium | 100%95% CI 84–100 | 20 |
| 1 | Claude Opus 5.5 effort low | 100%95% CI 84–100 | 20 |
| 1 | Claude Sonnet 5.5 effort high | 100%95% CI 84–100 | 20 |
| 1 | Claude Sonnet 5.5 effort medium | 100%95% CI 84–100 | 20 |
| 1 | Gemini 3.8 Flash effort medium | 100%95% CI 84–100 | 20 |
| 1 | GPT-6 Astra effort medium | 100%95% CI 84–100 | 20 |
| 1 | GPT-6 Astra effort high | 100%95% CI 84–100 | 20 |
| 1 | GPT-6.1 Sol effort high | 100%95% CI 84–100 | 20 |
| 1 | Grok 4.6 effort high | 100%95% CI 84–100 | 20 |
| 1 | Grok 4.7 effort high | 100%95% CI 84–100 | 20 |
| 14 | Claude Opus 4.8 effort high | 95%95% CI 76–99 | 20 |
| 14 | Claude Sonnet 5.5 effort low | 95%95% CI 76–99 | 20 |
| 14 | DeepSeek V4.1 Flash FP8 · effort high | 95%95% CI 76–99 | 20 |
| 14 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 95%95% CI 76–99 | 20 |
| 14 | GPT-5.6 Sol effort medium | 95%95% CI 76–99 | 20 |
| 19 | Claude Haiku 5.5 effort medium | 90%95% CI 70–97 | 20 |
| 19 | Claude Haiku 5.5 effort high | 90%95% CI 70–97 | 20 |
| 19 | Claude Haiku 5.5 effort low | 90%95% CI 70–97 | 20 |
| 19 | Claude Opus 4.8 effort low | 90%95% CI 70–97 | 20 |
| 19 | Claude Opus 4.8 thinking off | 90%95% CI 70–97 | 20 |
| 19 | Claude Sonnet 5 effort high | 90%95% CI 70–97 | 20 |
| 19 | Claude Sonnet 5 effort medium | 90%95% CI 70–97 | 20 |
| 19 | Claude Sonnet 5 effort low | 90%95% CI 70–97 | 20 |
| 19 | DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high | 90%95% CI 70–97 | 20 |
| 19 | GLM-5.3 Flash FP8 · effort high | 90%95% CI 70–97 | 20 |
| 19 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high | 90%95% CI 70–97 | 20 |
| 19 | MiMo V2.6 Flash FP8 · thinking on | 90%95% CI 70–97 | 20 |
| 19 | MiMo V2.6 Pro FP8 · thinking on | 90%95% CI 70–97 | 20 |
| 32 | Claude Sonnet 5 thinking off | 85%95% CI 64–95 | 20 |
| 32 | GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high | 85%95% CI 64–95 | 20 |
| 32 | GPT-6 Luna effort medium | 85%95% CI 64–95 | 20 |
| 32 | MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on | 85%95% CI 64–95 | 20 |
| 36 | DeepSeek V4 Pro FP8 · effort high | 80%95% CI 58–92 | 20 |
| 36 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low | 80%95% CI 58–92 | 20 |
| 36 | Inkling Small FP8 · effort high | 80%95% CI 58–92 | 20 |
| 36 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 80%95% CI 58–92 | 20 |
| 40 | Claude Haiku 4.5 thinking on | 75%95% CI 53–89 | 20 |
| 40 | Claude Haiku 4.5 thinking off | 75%95% CI 53–89 | 20 |
| 40 | Ling 3.0 Flash BF16 · thinking on | 75%95% CI 53–89 | 20 |
| 40 | Mistral Large 4 effort high | 75%95% CI 53–89 | 20 |
| 40 | Nemotron 3 Ultra thinking on | 75%95% CI 53–89 | 20 |
| 45 | Gemma 4 26B-A4B NVFP4 · vLLM · thinking on | 70%95% CI 48–85 | 20 |
| 45 | Gemma 4 31B QAT W4A16 · vLLM · thinking on | 70%95% CI 48–85 | 20 |
| 45 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 70%95% CI 48–85 | 20 |
| 45 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 70%95% CI 48–85 | 20 |
| 49 | DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high | 60%95% CI 39–78 | 20 |
| 49 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 60%95% CI 39–78 | 20 |
| 49 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 60%95% CI 39–78 | 20 |
| 49 | Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium | 60%95% CI 39–78 | 20 |
| 49 | Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium | 60%95% CI 39–78 | 20 |
| 54 | Laguna S 2.1 FP4 · thinking on | 55%95% CI 34–74 | 20 |
| 54 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 55%95% CI 34–74 | 20 |
| 56 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off | 50%95% CI 30–70 | 20 |
| 57 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 48%95% CI 28–68 | 20 |
| 58 | Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on | 45%95% CI 26–66 | 20 |
| 59 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 30%95% CI 15–52 | 20 |
| 59 | Qwen3.8 Flash-Next thinking on | 30%95% CI 15–52 | 20 |
| 61 | Nemotron 3.5 Lightning NVFP4 · thinking on | 15%95% CI 5–36 | 20 |
Tasks
What's in the suite
Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.
| Task | Difficulty | From results | Solve rate |
|---|---|---|---|
| Active picklist values via Schema describeapex-picklist-describe | easy | medium | 63% |
| Bulk-safe invocable action for Flowapex-invocable-contact-lookup | easy | medium | 56% |
| Closed-won amount per account with aggregate SOQLapex-won-amount-by-account | easy | easy | 78% |
| Group contacts by email domainapex-contact-email-domain | easy | easy | 70% |
| Nightly schedulable log purge with retention rulesapex-schedulable-log-purge | easy | medium | 45% |
| Payment terms parser with a custom exception hierarchyapex-custom-exception-hierarchy | easy | medium | 35% |
| Bulkify a legacy Contact trigger without changing behaviourapex-bulkify-legacy-trigger | medium | easy | 80% |
| Parse an order payload whose keys are Apex reserved wordsapex-json-reserved-keys | medium | medium | 42% |
| Partial-success lead import with per-row errorsapex-partial-success-lead-import | medium | easy | 74% |
| Publish and subscribe to shipment platform eventsapex-platform-event-shipments | medium | easy | 69% |
| Queueable with a transaction finalizer that logs and retriesapex-queueable-finalizer-retry | medium | medium | 58% |
| REST callout through a Named Credential with a single retryapex-callout-named-credential-retry | medium | medium | 51% |
| Stateful batch that flags overdue invoices and reports totalsapex-batch-stateful-overdue | medium | medium | 47% |
| Write tests that catch bugs in a shipping calculatorapex-mutation-tests-shipping | medium | medium | 63% |
| Chained Queueable that is testable end to endapex-queueable-chain-depth | hard | hard | 16% |
| Injection-safe dynamic SOQL search in user modeapex-dynamic-soql-user-mode | hard | medium | 52% |
| Open-pipeline rollup trigger that survives recursionapex-rollup-trigger-recursion | hard | medium | 55% |
| Upgrade a service to API 67.0 without changing its security behaviourapex-api67-access-modes | hard | hard | 18% |
| Write tests that catch bugs in an Opportunity trigger handlerapex-mutation-tests-won-handler | hard | hard | 6% |
| Write tests with an HttpCalloutMock that pin a REST contractapex-mutation-tests-callout | hard | medium | 48% |
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au