Suites / ci
CI/CD
Salesforce CI/CD engineering: GitHub Actions workflows and pipeline scripts that authenticate with JWT or SFDX auth URLs from secrets, validate pull requests, quick deploy validated releases, run delta deployments with sfdx-git-delta, manage scratch orgs per pull request, build and promote unlocked packages, gate builds on Code Analyzer and run LWC Jest, plus CI/CD judgement questions. Every sf command in an answer is validated against the pinned sf CLI manifest; workflow properties (triggers, job ordering, environments, cleanup, secrets) are checked deterministically, including simulated events. No org is needed.
How it's graded
GitHub Actions workflows are checked for structure and secrets hygiene, and every sf command in them is validated against the pinned sf command manifest.
Difficulty mix
Leaderboard
Who's best at CI/CD
Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 17 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →
| # | Model and config | Score c@3 · 95% CI | Tasks |
|---|---|---|---|
| 1 | Claude Opus 4.8 effort medium | 100%95% CI 82–100 | 17 |
| 1 | Claude Opus 4.8 effort low | 100%95% CI 82–100 | 17 |
| 1 | Claude Opus 4.8 thinking off | 100%95% CI 82–100 | 17 |
| 1 | Claude Sonnet 5.5 effort high | 100%95% CI 82–100 | 17 |
| 1 | GPT-6 Astra effort medium | 100%95% CI 82–100 | 17 |
| 6 | Claude Fable 5.1 effort high | 94%95% CI 73–99 | 17 |
| 6 | Claude Haiku 5.5 effort medium | 94%95% CI 73–99 | 17 |
| 6 | Claude Opus 4.8 effort high | 94%95% CI 73–99 | 17 |
| 6 | Claude Opus 5.5 effort high | 94%95% CI 73–99 | 17 |
| 6 | Claude Opus 5.5 effort low | 94%95% CI 73–99 | 17 |
| 6 | Claude Sonnet 5 effort medium | 94%95% CI 73–99 | 17 |
| 6 | GPT-6 Astra effort high | 94%95% CI 73–99 | 17 |
| 6 | GPT-6.1 Sol effort high | 94%95% CI 73–99 | 17 |
| 6 | Grok 4.6 effort high | 94%95% CI 73–99 | 17 |
| 15 | Claude Haiku 5.5 effort high | 88%95% CI 66–97 | 17 |
| 15 | Claude Opus 5.5 effort medium | 88%95% CI 66–97 | 17 |
| 15 | Claude Sonnet 5 thinking off | 88%95% CI 66–97 | 17 |
| 15 | Claude Sonnet 5.5 effort low | 88%95% CI 66–97 | 17 |
| 15 | Claude Sonnet 5.5 effort medium | 88%95% CI 66–97 | 17 |
| 15 | DeepSeek V4 Pro FP8 · effort high | 88%95% CI 66–97 | 17 |
| 15 | DeepSeek V4.1 Flash FP8 · effort high | 88%95% CI 66–97 | 17 |
| 15 | Gemini 3.8 Flash effort medium | 88%95% CI 66–97 | 17 |
| 15 | GPT-5.6 Sol effort medium | 88%95% CI 66–97 | 17 |
| 15 | GPT-6 Luna effort medium | 88%95% CI 66–97 | 17 |
| 15 | Grok 4.7 effort high | 88%95% CI 66–97 | 17 |
| 26 | Claude Haiku 5.5 effort low | 82%95% CI 59–94 | 17 |
| 26 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 82%95% CI 59–94 | 17 |
| 26 | GLM-5.3 Flash FP8 · effort high | 82%95% CI 59–94 | 17 |
| 26 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high | 82%95% CI 59–94 | 17 |
| 26 | Mistral Large 4 effort high | 82%95% CI 59–94 | 17 |
| 31 | Claude Sonnet 5 effort high | 76%95% CI 53–90 | 17 |
| 31 | DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high | 76%95% CI 53–90 | 17 |
| 31 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low | 76%95% CI 53–90 | 17 |
| 34 | Claude Sonnet 5 effort low | 71%95% CI 47–87 | 17 |
| 34 | MiMo V2.6 Pro FP8 · thinking on | 71%95% CI 47–87 | 17 |
| 34 | Nemotron 3 Ultra thinking on | 71%95% CI 47–87 | 17 |
| 34 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 71%95% CI 47–87 | 17 |
| 38 | Claude Haiku 4.5 thinking on | 65%95% CI 41–83 | 17 |
| 38 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off | 65%95% CI 41–83 | 17 |
| 38 | Inkling Small FP8 · effort high | 65%95% CI 41–83 | 17 |
| 38 | MiMo V2.6 Flash FP8 · thinking on | 65%95% CI 41–83 | 17 |
| 42 | Claude Haiku 4.5 thinking off | 59%95% CI 36–78 | 17 |
| 42 | Gemma 4 31B QAT W4A16 · vLLM · thinking on | 59%95% CI 36–78 | 17 |
| 42 | Qwen3.8 Flash-Next thinking on | 59%95% CI 36–78 | 17 |
| 45 | DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high | 53%95% CI 31–74 | 17 |
| 45 | Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium | 53%95% CI 31–74 | 17 |
| 45 | Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium | 53%95% CI 31–74 | 17 |
| 48 | GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high | 47%95% CI 26–69 | 17 |
| 48 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 47%95% CI 26–69 | 17 |
| 50 | Laguna S 2.1 FP4 · thinking on | 41%95% CI 22–64 | 17 |
| 50 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 41%95% CI 22–64 | 17 |
| 50 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 41%95% CI 22–64 | 17 |
| 50 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 41%95% CI 22–64 | 17 |
| 50 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 41%95% CI 22–64 | 17 |
| 55 | Gemma 4 26B-A4B NVFP4 · vLLM · thinking on | 35%95% CI 17–59 | 17 |
| 55 | Ling 3.0 Flash BF16 · thinking on | 35%95% CI 17–59 | 17 |
| 55 | MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on | 35%95% CI 17–59 | 17 |
| 55 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 35%95% CI 17–59 | 17 |
| 59 | Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on | 29%95% CI 13–53 | 17 |
| 59 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 29%95% CI 13–53 | 17 |
| 61 | Nemotron 3.5 Lightning NVFP4 · thinking on | 18%95% CI 6–41 | 17 |
Tasks
What's in the suite
Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.
| Task | Difficulty | From results | Solve rate |
|---|---|---|---|
| Deploy to a UAT sandbox with an SFDX auth URLci-uat-deploy-sfdx-url | easy | medium | 48% |
| LWC lint and Jest job with cached npm dependenciesci-lwc-jest-eslint | easy | easy | 100% |
| Nightly Apex regression run in a full sandboxci-nightly-apex-tests | easy | medium | 34% |
| Pre or post destructive changes for a referenced fieldci-choice-destructive-order | easy | easy | 95% |
| Test level for a production deploy with managed packagesci-choice-test-level-prod | easy | easy | 98% |
| Block pull requests on high-severity Code Analyzer violationsci-code-analyzer-gate | medium | medium | 35% |
| Fix a production deploy that lost its environment secretsci-environment-secrets-fix | medium | easy | 100% |
| Scratch orgs versus sandboxes as per-pull-request CI orgsci-choice-scratch-vs-sandbox | medium | easy | 95% |
| Script a disposable review scratch orgci-review-org-script | medium | medium | 53% |
| Source tracking behaviour in CI jobsci-choice-source-tracking-ci | medium | medium | 65% |
| Validate pull requests against production with JWT authci-pr-validate-jwt | medium | hard | 31% |
| Why a quick deploy finds no recent job IDci-choice-quick-deploy-cache | medium | easy | 98% |
| Build and promote an unlocked package version on release tagsci-unlocked-package-release | hard | hard | 32% |
| Delta validation with sfdx-git-delta and destructive changesci-delta-deploy-sgd | hard | hard | 18% |
| One workflow that routes validations and deploys by branchci-branch-routed-validation | hard | medium | 48% |
| Throwaway scratch org per pull request with guaranteed cleanupci-scratch-org-per-pr | hard | medium | 40% |
| Validate on merge, then quick deploy after approvalci-quick-deploy-on-merge | hard | medium | 37% |
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au