Suites / packaging
Packaging
Package development with unlocked, second-generation (2GP) and first-generation (1GP) managed packages: sfdx-project.json package directories, aliases, dependencies, ancestry and test setup; `sf package` and `sf package1` commands for creating, versioning, promoting, installing, upgrading and push-upgrading packages; and the platform rules behind them (IDs, promotion requirements, upgrade paths, component removal). Graded deterministically: project files are resolved the way the CLI resolves them, commands are validated against the pinned `sf` command manifest, knowledge questions are multiple choice.
How it's graded
sfdx-project.json files are resolved the way the CLI resolves them, sf package and sf package1 commands are validated against the pinned sf command manifest, and packaging-judgement questions (1GP, managed 2GP or unlocked) are multiple choice.
Difficulty mix
Leaderboard
Who's best at Packaging
Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 18 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →
| # | Model and config | Score c@3 · 95% CI | Tasks |
|---|---|---|---|
| 1 | GPT-6 Astra effort medium | 100%95% CI 82–100 | 18 |
| 2 | Claude Opus 5.5 effort high | 94%95% CI 74–99 | 18 |
| 2 | Claude Opus 5.5 effort medium | 94%95% CI 74–99 | 18 |
| 2 | Claude Opus 5.5 effort low | 94%95% CI 74–99 | 18 |
| 2 | GPT-6 Astra effort high | 94%95% CI 74–99 | 18 |
| 2 | GPT-6.1 Sol effort high | 94%95% CI 74–99 | 18 |
| 7 | Claude Opus 4.8 effort high | 89%95% CI 67–97 | 18 |
| 7 | GPT-5.6 Sol effort medium | 89%95% CI 67–97 | 18 |
| 9 | Claude Fable 5.1 effort high | 83%95% CI 61–94 | 18 |
| 9 | Claude Opus 4.8 effort medium | 83%95% CI 61–94 | 18 |
| 9 | Claude Sonnet 5.5 effort high | 83%95% CI 61–94 | 18 |
| 9 | Claude Sonnet 5.5 effort low | 83%95% CI 61–94 | 18 |
| 9 | Grok 4.6 effort high | 83%95% CI 61–94 | 18 |
| 9 | Grok 4.7 effort high | 83%95% CI 61–94 | 18 |
| 15 | Claude Opus 4.8 effort low | 78%95% CI 55–91 | 18 |
| 15 | Claude Sonnet 5.5 effort medium | 78%95% CI 55–91 | 18 |
| 15 | DeepSeek V4.1 Flash FP8 · effort high | 78%95% CI 55–91 | 18 |
| 15 | Gemini 3.8 Flash effort medium | 78%95% CI 55–91 | 18 |
| 15 | GPT-6 Luna effort medium | 78%95% CI 55–91 | 18 |
| 20 | Claude Opus 4.8 thinking off | 72%95% CI 49–88 | 18 |
| 20 | Claude Sonnet 5 effort high | 72%95% CI 49–88 | 18 |
| 22 | Claude Haiku 5.5 effort medium | 67%95% CI 44–84 | 18 |
| 22 | Claude Sonnet 5 effort medium | 67%95% CI 44–84 | 18 |
| 24 | Claude Haiku 5.5 effort high | 61%95% CI 39–80 | 18 |
| 24 | Claude Haiku 5.5 effort low | 61%95% CI 39–80 | 18 |
| 24 | MiMo V2.6 Pro FP8 · thinking on | 61%95% CI 39–80 | 18 |
| 27 | Claude Sonnet 5 effort low | 56%95% CI 34–75 | 18 |
| 27 | Inkling Small FP8 · effort high | 56%95% CI 34–75 | 18 |
| 29 | Claude Sonnet 5 thinking off | 50%95% CI 29–71 | 18 |
| 29 | DeepSeek V4 Pro FP8 · effort high | 50%95% CI 29–71 | 18 |
| 29 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low | 50%95% CI 29–71 | 18 |
| 29 | Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium | 50%95% CI 29–71 | 18 |
| 33 | DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high | 44%95% CI 25–66 | 18 |
| 33 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 44%95% CI 25–66 | 18 |
| 33 | GLM-5.3 Flash FP8 · effort high | 44%95% CI 25–66 | 18 |
| 33 | MiMo V2.6 Flash FP8 · thinking on | 44%95% CI 25–66 | 18 |
| 33 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 44%95% CI 25–66 | 18 |
| 33 | Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium | 44%95% CI 25–66 | 18 |
| 33 | Qwen3.8 Flash-Next thinking on | 44%95% CI 25–66 | 18 |
| 40 | DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high | 39%95% CI 20–61 | 18 |
| 40 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high | 39%95% CI 20–61 | 18 |
| 40 | GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high | 39%95% CI 20–61 | 18 |
| 40 | MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on | 39%95% CI 20–61 | 18 |
| 40 | Mistral Large 4 effort high | 39%95% CI 20–61 | 18 |
| 40 | Nemotron 3 Ultra thinking on | 39%95% CI 20–61 | 18 |
| 40 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 39%95% CI 20–61 | 18 |
| 47 | Claude Haiku 4.5 thinking on | 33%95% CI 16–56 | 18 |
| 47 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off | 33%95% CI 16–56 | 18 |
| 49 | Claude Haiku 4.5 thinking off | 28%95% CI 13–51 | 18 |
| 49 | Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on | 28%95% CI 13–51 | 18 |
| 49 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 28%95% CI 13–51 | 18 |
| 52 | Laguna S 2.1 FP4 · thinking on | 22%95% CI 9–45 | 18 |
| 52 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 22%95% CI 9–45 | 18 |
| 52 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 22%95% CI 9–45 | 18 |
| 55 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 19%95% CI 7–42 | 18 |
| 56 | Gemma 4 31B QAT W4A16 · vLLM · thinking on | 17%95% CI 6–39 | 18 |
| 56 | Ling 3.0 Flash BF16 · thinking on | 17%95% CI 6–39 | 18 |
| 56 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 17%95% CI 6–39 | 18 |
| 56 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 17%95% CI 6–39 | 18 |
| 60 | Gemma 4 26B-A4B NVFP4 · vLLM · thinking on | 11%95% CI 3–33 | 18 |
| 60 | Nemotron 3.5 Lightning NVFP4 · thinking on | 11%95% CI 3–33 | 18 |
Tasks
What's in the suite
Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.
| Task | Difficulty | From results | Solve rate |
|---|---|---|---|
| Configure a new unlocked package in sfdx-project.jsonpackaging-unlocked-project-config | easy | medium | 66% |
| How second-generation managed packaging differs from 1GPpackaging-1gp-vs-2gp | easy | easy | 98% |
| Match packaging IDs to the commands that use thempackaging-package-id-prefixes | easy | easy | 83% |
| Package metadata that depends on unpackaged org metadatapackaging-create-org-dependent | easy | medium | 34% |
| Stop installs of an accidentally promoted managed versionpackaging-lock-promoted-version | easy | medium | 53% |
| Build a promotable package version in CIpackaging-version-create-release-candidate | medium | medium | 44% |
| Build an extension package against the promoted base versionpackaging-dependency-released-build | medium | easy | 84% |
| Create an unlocked package without the project namespacepackaging-create-unlocked-no-namespace | medium | medium | 47% |
| Managed 2GP project with ancestry and test-only metadatapackaging-managed-test-setup | medium | hard | 29% |
| Promote a package version and install it in production from CIpackaging-promote-and-install | medium | medium | 39% |
| Schedule a push upgrade for a wave of subscriber orgspackaging-push-upgrade-schedule | medium | hard | 15% |
| What --skip-validation means for a release pipelinepackaging-skip-validation-consequences | medium | medium | 57% |
| Abandon a released managed 2GP version through ancestrypackaging-abandon-ancestor | hard | medium | 52% |
| Declare multi-level dependencies across unlocked packagespackaging-transitive-dependencies | hard | hard | 23% |
| Move metadata from one unlocked package to another in productionpackaging-migrate-metadata-between-packages | hard | hard | 23% |
| Upload a Managed - Released first-generation package versionpackaging-1gp-managed-released-upload | hard | medium | 34% |
| Which removed managed 2GP components are hard deleted on upgrade?packaging-removed-components-upgrade | hard | hard | 3% |
| Which subscriber upgrades does a 2GP ancestry tree allow?packaging-ancestry-upgrade-paths | hard | hard | 21% |
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au