Suites / permissions
Permissions & access
Least-privilege access design: permission sets (object, field, tab, app, Apex class, user and custom permissions), session-based permission sets, permission set groups with muting, fixing over-permissive access, and Apex that checks access (custom permissions, stripInaccessible, UserRecordAccess) under API 67.0's user-mode default. Metadata answers are deployed check-only to a scratch org with hidden custom objects; hidden Apex tests create Minimum Access users, assign the answer's permission sets or groups and assert effective access with System.runAs. A few knowledge tasks cover how profiles, permission sets, groups, muting and sharing combine.
How it's graded
Permission sets and groups are deployed as a check-only validation to a scratch org. Hidden Apex tests create Minimum Access users, assign the answer’s permission sets or groups and assert effective access with System.runAs. A few knowledge tasks are multiple choice or have a short exact answer.
Difficulty mix
Leaderboard
Who's best at Permissions & access
Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 15 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →
| # | Model and config | Score c@3 · 95% CI | Tasks |
|---|---|---|---|
| 1 | Claude Opus 5.5 effort high | 100%95% CI 80–100 | 15 |
| 1 | Claude Opus 5.5 effort medium | 100%95% CI 80–100 | 15 |
| 1 | Claude Opus 5.5 effort low | 100%95% CI 80–100 | 15 |
| 1 | GPT-6 Astra effort medium | 100%95% CI 80–100 | 15 |
| 1 | GPT-6.1 Sol effort high | 100%95% CI 80–100 | 15 |
| 6 | Claude Fable 5.1 effort high | 93%95% CI 70–99 | 15 |
| 6 | Claude Opus 4.8 effort high | 93%95% CI 70–99 | 15 |
| 6 | Claude Opus 4.8 effort medium | 93%95% CI 70–99 | 15 |
| 6 | Claude Sonnet 5.5 effort high | 93%95% CI 70–99 | 15 |
| 6 | Claude Sonnet 5.5 effort low | 93%95% CI 70–99 | 15 |
| 6 | Claude Sonnet 5.5 effort medium | 93%95% CI 70–99 | 15 |
| 6 | GPT-5.6 Sol effort medium | 93%95% CI 70–99 | 15 |
| 6 | GPT-6 Astra effort high | 93%95% CI 70–99 | 15 |
| 6 | Grok 4.6 effort high | 93%95% CI 70–99 | 15 |
| 6 | Grok 4.7 effort high | 93%95% CI 70–99 | 15 |
| 16 | Claude Haiku 5.5 effort medium | 87%95% CI 62–96 | 15 |
| 16 | Claude Haiku 5.5 effort high | 87%95% CI 62–96 | 15 |
| 16 | Claude Haiku 5.5 effort low | 87%95% CI 62–96 | 15 |
| 16 | Claude Opus 4.8 effort low | 87%95% CI 62–96 | 15 |
| 16 | Claude Opus 4.8 thinking off | 87%95% CI 62–96 | 15 |
| 16 | Claude Sonnet 5 thinking off | 87%95% CI 62–96 | 15 |
| 16 | DeepSeek V4 Pro FP8 · effort high | 87%95% CI 62–96 | 15 |
| 16 | DeepSeek V4.1 Flash FP8 · effort high | 87%95% CI 62–96 | 15 |
| 16 | Gemini 3.8 Flash effort medium | 87%95% CI 62–96 | 15 |
| 16 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low | 87%95% CI 62–96 | 15 |
| 16 | GPT-6 Luna effort medium | 87%95% CI 62–96 | 15 |
| 27 | Claude Sonnet 5 effort high | 80%95% CI 55–93 | 15 |
| 27 | Claude Sonnet 5 effort medium | 80%95% CI 55–93 | 15 |
| 27 | Claude Sonnet 5 effort low | 80%95% CI 55–93 | 15 |
| 27 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 80%95% CI 55–93 | 15 |
| 27 | DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high | 80%95% CI 55–93 | 15 |
| 27 | GLM-5.3 Flash FP8 · effort high | 80%95% CI 55–93 | 15 |
| 27 | MiMo V2.6 Pro FP8 · thinking on | 80%95% CI 55–93 | 15 |
| 34 | Claude Haiku 4.5 thinking on | 73%95% CI 48–89 | 15 |
| 34 | GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high | 73%95% CI 48–89 | 15 |
| 34 | MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on | 73%95% CI 48–89 | 15 |
| 34 | MiMo V2.6 Flash FP8 · thinking on | 73%95% CI 48–89 | 15 |
| 38 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high | 67%95% CI 42–85 | 15 |
| 38 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off | 67%95% CI 42–85 | 15 |
| 38 | Inkling Small FP8 · effort high | 67%95% CI 42–85 | 15 |
| 38 | Mistral Large 4 effort high | 67%95% CI 42–85 | 15 |
| 38 | Nemotron 3 Ultra thinking on | 67%95% CI 42–85 | 15 |
| 38 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 67%95% CI 42–85 | 15 |
| 38 | Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium | 67%95% CI 42–85 | 15 |
| 45 | DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high | 60%95% CI 36–80 | 15 |
| 45 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 60%95% CI 36–80 | 15 |
| 45 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 60%95% CI 36–80 | 15 |
| 45 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 60%95% CI 36–80 | 15 |
| 45 | Qwen3.8 Flash-Next thinking on | 60%95% CI 36–80 | 15 |
| 50 | Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium | 53%95% CI 30–75 | 15 |
| 51 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 50%95% CI 27–73 | 15 |
| 52 | Claude Haiku 4.5 thinking off | 47%95% CI 25–70 | 15 |
| 52 | Gemma 4 31B QAT W4A16 · vLLM · thinking on | 47%95% CI 25–70 | 15 |
| 52 | Laguna S 2.1 FP4 · thinking on | 47%95% CI 25–70 | 15 |
| 52 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 47%95% CI 25–70 | 15 |
| 52 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 47%95% CI 25–70 | 15 |
| 57 | Gemma 4 26B-A4B NVFP4 · vLLM · thinking on | 40%95% CI 20–64 | 15 |
| 57 | Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on | 40%95% CI 20–64 | 15 |
| 57 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 40%95% CI 20–64 | 15 |
| 60 | Ling 3.0 Flash BF16 · thinking on | 33%95% CI 15–58 | 15 |
| 61 | Nemotron 3.5 Lightning NVFP4 · thinking on | 7%95% CI 1–30 | 15 |
Tasks
What's in the suite
Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.
| Task | Difficulty | From results | Solve rate |
|---|---|---|---|
| App and tab visibility in a permission setpermissions-tab-app-visibility | easy | hard | 32% |
| Integration user permission set for Apex RESTpermissions-integration-api-access | easy | medium | 51% |
| Read-only claims viewer permission setpermissions-readonly-claims-viewer | easy | medium | 65% |
| Session-based permission set for refund elevationpermissions-session-based-elevation | easy | medium | 52% |
| Auditors see every review but edit only their ownpermissions-view-all-not-modify-all | medium | easy | 69% |
| Fix an over-permissive technician permission setpermissions-fix-over-permissive | medium | easy | 94% |
| Gate large discounts with a custom permissionpermissions-custom-permission-apex | medium | medium | 63% |
| Invoice clerk access with a required fieldpermissions-required-field-fls | medium | hard | 27% |
| Object permissions versus record sharingpermissions-sharing-vs-object-perms | medium | easy | 84% |
| What must stay on the profilepermissions-profile-only-settings | medium | medium | 51% |
| Bulk-safe "which records can I edit" checkpermissions-bulk-editable-ids | hard | medium | 40% |
| Directory query that drops unreadable fields at API 67.0permissions-strip-inaccessible-v67 | hard | medium | 45% |
| Effective access with a muted group plus direct assignmentspermissions-effective-access-muting | hard | easy | 94% |
| Permission set group with a muting permission setpermissions-psg-muting | hard | hard | 11% |
| What sharing keywords enforce at API 66.0 and 67.0permissions-with-sharing-api-versions | hard | hard | 8% |
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au