Suites / npsp
Nonprofit Success Pack
Engineering on the Nonprofit Success Pack (NPSP) managed packages (npsp, npe01, npo02, npe03, npe4, npe5): the Household Account model, gifts and payments, GAU allocations, soft credits, Enhanced Recurring Donations, affiliations and relationships, TDTM trigger handlers and the BDI data import API. Code answers are deployed to a scratch org with NPSP installed and must pass hidden Apex tests that exercise NPSP's own automation; SOQL answers run against seeded NPSP data; knowledge questions cover NPSP architecture and configuration.
How it's graded
Code answers are deployed as a check-only validation to a scratch org with NPSP installed and must pass hidden Apex tests that exercise NPSP’s own automation. SOQL answers run against seeded NPSP data; knowledge questions cover NPSP architecture and configuration.
Difficulty mix
Leaderboard
Who's best at Nonprofit Success Pack
Within three attempts (c@3) on this suite only, with 95% confidence intervals. With 18 tasks the intervals are wide: where two overlap, those configurations are not clearly different. 1 finished configuration isn’t shown yet: its retries haven’t finished. Open in the full leaderboard →
| # | Model and config | Score c@3 · 95% CI | Tasks |
|---|---|---|---|
| 1 | Gemini 3.8 Flash effort medium | 100%95% CI 82–100 | 18 |
| 1 | Grok 4.6 effort high | 100%95% CI 82–100 | 18 |
| 3 | Claude Fable 5.1 effort high | 94%95% CI 74–99 | 18 |
| 3 | Claude Opus 5.5 effort medium | 94%95% CI 74–99 | 18 |
| 3 | Claude Opus 5.5 effort low | 94%95% CI 74–99 | 18 |
| 3 | Claude Sonnet 5.5 effort high | 94%95% CI 74–99 | 18 |
| 3 | Claude Sonnet 5.5 effort low | 94%95% CI 74–99 | 18 |
| 3 | Claude Sonnet 5.5 effort medium | 94%95% CI 74–99 | 18 |
| 3 | GPT-5.6 Sol effort medium | 94%95% CI 74–99 | 18 |
| 3 | GPT-6 Astra effort medium | 94%95% CI 74–99 | 18 |
| 3 | Grok 4.7 effort high | 94%95% CI 74–99 | 18 |
| 12 | Claude Opus 4.8 effort high | 89%95% CI 67–97 | 18 |
| 12 | Claude Opus 4.8 effort medium | 89%95% CI 67–97 | 18 |
| 12 | Claude Opus 4.8 effort low | 89%95% CI 67–97 | 18 |
| 12 | Claude Opus 4.8 thinking off | 89%95% CI 67–97 | 18 |
| 12 | Claude Opus 5.5 effort high | 89%95% CI 67–97 | 18 |
| 12 | DeepSeek V4.1 Flash FP8 · effort high | 89%95% CI 67–97 | 18 |
| 12 | DeepSeek V4.1 Flash Native FP8, FP4 experts · SGLang · effort high | 89%95% CI 67–97 | 18 |
| 12 | GPT-6 Astra effort high | 89%95% CI 67–97 | 18 |
| 20 | Claude Haiku 5.5 effort medium | 83%95% CI 61–94 | 18 |
| 20 | Claude Sonnet 5 effort high | 83%95% CI 61–94 | 18 |
| 20 | DeepSeek V4.1 Flash EXL3 2.9bpw · vLLM + ExLlamaV3 · effort high | 83%95% CI 61–94 | 18 |
| 20 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort high | 83%95% CI 61–94 | 18 |
| 20 | GPT-6 Luna effort medium | 83%95% CI 61–94 | 18 |
| 25 | Claude Haiku 5.5 effort high | 78%95% CI 55–91 | 18 |
| 25 | Claude Haiku 5.5 effort low | 78%95% CI 55–91 | 18 |
| 25 | Claude Sonnet 5 effort low | 78%95% CI 55–91 | 18 |
| 25 | GLM-5.3 Flash EXL3 4.0bpw experts, 4-bit dense · TensorFold · effort high | 78%95% CI 55–91 | 18 |
| 25 | GPT-6.1 Sol effort high | 78%95% CI 55–91 | 18 |
| 25 | MiMo V2.6 Pro FP8 · thinking on | 78%95% CI 55–91 | 18 |
| 31 | Claude Sonnet 5 thinking off | 72%95% CI 49–88 | 18 |
| 31 | DeepSeek V4 Pro FP8 · effort high | 72%95% CI 49–88 | 18 |
| 33 | Claude Sonnet 5 effort medium | 67%95% CI 44–84 | 18 |
| 33 | GLM-5.3 Flash FP8 · effort high | 67%95% CI 44–84 | 18 |
| 33 | Mistral Large 4 effort high | 67%95% CI 44–84 | 18 |
| 36 | Gemma 4 26B-A4B NVFP4 · vLLM · thinking on | 61%95% CI 39–80 | 18 |
| 36 | Gemma 4 31B QAT W4A16 · vLLM · thinking on | 61%95% CI 39–80 | 18 |
| 36 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · effort low | 61%95% CI 39–80 | 18 |
| 36 | GLM-5.3 Flash EXL3 4.0bpw experts, FP8 dense · vLLM + ExLlamaV3 · thinking off | 61%95% CI 39–80 | 18 |
| 36 | Qwen3.8 Flash-Next NVFP4 · vLLM · effort medium | 61%95% CI 39–80 | 18 |
| 36 | Qwen3.8 Flash-Next MLX 4-bit · MTPLX · effort medium | 61%95% CI 39–80 | 18 |
| 36 | Qwen3.8 Flash-Next INT4-AutoRound, 5 of 10 experts · TensorFold (Zig) · effort medium | 61%95% CI 39–80 | 18 |
| 43 | Claude Haiku 4.5 thinking on | 56%95% CI 34–75 | 18 |
| 43 | Claude Haiku 4.5 thinking off | 56%95% CI 34–75 | 18 |
| 43 | Inkling Small FP8 · effort high | 56%95% CI 34–75 | 18 |
| 43 | MiMo V2.6 Flash FP8 + MXFP4 experts · SGLang · thinking on | 56%95% CI 34–75 | 18 |
| 43 | MiMo V2.6 Flash FP8 · thinking on | 56%95% CI 34–75 | 18 |
| 43 | Nemotron 3 Ultra thinking on | 56%95% CI 34–75 | 18 |
| 43 | Qwen3.8 27B MLX 8-bit · MTPLX · effort low | 56%95% CI 34–75 | 18 |
| 43 | Qwen3.8 27B MLX 4-bit · MTPLX · effort low | 56%95% CI 34–75 | 18 |
| 43 | Qwen3.8 27B Splash 4-bit · Splash · effort low | 56%95% CI 34–75 | 18 |
| 43 | Qwen3.8 Flash-Next MLX 4-bit · TensorFold · effort medium | 56%95% CI 34–75 | 18 |
| 53 | Qwen3.6 35B-A3B NVFP4 · vLLM · thinking on | 50%95% CI 29–71 | 18 |
| 53 | Qwen3.8 27B AWQ-INT4 · vLLM · effort low | 50%95% CI 29–71 | 18 |
| 55 | Qwen3.8 27B AWQ-INT4 · vLLM · effort medium | 47%95% CI 27–69 | 18 |
| 56 | DeepSeek V4 Flash Vision experimental FP8 + FP4 experts · vLLM · effort high | 44%95% CI 25–66 | 18 |
| 56 | Ling 3.0 Flash BF16 · thinking on | 44%95% CI 25–66 | 18 |
| 56 | Qwen3.8 Flash-Next thinking on | 44%95% CI 25–66 | 18 |
| 59 | Laguna S 2.1 FP4 · thinking on | 39%95% CI 20–61 | 18 |
| 59 | Nemotron 3.5 Lightning NVFP4 · thinking on | 39%95% CI 20–61 | 18 |
| 59 | Qwen3.8 27B AWQ-INT4 · vLLM · effort xhigh | 39%95% CI 20–61 | 18 |
Tasks
What's in the suite
Solve rate is the share of finished runs whose first answer to the task passed (pass@1), a rough guide to how hard models find it. Difficulty is the task author’s label; from results is what the solve rate says: easy when two thirds or more of the finished runs pass it, hard at a third or fewer, once five have graded it.
| Task | Difficulty | From results | Solve rate |
|---|---|---|---|
| NPSP versus Nonprofit Cloudnpsp-vs-nonprofit-cloud | easy | easy | 100% |
| Query pledge payments still duenpsp-soql-unpaid-payments-due | easy | easy | 96% |
| Record a donation the NPSP waynpsp-record-gift | easy | medium | 40% |
| Set donors' current employer with NPSP affiliationsnpsp-affiliation-primary-employer | easy | easy | 69% |
| Which NPSP object API names are real?npsp-namespace-objects | easy | medium | 50% |
| Annual-report totals per GAUnpsp-soql-gau-totals | medium | easy | 77% |
| Create one NPSP household for several new contactsnpsp-household-create-members | medium | hard | 26% |
| Predict NPSP's household formal greetingnpsp-household-formal-greeting | medium | hard | 31% |
| Promote NPSP Customizable Rollups to productionnpsp-crlp-promote-to-production | medium | easy | 99% |
| Record parent relationships with NPSP reciprocalsnpsp-relationship-parent-child | medium | medium | 61% |
| Replace a pledge's open payments with an installment plannpsp-payment-installments | medium | easy | 67% |
| Skip an NPSP trigger handler for one integration usernpsp-trigger-handler-exclude-user | medium | easy | 100% |
| Split a gift across GAUs by percentagenpsp-gau-percent-allocations | medium | easy | 77% |
| Bulk partial soft credits linked to contact rolesnpsp-partial-soft-credits | hard | hard | 27% |
| Create monthly Enhanced Recurring Donationsnpsp-enhanced-rd-monthly | hard | medium | 35% |
| Import web gifts through NPSP Data Importer (BDI)npsp-bdi-gift-import | hard | hard | 2% |
| Major gift follow-up as an NPSP TDTM handlernpsp-tdtm-major-gift-handler | hard | hard | 32% |
| Testing a custom TDTM handler in a subscriber orgnpsp-tdtm-test-registration | hard | hard | 13% |
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au