Models / nemotron-3-super
Nemotron 3 Super
NVIDIA's model, with open weights. Forcebench has run it in 1 configuration on the API board. API is models reached through a hosted service; Local, models served on Forcebench’s own hardware.
On the API board
Models reached through a hosted service. None of its configurations here has finished. Scores are within 3 attempts (c@3) with 95% intervals; each configuration is compared with the highest-scoring one by whether their 95% intervals overlap. Every configuration on the board → Cost and speed →
| Configuration | Tokens mean · median · p90 | No answer |
|---|---|---|
| OpenRouter · thinking onretries pending | 2.6k1.6k · 5.3k | <1% |
In the studies
- Local vs API: where it lands among the configurations of both boards.
- Pushback: whether it pushed back on requests that break at scale.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au