Sovereign AI
AI you own
Azul Labs focuses on AI models you own: open weights, running on infrastructure you control. Forcebench measures open-weight models served locally, alongside API models for comparison, on Salesforce work, and Azul Labs can run it privately for your team for a fixed fee.
Why own the model
Sovereign AI means the model is yours
Open-weight models run on infrastructure you control, on-premises or in your own cloud account. Nothing about how they behave depends on a vendor’s roadmap.
Your data stays yours
Prompts, code and customer records never leave infrastructure you control. No third-party retention policy to vet, and no data-residency question to answer for every new tool.
It can’t change under you
Models reached through an API, a hosted service, get updated, repriced and retired on someone else’s schedule. Weights you own don’t change unless you change them; re-test after any change to the model, quantisation or engine.
Costs you can plan
You pay for hardware or a fixed hosting bill, not per token. A busy month doesn’t turn into a surprise invoice.
You can check it
You can test the exact weights, quantisation and settings you’ll run in production, and test them again after every change. That is what Forcebench is for.
The catch
Which model, at what size?
Open-weight models come in dozens of sizes, each in several quantisations, served by different engines at different reasoning efforts. A smaller quantisation runs on cheaper hardware, but what does it cost you on real work? Generic benchmarks can’t say, because they don’t test Salesforce.
That is why the Forcebench leaderboard has a Local board: models served on Forcebench’s own hardware, all with open weights, with the quantisation, engine, effort and hardware of each recorded. Right now DeepSeek V4.1 Flash leads it at 73% within 3 attempts (c@3), at its best configuration (EXL3 2.9bpw · effort high), but its 95% interval overlaps 1 other local model at its best, so it is not clearly different from that one. The quantisation study and the reasoning effort study show what smaller builds and extra thinking actually buy you.
On first answers (pass@1), the best Local configuration, DeepSeek V4.1 Flash (EXL3 2.9bpw, effort high), scores 59%, and the best API configuration, Claude Opus 5.5 (effort high), 90%: the API one is 31 points higher, clearly above the Local one. The local vs API study compares them, with the tokens each spent.
Private benchmark
Your shortlist, your kind of work, a fixed fee
Before you buy hardware or commit to a model, find out how your shortlist does on the Salesforce work your team actually does. Azul Labs runs the benchmark and reports back.
- 1
Shortlist
You and Azul Labs agree a shortlist of models you could realistically own and run, typically three to five, at the quantisations and reasoning efforts that fit your hardware and budget. An API model can join as a baseline on the public tasks.
- 2
Test on your kind of work
Azul Labs runs the Forcebench suites that match what your team does, and can add private tasks written from examples of your real work. Same harness as the public leaderboard: check-only deploys to scratch orgs created for the benchmark, hidden tests, error bars.
- 3
Report back
A written report you can share internally: scores with confidence intervals per suite, speed and token cost, examples of where each model fails, and a clear recommendation of which model, quantisation and effort to run.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
- Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
- Your tasks and results stay private.
Questions
Good to know
Does Azul Labs need access to your Salesforce org?
No. Everything runs in scratch orgs created for the benchmark, inside a sandbox that refuses to touch any other org. For private tasks you share examples of the work, anonymised if needed, and Azul Labs turns them into tasks with hidden tests that only ever run on local models.
Which models can be tested?
Any open-weight model served behind an OpenAI-compatible API (vLLM, SGLang, llama.cpp, MLX and similar), at the quantisations you would actually run. API models from the big labs can be included as a baseline, so you can see what owning your model costs you in score, task by task.
Will your results be published?
No. Private benchmarks and private tasks stay private unless you ask for them to be published.
Can your team run Forcebench itself?
Yes. It is open source and free to use (Apache-2.0 code, CC BY 4.0 tasks); you just pay for your own GPUs, plus a Salesforce Dev Hub for the scratch orgs (a free Developer Edition org works). The fixed-fee benchmark is for teams that want it done properly without the setup: choosing candidates, serving each one with the right settings, writing private tasks and interpreting the results.
What does it cost?
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested. Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration. Get in touch for a quote.
Anything else? Email leo@azl.au.