How the numbers are made

The short version

• Models get real Salesforce tasks across fifteen suites. Their answers are run wherever possible: deployed as a check-only validation to a clean scratch org with hidden tests, run under Jest, executed against seeded data, or checked by deterministic validators. No AI model grades another.
• The headline score is c@3: the share of tasks a configuration solves within three attempts. After a failed answer, the model is shown what the deploy, tests or tools reported, and answers again, the way a developer works. pass@1, the first answer alone, is a choice away on the leaderboard, and it's what the studies compare, task by task. Every score on the site names which one it is.
• Every score comes with a 95% confidence interval. Where two configurations' intervals overlap, they are not clearly different; where they don't, one is clearly above or clearly below the other. Overlap never means equal.
• Every model gets the same prompt, word for word (one turn, no tools), and runs with its vendor's recommended sampling settings and a 32,768-token output budget.
• Reasoning effort and quantisation are recorded as part of each configuration, not hidden.
• Answers are generated first, straight from the inference server with no hidden retries, then graded in a sandbox. Answers that time out are re-run, not scored.
• Headline results use the full task set. Effort sweeps may use a fixed lite subset, which is only ever compared with other lite results.
• Every task's reference answer must pass its grader and known-wrong answers must fail it. Public tasks carry a canary GUID. Final answers, grades and token counts for public tasks are published in the repo; reasoning traces are available on request through the gated Hugging Face dataset; nothing from the private pool is published except approved aggregate scores.

Glossary

The words this site uses, and what each one means everywhere it appears.

API

A configuration reached through a hosted service: the model vendor's own API or a router such as OpenRouter. Its quantisation and serving engine aren't disclosed. The API board ranks these.

Local

A configuration served on Forcebench’s own hardware, with its quantisation, engine and hardware recorded. The Local board ranks these.

Weights

Open or closed: whether the model's weights are published. A separate attribute from API and Local.

Configuration

One model run one way: the model, API or Local, its quantisation and engine where they're known, and its reasoning effort. Each configuration has its own row. Table headers shorten it to “config”.

Effort

How hard a model is told to think before it answers, in the vendor's own terms (low, high, thinking on or off and so on), with a common tier (off, low, medium, high or max) for comparing across vendors.

c@3 (and c@2)

The headline score: the share of tasks solved within three (or two) attempts, each attempt after the first shown what the environment reported about the one before. Exactly what counts is under Within three attempts.

pass@1

The share of tasks solved by the first answer alone, averaged over the suites so each suite carries equal weight. The studies compare configurations on it, task by task.

Fixed

Of the answers that failed their first attempt, how many a second or third attempt fixed: “14 of 26”.

Retries pending

A finished configuration whose second and third attempts haven't all been answered and graded yet. It has no c@3 score, so it's listed but not ranked.

No answer

The share of tasks where a configuration gave no answer at all, usually because it used its whole 32,768-token output budget thinking. A task with no answer fails.

Not clearly different

Two scores whose 95% intervals overlap. It never means equal. When the intervals don't overlap, one is clearly above or clearly below the other.

Run it, don’t judge it

The benchmarks that have held up best in AI coding, like HumanEval and SWE-bench, share one idea: a task passes when the code works, not when it looks right. Forcebench follows the same rule wherever it can.

• Check-only deploys with hidden tests. Apex, Trigger Actions Framework, fflib, Flow, NPSP code and permissions answers are deployed as a check-only validation to a clean scratch org, together with hidden Apex tests, including 200-record bulk tests and governor-limit assertions. The tests have to pass, the way a reviewer's test class would exercise your code. If the answer doesn't compile or a test fails, it scores zero. Where the task is to write tests, mutation testing checks that the model's tests pass against correct code and fail against seeded bugs.
• Effective access. For permission sets and groups, hidden tests create Minimum Access users, assign the answer's permissions and assert what those users can actually read, edit and do, using System.runAs.
• Jest. Lightning Web Components run under hidden Jest tests (@salesforce/sfdx-lwc-jest).
• Execution accuracy. SOQL answers run against a scratch org seeded with fixed data, and the result set must equal the gold query's, as in the BIRD and Spider text-to-SQL benchmarks.
• Deterministic validators. Some answers can't simply be run: a CLI command, a packaging config, a GitHub Actions pipeline, an HTTP request. These are parsed and checked by code. Every sf command is checked against the command manifest of a pinned Salesforce CLI release, so an invented or deprecated flag fails. Scratch org definitions are also validated against the official schemas and feature list, and their settings are deployed check-only.
• Docs. Answers are short, exact facts, and the cited documentation page must be right too.

No AI model grades any task. Model-as-judge grading is cheap and flexible, but it's biased towards confident, well-formatted answers, and it can't tell you whether a trigger survives a bulk insert. Where a task is about knowledge or judgement (before-save flow, after-save flow or Apex?), it's multiple choice or has a short exact answer.

Answer extraction is forgiving about prose and strict about the answer: explain as much as you like, but if the command block, file or Answer: line is missing, the task fails.

Suite by suite

What a score means

Every score is averaged the same way. Take pass@1, the first answer alone: each configuration answers each task once. A task scores 1 if that answer passes its grader and 0 if it doesn't. A suite score is the mean over the suite's tasks, and the overall score is the average of the suite scores, so a suite with more tasks doesn't dominate.

A score of 62% means: pick a suite and a task in it, ask the model once and use what comes back, and it works about 62% of the time. That's the number that matters when a developer asks a model once and uses the answer.

Models aren't deterministic: ask the same one twice and you can get one answer that works and one that doesn't. With one sample per task, any single task's result is partly luck, which is why Forcebench only reports scores across many tasks, with intervals. A run can take repeated samples (n per task) and count how many pass (c): pass@1 is then the mean of c / n, and the same samples give pass@k, the chance that at least one of k independent attempts works, via the standard unbiased estimator from the HumanEval paper:

where C(a, b) is "a choose b". For k = 1 this reduces to c / n.

The headline score, c@3, is averaged exactly the same way: a task counts as solved if any of its first three attempts passed. It's what each board ranks by, and the next section says exactly what an attempt is.

The headline: within three attempts (c@3)

Nobody ships a first draft. You deploy, read what the org or the CLI tells you, and fix it. From v0.2, Forcebench measures that loop, and it's the headline: each board ranks by c@3, the share of tasks solved within three attempts, once any finished configuration on it has one. Exactly what counts:

• An attempt is one complete answer to the task, from the same model at the same effort, with the same request settings and the same 32,768-token output budget, graded by the same grader. Attempt 1 is the run itself.
• Who gets another attempt. A task whose answer failed, when the environment said why. A task that passed is never asked again.
• What the model is shown. The conversation so far (the task and its earlier answers, without their reasoning) and what a real tool printed about the failure: the org's deploy and compile errors, the names and messages of failing tests, query errors, Jest and ESLint output, the sf CLI's complaints about flags, scratch org definition and sfdx-project.json validation, or that the reply couldn't be read or ran out of budget. Then it's asked for a complete corrected answer. Hidden tests' names and failure messages are shown, the way a CI pipeline shows them; their source never is.
• When there's no retry. If the only problem is that the answer doesn't match what the task expects (another command, a missing value, the wrong choice), telling the model would hand it the answer, so the failure stands. Multiple-choice and short-answer tasks never get a retry.
• The score. A task counts as solved within three attempts if any of its first three passed. Suites and the overall score are averaged exactly like pass@1, with the same intervals. c@2 counts two attempts instead of three.
• Fixed counts the answers that failed attempt 1 and passed attempt 2 or 3, out of all that failed attempt 1: “14 of 26”. It accounts for the gap between pass@1 and c@3, though not exactly: the scores average suites, and Fixed counts answers.
• When a configuration gets a c@3 score. Only when every one of its retries has been answered and graded. Until then it's listed as “retries pending” and isn't ranked, so a ranking never mixes c@3 with pass@1.

Isn't that just another roll of the dice? Forcebench checked. On tasks each model had already failed, three attempts with feedback solved clearly more than three fresh attempts without it, for all four models tested: from about a third more to about twice as many. The models and numbers are in the feedback study. The feedback is doing the work. Aider's coding benchmark does something similar, with two attempts.

Where pass@1 is used instead: the studies (reasoning effort, quantisation, max effort, local vs API and the coding agent) compare configurations task by task on first answers, because a paired comparison needs both sides' answers to the same tasks and not every configuration in them has had its retries. Each task's difficulty comes from first answers too. Every score on the site names its metric, and hovering the name shows what it means.

Error bars: 95% confidence intervals

With tens of tasks per suite rather than thousands, a score is an estimate. Swap in a slightly different set of equally fair tasks and it would move. The confidence interval says by how much.

Forcebench uses a bootstrap: take the task list, draw a new list of the same length by picking tasks at random with replacement (so some appear twice and some not at all), recompute the score, and do that 10,000 times. The middle 95% of those recomputed scores is the interval. For the overall score, tasks are resampled within each suite, so every resample keeps the same suite mix.

The resampling is over tasks, not individual answers. With one answer per task that's the same thing. It matters for c@k and for repeated samples: several attempts at the same task aren't independent pieces of evidence (a model with a blind spot for polymorphic SOQL will probably miss every attempt), and treating them as independent would make the intervals look narrower, and the results more certain, than they really are.

How to read them: if two configurations' intervals overlap, Forcebench calls them not clearly different, and doesn't claim either is better. That rule is deliberately conservative. Two configurations whose intervals overlap slightly can still differ in a paired comparison on the same tasks (below), but the leaderboard won't call a winner on that basis. When the intervals don't overlap, one configuration scores clearly above the other on this benchmark.

Per-suite intervals are much wider than overall ones because each suite has fewer tasks. Treat a per-suite ranking as a hint, the overall ranking as the evidence.

A suite's own score gets a different interval, the Wilson score interval. A suite has between 15 and 20 tasks, and when a configuration passes every one of them, every bootstrap resample passes them all too, so the bootstrap says 100–100: a certainty that 20 tasks can't give. The Wilson interval is made for a share of a small number of tries and stays honest at both ends: 20 of 20 tasks gives 84–100%, and 0 of 20 gives 0–16%. The overall scores and the paired differences below keep the bootstrap.

Paired differences: two runs of one model

When the studies compare two runs of the same model (two quantisations, two efforts, with and without an agent), both runs answered the same tasks. That allows a sharper comparison than checking whether two intervals overlap: take each task's difference between the two runs' grades, and bootstrap those differences, resampling tasks within each suite as for the overall score, 5,000 times with a fixed seed so a page shows the same interval every build. The middle 95% of the resampled differences is the interval of the difference.

How the studies word it: if that interval excludes zero, one run scored clearly higher or clearly lower. If it includes zero, there is no clear difference, and the interval says how large a change this data can't rule out: “no clear difference (−2 pts, 95% CI −6 pts to +3 pts); this data can't rule out a change of up to 6 points”. No clear difference doesn't mean equal scores: with a few dozen tasks, a change of several points can sit inside the interval. Where several comparisons point the same way but none is clear on its own, the studies say that too.

One prompt for every model

Every model gets the same system prompt and the same fixed output-format instructions for each kind of answer, word for word. There are no worked examples (it's zero-shot), no tools and no per-model prompt tuning: one user turn, one reply.

Some models would score higher with a prompt written just for them. That's true and it's the point: Forcebench measures how a model does with a reasonable, fixed instruction, which is closer to how most developers use them. The only thing that differs is the chat template, which the inference engine applies the way each model expects.

Sampling settings

Each model runs with its vendor's recommended sampling settings for thinking mode (temperature, top-p and so on), and the exact request fields are published with every run. Forcing every model to the same settings sounds fairer but isn't: temperature 0 is not a neutral default, and some reasoning models do noticeably worse, or get stuck repeating themselves, when run greedily. Using the settings the vendor tested with measures each model as it's meant to be used.

Reasoning effort is a dimension, not a footnote

Many models can be told how hard to think before answering. The same model at low and high effort can differ by more than two different models do, and uses very different amounts of time and tokens. So Forcebench treats the same model at a different effort as a different configuration with its own row.

Vendors name effort differently (low, medium, high, xhigh, thinking on or off). Each result records the vendor's own setting and a normalised tier (off, low, medium, high, max) for comparison, along with mean output tokens, reasoning tokens and latency per task. The effort study shows what extra thinking buys.

Output budget, not a clock

Every answer gets an output budget of 32,768 tokens, reasoning included. A configuration that thinks past it without producing an answer fails the task. That's a real cost of running a model at that effort, so it counts against the score. It's the same length DeepSeek-R1 was evaluated with and Qwen3 recommends for most tasks.

The leaderboard reports these separately as the No answer rate, so "answered wrongly" and "never answered" can be told apart. The reasoning a model wrote before running out is kept with the run's full replies, available on request through the gated Hugging Face dataset, so you can check whether it was working through the problem or going in circles.

The budget is tokens, never wall-clock time: a slow machine mustn't cost a model points. A request that times out is treated like an endpoint failure and re-run, not scored.

Cost and speed per task

The cost and speed page shows, beside each score, what a task cost and how long it took. For now it covers API configurations reached through Anthropic or OpenRouter, whose prices and times are measured the same way; configurations reached through another service are named under each chart, and Local ones come later, with charts of their own. Every number comes from what the runs recorded and a dated price list; where a configuration lacks something a chart needs, it's left out of that chart and named under it, never estimated.

• A task counts every attempt its score counts. For the first-answer score, the first answer. Within three attempts, also each retry of a failed answer, until one passes or the environment had nothing to report. Failed answers and answers cut off by the budget count like any other; an answer the endpoint never returned counts as a failed task but adds no tokens, time or cost.
• Cost is the list price of the route the run took, from the price list in the benchmark's repository (prices/<date>.yaml, never edited once published): a router's price for the endpoint it was pinned to, not the vendor's direct price. Every token is priced as published, with no caching or batch discount, so a cached input token costs the full input price, and reasoning is priced as output, as every service here bills it. Cost per solved task is the total cost divided by the tasks solved; it has no value when none was. A configuration has no cost where its service publishes no price, where any answer it counts has no token counts, or where a prompt was longer than its price holds for.
• Time is wall-clock seconds from the request to the last token of each answer, added up over a task's attempts, shown as the median and the 90th percentile over tasks. A try the endpoint failed and that was started again isn't counted. It includes the network and the service's queueing, and depends on how many answers each run asked for at once, which the page shows: it describes the service as Forcebench used it, not the model alone. The leaderboard and the model pages show the same median per task.
• Tokens are as each service counted them: input, cached input (recorded from 10 October 2026), and output, split into reasoning and the answer where the service reported reasoning separately. Each model family uses its own tokeniser, so token counts compare only roughly across vendors.

How a run executes

Runs follow the shape of established code benchmarks such as SWE-bench, EvalPlus and LiveCodeBench, in two phases:

1. Generate. Every answer is requested from the model and stored; nothing else happens in this phase. Requests go directly to the inference server (vLLM, SGLang and so on) or the vendor's API, never through a general-purpose proxy whose timeouts, retries or defaults would become part of the measurement. Responses are streamed, and the exact request fields (sampling, effort switch, token budget) are recorded. Interrupted runs resume without redoing finished answers.
2. Grade. Stored answers are graded in a sandbox: check-only deploys with hidden tests, Jest, SOQL execution and deterministic validators. Grading can be repeated at any time without calling the model, for example after a grader fix.

No hidden retries. Client-side retries are off. A proxy or SDK that silently restarts slow requests would keep only the answers that happened to finish quickly, biasing slow configurations towards short answers. Answers affected before this was fixed have been invalidated and regenerated, with the history kept.

Full and lite. Headline results use the full task set. Expensive sweeps, like one model at several effort levels, may use lite: a fixed, stratified subset of 4 tasks per suite (1 easy, 2 medium, 1 hard), chosen by a hash of the task id. Lite results are only compared with other lite results, are labelled wherever they appear, and never enter the leaderboard.

Quantisation and inference engines are recorded

An open-weights model isn't one thing. The same model can be served at full precision (BF16), at 8 bits (FP8) or at 4 bits (AWQ-INT4, GGUF Q4_K_M and others), by different inference engines such as vLLM, SGLang or llama.cpp. Each can change the results, so every configuration records its model, quantisation, engine and effort.

API configurations are reached through a hosted service: the vendor's own API or a router such as OpenRouter. Their precision and serving engine aren't public, so those are left blank. The quantisation study compares the same model across precisions.

Every task is tested too

A grader that passes everything, or nothing, is worse than no grader. So every task ships with:

• a reference answer, written exactly as a model would reply, that must pass its grader;
• alternative answers, correct solutions in a different style, that must also pass, so a grader can't quietly accept only one phrasing;
• negative answers, plausible common mistakes (SOQL in a loop, the wrong sharing keyword, a deprecated sfdx force: command, an invalid scratch org feature), that must fail;
• and an empty reply, which must fail.

forcebench validate runs all of these, and CI runs it on every change. Prompts state everything the grader checks (names, signatures, paths, return shapes), so any correct solution passes. It's the same reason you'd want a test class that fails when the trigger is commented out.

Isolation: nothing is committed

Answers that need an org are deployed as check-only validations, together with the hidden tests, to long-lived grader scratch orgs. A check-only deploy compiles the code and runs the tests, then rolls everything back, so nothing is ever committed. Every attempt starts from the same org state, and attempts can't affect each other or later runs. SOQL tasks query a scratch org seeded with fixed data.

Salesforce enforces 75% per-class code coverage on validations that run specified tests, even in scratch orgs. Forcebench ignores those coverage warnings: only compile results and test outcomes count.

Contamination controls

If a model has seen the tasks during training, its score measures memory, not skill. Forcebench guards against that in a few ways:

• A canary GUID in every task. Each task file carries the Forcebench canary string (the BIG-bench convention). Model trainers who respect canaries can filter the files out of training data, and a model that can reproduce the string has almost certainly seen them. Private tasks carry a separate canary whose value is never published, so contamination by them can be told apart.
• Original tasks. Tasks are written for Forcebench, never copied from certification exams, Trailhead or blogs.
• Dated tasks. Each task records when it was created, so results can be sliced by task age.
• Held-out tasks. A private pool of held-out tasks is written the same way as the public ones and kept apart from them by the harness. Nothing from it is published except approved aggregate scores. A model that does much better on the public tasks than on the private ones gets flagged.

Published answers

Final answers, grades and token counts for public tasks are published in the repo; reasoning traces are available on request through the gated Hugging Face dataset; nothing from the private pool is published except approved aggregate scores. In the repo, each run's results/runs/<run>/cases.jsonl holds every final answer, its grade and its token counts, and records only how much reasoning each answer used. The dataset holds the full replies, reasoning included where the model's provider returns it, for evaluation and research under its access terms. Each run also records the benchmark version, the harness commit, the task versions, the configuration and the exact request fields. If a number looks wrong, you can read exactly what the model answered and why it passed or failed. Stored answers can be re-graded without calling the model again, so a grader fix applies to past runs. Found a grading bug? Open an issue.

Versioning

Forcebench is versioned; this site is built from v0.2.0, with fifteen suites. Scores are only comparable within the same benchmark version. Changing a task bumps that task's version, and results from older versions of a task are excluded from the leaderboard.

What Forcebench doesn’t tell you

• The leaderboard is single-turn with no tools: it measures what a model knows and can produce in one go, not how well it works as an agent inside a repo with an org. Agent behaviour is covered by the studies: the coding agent study and the harness study.
• Tasks are small and self-contained so they can be graded automatically. Real projects are messier, with more context, legacy code and people.
• It can't tell you how a model will do on your org, your naming conventions or your managed packages.
• Scratch orgs are Developer edition unless a suite needs otherwise, so some enterprise features are validated structurally rather than in an org of that edition.
• Salesforce changes three times a year. Tasks pin the API version they rely on, and the sf command manifest is pinned and refreshed deliberately.
• Suites have tens of tasks, not thousands, and pass@1 takes one answer per task. The intervals say so honestly; read them.

If you need those answers for your own team's work, Azul Labs runs Forcebench privately.