Study · Reasoning effort

Is more thinking worth the tokens?

Most current models let you dial how long they reason before answering. More effort usually means more output tokens, more waiting and a bigger bill. These charts show what it buys on Salesforce work.

Vendors name effort differently (low, medium, high, xhigh, thinking on or off). Forcebench records the vendor's own setting and maps it to a common tier (off, low, medium, high, max) so models can be compared. Each line is one model at one quantisation on one engine, and the tables mark the vendor's documented default effort where there is one. Two efforts of one model are compared on the same tasks, so their difference comes with its own 95% interval from resampling those tasks (see the methodology).

Every answer has a 32,768-token output budget, reasoning included. A configuration still thinking when it reaches the budget gives no answer and fails the task, so a high effort can lose tasks that way. The No answer column shows how often that happened.

Full benchmark

Every task in every suite (272 tasks).

Score by effort tier
Overall pass@1; vertical bars are 95% confidence intervals.
  • Claude Opus 5.5
  • GPT-6 Astra
  • Claude Sonnet 5.5
  • Claude Opus 4.8
  • Claude Sonnet 5
  • Claude Haiku 5.5
  • GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense
  • Claude Haiku 4.5
  • Qwen3.8 27B · AWQ-INT4
Full benchmark: pass@1 by reasoning effort tierClaude Opus 5.5: low 89%, medium 90%, high 90%; GPT-6 Astra: medium 84%, high 80%; Claude Sonnet 5.5: low 77%, medium 75%, high 82%; Claude Opus 4.8: off 69%, low 70%, medium 75%, high 77%; Claude Sonnet 5: off 58%, low 55%, medium 60%, high 65%; Claude Haiku 5.5: low 59%, medium 64%, high 63%; GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense: off 34%, low 34%, high 43%; Claude Haiku 4.5: off 32%, on 37%; Qwen3.8 27B · AWQ-INT4: low 24%, medium 24%, max 21%0%25%50%75%100%Thinking offThinking onLowMediumHighMaxReasoning effort tierpass@1
Show the numbers
ModelEffort (tier)pass@195% CITasksNo answerTokens: meanmedianReasoning tokensLatency: median per answer
Claude Opus 5.5low89%86–932721%1.2k936—9.5 s
Claude Opus 5.5medium · vendor default90%87–93272<1%1.8k1.3k—13 s
Claude Opus 5.5high90%87–94272<1%2.4k1.5k—16 s
GPT-6 Astramedium84%80–872720%8465024136.9 s
GPT-6 Astrahigh80%76–842720%1.8k8651.4k12 s
Claude Sonnet 5.5low77%73–822720%1k699—5.6 s
Claude Sonnet 5.5medium75%70–792720%1.1k747—5.6 s
Claude Sonnet 5.5high · vendor default82%77–862720%1.8k1.2k—9.0 s
Claude Opus 4.8thinking off69%64–732720%882549—6.8 s
Claude Opus 4.8low70%65–752720%940573—7.1 s
Claude Opus 4.8medium75%71–802720%1.6k828—9.9 s
Claude Opus 4.8high77%73–812720%2k1k—12 s
Claude Sonnet 5thinking off58%52–632720%919586—6.6 s
Claude Sonnet 5low55%50–612720%730408—5.0 s
Claude Sonnet 5medium60%55–652720%1.2k562—7.2 s
Claude Sonnet 5high · vendor default65%60–702720%2.4k978—9.8 s
Claude Haiku 5.5low59%54–642720%2.2k1.4k—6.6 s
Claude Haiku 5.5medium64%59–692720%2.8k1.7k—7.7 s
Claude Haiku 5.5high63%58–68272<1%4.5k2.5k—11 s
GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 densethinking off34%29–392720%2.3k644—38 s
GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 denselow34%29–402720%733401—23 s
GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 densehigh43%38–482722%3.4k1.1k—64 s
Claude Haiku 4.5thinking off · vendor default32%27–372720%1.1k657—5.8 s
Claude Haiku 4.5thinking on37%32–422720%5.1k4.6k—33 s
Qwen3.8 27B · AWQ-INT4low24%19–282721%3.3k2.2k2.6k15 s
Qwen3.8 27B · AWQ-INT4medium24%19–282722%4.7k2.8k3.4k20 s
Qwen3.8 27B · AWQ-INT4xhigh (max) · vendor default21%17–2627225%18k18k10k141 s
Score against output tokens
Each point is one effort setting. Further right means more output tokens per task, which means more cost. The numbers are under the chart above.
  • Claude Opus 5.5
  • GPT-6 Astra
  • Claude Sonnet 5.5
  • Claude Opus 4.8
  • Claude Sonnet 5
  • Claude Haiku 5.5
  • GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense
  • Claude Haiku 4.5
  • Qwen3.8 27B · AWQ-INT4
Full benchmark: pass@1 against mean output tokens per taskClaude Opus 5.5: 1.2k tokens 89%, 1.8k tokens 90%, 2.4k tokens 90%; GPT-6 Astra: 846 tokens 84%, 1.8k tokens 80%; Claude Sonnet 5.5: 1k tokens 77%, 1.1k tokens 75%, 1.8k tokens 82%; Claude Opus 4.8: 882 tokens 69%, 940 tokens 70%, 1.6k tokens 75%, 2k tokens 77%; Claude Sonnet 5: 919 tokens 58%, 730 tokens 55%, 1.2k tokens 60%, 2.4k tokens 65%; Claude Haiku 5.5: 2.2k tokens 59%, 2.8k tokens 64%, 4.5k tokens 63%; GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense: 2.3k tokens 34%, 733 tokens 34%, 3.4k tokens 43%; Claude Haiku 4.5: 1.1k tokens 32%, 5.1k tokens 37%; Qwen3.8 27B · AWQ-INT4: 3.3k tokens 24%, 4.7k tokens 24%, 18k tokens 21%0%25%50%75%100%1k2k5k10k20kMean output tokens per task (log scale)pass@1

Output tokens include reasoning everywhere. Reasoning tokens show — where the provider or engine doesn't report reasoning separately (Anthropic's API, and vLLM with ExLlamaV3, among others), and the column is left out where none of them does. Latency, the median seconds per answer, isn't compared: runs of one model can differ in how many requests ran at once.

Claude Opus 5.5

low → high: +1 pt

From 89% to 90% on the same 272 tasks: no clear difference (+1 pt, 95% CI −2 pts to +4 pts); this data can't rule out a change of up to 4 points. Mean output tokens went from 1.2k to 2.4k (median 936 to 1.5k).

GPT-6 Astra

medium → high: −4 pts

From 84% to 80% on the same 272 tasks: clearly lower (−4 pts, 95% CI −7 pts to −0.1 pts). Mean output tokens went from 846 to 1.8k (median 502 to 865).

Claude Sonnet 5.5

low → high: +5 pts

From 77% to 82% on the same 272 tasks: clearly higher (+5 pts, 95% CI +0.1 pts to +9 pts). Mean output tokens went from 1k to 1.8k (median 699 to 1.2k).

Claude Opus 4.8

thinking off → high: +8 pts

From 69% to 77% on the same 272 tasks: clearly higher (+8 pts, 95% CI +4 pts to +13 pts). Mean output tokens went from 882 to 2k (median 549 to 1k), thinking off against reasoning on.

Claude Sonnet 5

thinking off → high: +7 pts

From 58% to 65% on the same 272 tasks: clearly higher (+7 pts, 95% CI +3 pts to +12 pts). Mean output tokens went from 919 to 2.4k (median 586 to 978), thinking off against reasoning on.

Claude Haiku 5.5

low → high: +4 pts

From 59% to 63% on the same 272 tasks: no clear difference (+4 pts, 95% CI −0.3 pts to +7 pts); this data can't rule out a change of up to 7 points. Mean output tokens went from 2.2k to 4.5k (median 1.4k to 2.5k).

GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense

thinking off → high: +9 pts

From 34% to 43% on the same 272 tasks: clearly higher (+9 pts, 95% CI +4 pts to +15 pts).

With thinking off, this run wrote its reasoning into the answer instead: 46% of its answers deliberate before answering, against 1% at low, and 12 very long answers make up 44% of its tokens. Its token count and time measure that visible reasoning, so they are not compared with the effort levels here.

Claude Haiku 4.5

thinking off → thinking on: +5 pts

From 32% to 37% on the same 272 tasks: clearly higher (+5 pts, 95% CI +0.4 pts to +9 pts). Mean output tokens went from 1.1k to 5.1k (median 657 to 4.6k), thinking off against reasoning on.

Qwen3.8 27B · AWQ-INT4

low → xhigh: −3 pts

From 24% to 21% on the same 272 tasks: no clear difference (−3 pts, 95% CI −6 pts to +2 pts); this data can't rule out a change of up to 6 points. Mean output tokens went from 3.3k to 18k (median 2.2k to 18k). At effort xhigh, 25% of answers reached the 32,768-token output cap before answering, and count as failed. This measures xhigh under Forcebench’s cap.

Lite subset, 60 tasks

Effort sweeps are expensive, so some models are swept on the lite subset: a fixed, stratified sample of 4 tasks per suite (1 easy, 2 medium, 1 hard). Lite scores are only compared with other lite scores, never with the full benchmark, and their intervals are wider.

Score by effort tier · lite subset, 60 tasks
Overall pass@1; vertical bars are 95% confidence intervals.
  • Claude Sonnet 5.5
  • GLM-5.3 · EXL3 2.75bpw experts, 4-bit dense
  • GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense
  • GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit dense
Lite subset (60 tasks): pass@1 by reasoning effort tierClaude Sonnet 5.5: low 73%, medium 77%, max 45%; GLM-5.3 · EXL3 2.75bpw experts, 4-bit dense: low 32%, high 43%; GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense: off 33%, low 32%, max 35%; GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit dense: off 23%, low 27%, max 33%0%25%50%75%100%Thinking offLowMediumHighMaxReasoning effort tierpass@1Claude Sonnet 5.5GLM-5.3 · EXL3 2.75bpw expe…GLM-5.3 Flash · EXL3 4.0bpw…GLM-5.3 Flash · EXL3 4.0bpw…
Show the numbers
ModelEffort (tier)pass@195% CITasksNo answerTokens: meanmedianReasoning tokensLatency: median per answer
Claude Sonnet 5.5low73%63–82600%1k753—6.1 s
Claude Sonnet 5.5medium77%68–85600%1.1k919—6.5 s
Claude Sonnet 5.5max45%37–536048%21k29k—201 s
GLM-5.3 · EXL3 2.75bpw experts, 4-bit denselow32%22–42600%1.2k53569819 s
GLM-5.3 · EXL3 2.75bpw experts, 4-bit densehigh43%33–53608%7.8k2.1k4.4k72 s
GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 densethinking off33%23–43600%2.6k682—39 s
GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 denselow32%23–40600%710442—25 s
GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 densemax · vendor default35%25–456027%15k12k—688 s
GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit densethinking off23%15–32600%3k860—37 s
GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit denselow27%17–37602%1.5k44645017 s
GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit densemax · vendor default33%25–436028%16k13k6.5k570 s
Score against output tokens · lite subset, 60 tasks
Each point is one effort setting. Further right means more output tokens per task, which means more cost. The numbers are under the chart above.
  • Claude Sonnet 5.5
  • GLM-5.3 · EXL3 2.75bpw experts, 4-bit dense
  • GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense
  • GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit dense
Lite subset (60 tasks): pass@1 against mean output tokens per taskClaude Sonnet 5.5: 1k tokens 73%, 1.1k tokens 77%, 21k tokens 45%; GLM-5.3 · EXL3 2.75bpw experts, 4-bit dense: 1.2k tokens 32%, 7.8k tokens 43%; GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 dense: 2.6k tokens 33%, 710 tokens 32%, 15k tokens 35%; GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit dense: 3k tokens 23%, 1.5k tokens 27%, 16k tokens 33%0%25%50%75%100%1k2k5k10k20kMean output tokens per task (log scale)pass@1Claude Sonnet 5.5GLM-5.3 · EXL3 2.75bpw expe…GLM-5.3 Flash · EXL3 4.0bpw…GLM-5.3 Flash · EXL3 4.0bpw…

Output tokens include reasoning everywhere. Reasoning tokens show — where the provider or engine doesn't report reasoning separately (Anthropic's API, and vLLM with ExLlamaV3, among others), and the column is left out where none of them does. Latency, the median seconds per answer, isn't compared: runs of one model can differ in how many requests ran at once.

Claude Sonnet 5.5lite subset

low → max: −28 pts

From 73% to 45% on the same 60 tasks: clearly lower (−28 pts, 95% CI −38 pts to −18 pts). Mean output tokens went from 1k to 21k (median 753 to 29k). At effort max, 48% of answers reached the 32,768-token output cap before answering, and count as failed. Anthropic advises a large max_tokens at the higher effort levels, so this measures max under Forcebench’s cap, not max as the vendor intends it.

GLM-5.3 · EXL3 2.75bpw experts, 4-bit denselite subset

low → high: +11 pts

From 32% to 43% on the same 60 tasks: clearly higher (+11 pts, 95% CI +3 pts to +20 pts). Mean output tokens went from 1.2k to 7.8k (median 535 to 2.1k). At effort high, 8% of answers reached the 32,768-token output cap before answering, and count as failed.

GLM-5.3 Flash · EXL3 4.0bpw experts, FP8 denselite subset

thinking off → max: +2 pts

From 33% to 35% on the same 60 tasks: no clear difference (+2 pts, 95% CI −8 pts to +13 pts); this data can't rule out a change of up to 13 points. At effort max, 27% of answers reached the 32,768-token output cap before answering, and count as failed. This measures max under Forcebench’s cap.

With thinking off, this run wrote its reasoning into the answer instead (38% of answers deliberate before answering), so its token count and time are not compared with the effort levels here.

GLM-5.3 Flash · EXL3 4.0bpw experts, 4-bit denselite subset

thinking off → max: +10 pts

From 23% to 33% on the same 60 tasks: no clear difference (+10 pts, 95% CI ±0 pts to +20 pts); this data can't rule out a change of up to 20 points. At effort max, 28% of answers reached the 32,768-token output cap before answering, and count as failed. This measures max under Forcebench’s cap.

With thinking off, this run wrote its reasoning into the answer instead (55% of answers deliberate before answering, 91% of its tokens), so its token count and time are not compared with the effort levels here.

Private evaluations

Want this for your team?

Azul Labs runs Forcebench privately for two kinds of Salesforce team.

Choosing a model to run yourself?

A private benchmark of your shortlist

Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

  • Private tasks written from your work only ever run on local models; they are never sent to an API.
  • Re-testing after any change of model, quantisation or engine is available.
  • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Get a private benchmarkHow it works

Building or buying a Salesforce AI tool?

A private evaluation of your tool

Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

  • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
  • Repeat evaluations after your releases are available.
  • Typical turnaround is 2–3 weeks.
Ask about an evaluationHow it works

Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

Or email leo@azl.au · azl.au