What each task costs, and how long it takes

Score against cost

Each dot is a configuration. Across: what a task costs at the list price of the route it was reached through, on a log scale. Up: its score with its 95% interval, within 3 attempts (c@3) unless you pick another under Score. The line is the Pareto frontier: no configuration scores higher for less. The shaded corner is cheaper and higher than the median of the configurations shown. A task's cost counts every attempt its score counts, failed ones too.

Cost per task, broken down

Cheapest first, each bar split by what was paid for: input, cached input, reasoning, the answer, and output a service didn't split into reasoning and answer. Every token is priced as published, with no caching or batch discount, and reasoning as output, as every service here bills it.

Score against time

Across, the median seconds a task took, every attempt its score counts added up, on a log scale; up, the score with its 95% interval. Times include the network and the service’s queueing, and each run asked for 1 to 16 answers at once (in the numbers table), so they describe the service as Forcebench used it, not the model alone.

Time per task

Fastest first: each bar is the median seconds per task, and the tick the 90th percentile, the time nine tasks in ten took or less.

Tokens per task

Fewest first: the reasoning and the answer a task took, every attempt counted, and output a service didn't split into the two. Each model family uses its own tokeniser, so token counts compare only roughly across vendors.

How this is calculated