Study · Harness
What the harness changes
A coding agent is a model inside a harness: the program that gives it a system prompt and tools, offers it skills and manages the conversation. This study holds the model, the task and the model server still and changes only the harness.
Gemma 4 26B-A4B (effort on) and Qwen3.8 27B (effort medium) answered one medium LWC task, lwc-registration-form-validation, 10 times in each of Claude Code, opencode and pi, with and without Salesforce's skills. Every session reached the same model server through the same proxy, which fixes the model's effort, sampling and 32,768-token output budget whatever the harness asks for. With one task, this shows how much a harness can change the result on this task, not how the harnesses compare in general.
Gemma 4 26B-A4B · effort on
| Harness | Passed | Pass rate | 95% CI | Output tokens: median | 90th pct | First prompt | Model requests | Tool calls | Loaded a skill | No answer | Time per session |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Code | 8/10 | 80% | 49–94 | 7.1k | 8.2k | 14k | 1 | 0 | — | 0 | 54 s |
| Claude Code with skills | 7/10 | 70% | 40–89 | 6.8k | 8k | 15k | 1 | 0 | 0/10 | 0 | 53 s |
| opencode | 9/10 | 90% | 60–98 | 9.6k | 14k | 5k | 5 | 4 | — | 1 | 70 s |
| opencode with skills | 8/10 | 80% | 49–94 | 10k | 15k | 7.7k | 7 | 6 | 0/10 | 1 | 79 s |
| pi | 9/10 | 90% | 60–98 | 10k | 12k | 2k | 1 | 0 | — | 0 | 71 s |
| pi with skills | 10/10 | 100% | 72–100 | 9.3k | 10k | 5k | 1 | 0 | 0/10 | 0 | 68 s |
Medians per session. Tokens are counted by the model server for every request in a session, in this model's own tokens, so they compare across harnesses. The first prompt is the session's first request: the harness's own system prompt and tool definitions, plus the same task message in every arm. Every arm ran on its own, at the same number of sessions at once, on the same server, so their times compare too.
Harness against harness
- opencode against Claude Code: no clear difference (+10 pts, 95% CI −24 pts to +42 pts); this data can't rule out a difference of up to 42 points.
- pi against Claude Code: no clear difference (+10 pts, 95% CI −24 pts to +42 pts); this data can't rule out a difference of up to 42 points.
- opencode with skills against Claude Code with skills: no clear difference (+10 pts, 95% CI −26 pts to +44 pts); this data can't rule out a difference of up to 44 points.
- pi with skills against Claude Code with skills: no clear difference (+30 pts, 95% CI −4 pts to +60 pts); this data can't rule out a difference of up to 60 points.
- pi against opencode: no clear difference (±0 pts, 95% CI −31 pts to +32 pts); this data can't rule out a difference of up to 32 points.
- pi with skills against opencode with skills: no clear difference (+20 pts, 95% CI −11 pts to +51 pts); this data can't rule out a difference of up to 51 points.
With and without skills
- Claude Code with skills against Claude Code: no clear difference (−10 pts, 95% CI −43 pts to +27 pts); this data can't rule out a difference of up to 43 points.
- opencode with skills against opencode: no clear difference (−10 pts, 95% CI −42 pts to +24 pts); this data can't rule out a difference of up to 42 points.
- pi with skills against pi: no clear difference (+10 pts, 95% CI −19 pts to +40 pts); this data can't rule out a difference of up to 40 points.
Qwen3.8 27B · effort medium
| Harness | Passed | Pass rate | 95% CI | Output tokens: median | 90th pct | First prompt | Model requests | Tool calls | Loaded a skill | No answer | Time per session |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Code | 3/10 | 30% | 11–60 | 13k | 26k | 15k | 6 | 5 | — | 0 | 112 s |
| Claude Code with skills | 4/10 | 40% | 17–69 | 13k | 19k | 15k | 6 | 5 | 1/10 | 0 | 108 s |
| opencode | 6/10 | 60% | 31–83 | 6.5k | 17k | 5.5k | 4 | 3 | — | 0 | 57 s |
| opencode with skills | 4/10 | 40% | 17–69 | 7.2k | 13k | 8k | 4 | 5 | 5/10 | 0 | 61 s |
| pi | 2/10 | 20% | 6–51 | 3k | 4.4k | 2.1k | 3 | 3 | — | 0 | 26 s |
| pi with skills | 6/10 | 60% | 31–83 | 3.9k | 6.3k | 5k | 5 | 5 | 6/10 | 0 | 36 s |
Medians per session. Tokens are counted by the model server for every request in a session, in this model's own tokens, so they compare across harnesses. The first prompt is the session's first request: the harness's own system prompt and tool definitions, plus the same task message in every arm. Every arm ran on its own, at the same number of sessions at once, on the same server, so their times compare too.
Harness against harness
- opencode against Claude Code: no clear difference (+30 pts, 95% CI −12 pts to +60 pts); this data can't rule out a difference of up to 60 points.
- pi against Claude Code: no clear difference (−10 pts, 95% CI −43 pts to +27 pts); this data can't rule out a difference of up to 43 points.
- opencode with skills against Claude Code with skills: no clear difference (±0 pts, 95% CI −37 pts to +37 pts); this data can't rule out a difference of up to 37 points.
- pi with skills against Claude Code with skills: no clear difference (+20 pts, 95% CI −21 pts to +53 pts); this data can't rule out a difference of up to 53 points.
- pi against opencode: no clear difference (−40 pts, 95% CI −67 pts to +2 pts); this data can't rule out a difference of up to 67 points.
- pi with skills against opencode with skills: no clear difference (+20 pts, 95% CI −21 pts to +53 pts); this data can't rule out a difference of up to 53 points.
With and without skills
- Claude Code with skills against Claude Code: no clear difference (+10 pts, 95% CI −28 pts to +45 pts); this data can't rule out a difference of up to 45 points.
- opencode with skills against opencode: no clear difference (−20 pts, 95% CI −53 pts to +21 pts); this data can't rule out a difference of up to 53 points.
- pi with skills against pi: no clear difference (+40 pts, 95% CI −2 pts to +67 pts); this data can't rule out a difference of up to 67 points.
How it was run
The harnesses are Claude Code 2.1.289, opencode 2.0.21 and pi 1.0.2, and the skills sf-skills 1.58.0. Each session ran in its own container with the task's files, a shell, no network and no Salesforce org; the only thing it could reach was the proxy in front of the model server. The proxy sets the model's effort and sampling on every request, including a harness's own thinking settings, caps each request at the 32,768-token output budget, and stops a session after 60 requests or an hour. Answers were graded exactly as on the leaderboard.
Each harness ran as shipped, with only what running headless, offline and on the proxy needs: permission to run its tools unattended, the model's window and output budget, and its telemetry and update checks switched off. Their system prompts, tools (except web tools, which can't work here) and handling of skills are their own.
Claude Code speaks only Anthropic's Messages API. The proxy translated its requests into the same chat completions request the other two harnesses send, and the replies back, so the model got the same fields whichever harness asked. Claude Code also sends system messages mid-conversation (its environment, and how many tokens are left); the model's chat template takes one system message, first, so the proxy passed these where Claude Code put them, as system reminders. Claude Code is built for Claude models, and Anthropic doesn't support running it on others; opencode and pi are built to work with any model.
The harnesses offer the same skills differently. opencode lists each skill's name and description; pi lists each one's name, description and file, in its system prompt. Claude Code lists skills within a budget of about 1% of the context window, shared with its own bundled skills: at its defaults, the 16 Salesforce skills were listed by name only, without their descriptions. "Loaded a skill" counts sessions that loaded one of the Salesforce skills, with the harness's skill tool or by reading the skill's file.
What this does not show
One task, a few sessions per arm: a difference has to be large to be clear, and the intervals say how large a difference the data can't rule out. Other tasks may favour other harnesses. Each harness ran in one version with its default prompt and tools; the skills were measured as guidance, without the org they are built to work against. Pass rates here are this task's, not the leaderboard's.
Private evaluations
Want this for your team?
Azul Labs runs Forcebench privately for two kinds of Salesforce team.
Choosing a model to run yourself?
A private benchmark of your shortlist
Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.
- Private tasks written from your work only ever run on local models; they are never sent to an API.
- Re-testing after any change of model, quantisation or engine is available.
- Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
Building or buying a Salesforce AI tool?
A private evaluation of your tool
Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.
- It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
- Repeat evaluations after your releases are available.
- Typical turnaround is 2–3 weeks.
Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.
Or email leo@azl.au · azl.au