Models / deepseek-v4-flash-vision

DeepSeek V4 Flash Vision experimental

DeepSeek's model, with open weights. Forcebench has run it in 1 configuration on the Local board. API is models reached through a hosted service; Local, models served on Forcebench’s own hardware.

On the Local board

Models served on Forcebench’s own hardware. DeepSeek V4 Flash Vision is ranked by its one configuration: the fourth-highest score of 9 models, not clearly different from 3 other models. Scores are within 3 attempts (c@3) with 95% intervals; each configuration is compared with the highest-scoring one by whether their 95% intervals overlap. Every configuration on the board →

52%
within 3 attempts (c@3), 95% CI 47–57 · high · fourth-highest score of 9 models, not clearly different from 3 other models
16k
output tokens per task · median 14k, p90 33k
1081 s
median per task, every attempt counted, on 2× NVIDIA DGX Spark (GB10, 128 GB each) · 90th percentile 4641 s
18%
no answer within the 32,768-token budget
Every configuration of DeepSeek V4 Flash Vision on the Local board
ConfigurationRank all configsWithin 3 attempts c@3 · 95% CITokens mean · median · p90No answerLatency median per task, every attempt counted
FP8 + FP4 experts · vLLM · high · 2× NVIDIA DGX Spark (GB10, 128 GB each)highest score7
52%95% CI 47–57
16k14k · 33k18%1081 s

Latency depends on the hardware: compare it only between configurations on the same hardware.

Suites, at its highest-scoring configuration
high · FP8 + FP4 experts · vLLM · 2× NVIDIA DGX Spark (GB10, 128 GB each): within 3 attempts (c@3) on each suite with 95% intervals, against the median of the Local board's models, each at its highest-scoring configuration.
  • 95% interval
  • Local board median
Apex
60%, 95% interval 39–78%
Salesforce APIs
50%, 95% interval 29–71%
CI/CD
53%, 95% interval 31–74%
Salesforce CLI
75%, 95% interval 53–89%
Salesforce docs
12%, 95% interval 3–34%
fflib Enterprise Patterns
17%, 95% interval 6–39%
Flow
17%, 95% interval 6–39%
Governor limits & pushback
84%, 95% interval 62–94%
Lightning Web Components
94%, 95% interval 74–99%
Nonprofit Success Pack
44%, 95% interval 25–66%
Packaging
44%, 95% interval 25–66%
Permissions & access
60%, 95% interval 36–80%
Scratch org definitions
39%, 95% interval 20–61%
SOQL
90%, 95% interval 70–97%
Trigger Actions Framework
44%, 95% interval 25–66%
    Show the numbers
    Per-suite scores of DeepSeek V4 Flash Vision's highest-scoring configuration on the Local board, with the board's median
    Suitec@3 95% CILocal medianTasks
    Apex
    60%95% CI 39–78
    		70%20
    Salesforce APIs
    50%95% CI 29–71
    		44%18
    CI/CD
    53%95% CI 31–74
    		53%17
    Salesforce CLI
    75%95% CI 53–89
    		45%20
    Salesforce docs
    12%95% CI 3–34
    		12%17
    fflib Enterprise Patterns
    17%95% CI 6–39
    		17%18
    Flow
    17%95% CI 6–39
    		17%18
    Governor limits & pushback
    84%95% CI 62–94
    		84%19
    Lightning Web Components
    94%95% CI 74–99
    		89%18
    Nonprofit Success Pack
    44%95% CI 25–66
    		61%18
    Packaging
    44%95% CI 25–66
    		39%18
    Permissions & access
    60%95% CI 36–80
    		60%15
    Scratch org definitions
    39%95% CI 20–61
    		17%18
    SOQL
    90%95% CI 70–97
    		90%20
    Trigger Actions Framework
    44%95% CI 25–66
    		44%18

    In the studies

    Private evaluations

    Want this for your team?

    Azul Labs runs Forcebench privately for two kinds of Salesforce team.

    Choosing a model to run yourself?

    A private benchmark of your shortlist

    Open-weight models at the quantisations and reasoning efforts you could run on infrastructure you control, tested on your kind of Salesforce work, with a written recommendation.

    • Private tasks written from your work only ever run on local models; they are never sent to an API.
    • Re-testing after any change of model, quantisation or engine is available.
    • Typical turnaround is 2–3 weeks; large models take about a day of machine time per configuration.
    Get a private benchmarkHow it works

    Building or buying a Salesforce AI tool?

    A private evaluation of your tool

    Your tool on held-out tasks it has never seen, against API baselines (models reached through a hosted service) and Local ones (served on Forcebench’s own hardware), with a written report on where it holds up and where it fails.

    • It can start as a smaller pilot, about 30 held-out tasks with your tool against two baselines, before a full evaluation.
    • Repeat evaluations after your releases are available.
    • Typical turnaround is 2–3 weeks.
    Ask about an evaluationHow it works

    Fixed fee, quoted up front. The quote depends on the number of tasks and the number of models, tools or configurations tested.

    Or email leo@azl.au · azl.au