Picking an agent model on a single leaderboard number means buying a result measured on somebody else's workload. DAREBench, published in September 2026, ran 35 models through one shared agent runtime across six workload groups. No model led more than three of the six, and the best score on tool-mediated multimodal work was 61.8 percent. Model selection is a workload decision, and the evidence has to be disaggregated before it counts as evidence.
Why does one benchmark score pick the wrong model?
The case against the aggregate score is not new. VECTOR Labs set it out on 24 August 2026 in a piece on agentic benchmark selection: composite benchmarks fold many sub-tasks into one headline number, a model can post a respectable composite while degrading sharply on the exact category your production workload lives in, and published scores therefore belong in a shortlist rather than in a decision. Their instruction: disaggregate the composite and map its categories onto your own task distribution.
That instruction had a practical problem. Disaggregating only works if somebody publishes a breakdown, and the breakdowns that existed came from different harnesses. Terminal-Bench measures command-line work, OSWorld measures desktop control, MMSearch measures multimodal search, and each brings its own runtime, tool interface and scorer, so comparing across them mixes capability with runtime compatibility. You could accept the advice and still have nothing to act on. DAREBench closes that gap.
What did DAREBench actually measure?
The benchmark takes 233 tasks from 22 published source benchmarks through a three-stage filter: anything without a definable artifact contract or reliable scorer is dropped, anything a mid-sized open model solves three times out of three is dropped as too easy, and the rest is selected for category coverage. Those tasks then go into a two-by-three workload matrix: input modality (text, multimodal) crossed with execution form (single-step, multi-step, multi-step plus tools).
Every model runs inside one shared execution environment built on OpenClaw, the open-source agent runtime, with the same agent loop, tool suite, workspace abstraction and logging. Twenty-three commercial API models came through OpenRouter; twelve open-weight models were served locally with vLLM on a single RTX PRO 6000 at FP16 and 32K context, without quantisation. That is 7,587 model-task runs.
Two design choices matter more than the scale. Each task carries a contract naming the workspace, the tools, the required output paths with validity predicates, the scorer and a time budget, so a run passes by producing a valid artifact rather than text resembling one. And every positive score from the judge goes through deterministic trigger rules, with suspicious cases escalated to a meta-judge from a different model family.
Which model wins depends on which workload group you are in
Across the six groups the leaders split three ways. Claude Opus 4.8 led three of them: single-step text at 98.8, text multi-step with tools at 65.3, and multimodal multi-step with tools at 61.8. GPT-5.5 led text multi-step at 83.6 and single-step multimodal at 90.0. Gemini 3.1 Pro led multimodal multi-step at 88.5. Three vendors, six groups, no sweep.
The more useful reading is the spread inside a single model rather than the ranking between them. Claude Opus 4.8 scores 98.8 on single-step text and 61.8 on tool-mediated multimodal work: 37 points of range within one model, on one harness. An aggregate that averages those two numbers describes no workload that exists in your estate.
Why is multi-step-plus-tools the number that should worry you?
Because it is the only column that resembles production. A single-step task is a question with an answer. Reading an inbound purchase order, checking the lines against the ERP and flagging the mismatch is multi-step plus tools by definition. In that column the best model reaches 65.3 percent on text and 61.8 percent on multimodal input. Roughly one run in three fails, on curated medium-to-hard tasks, in a controlled runtime, with the tools already wired up.
This lines up with the July finding we wrote up in the HANDBOOK.md guardrail benchmark, where the best model followed a long written policy in 36.2 percent of trials. Same shape of answer: capability measured on a single turn does not survive a long sequence of turns.
A production agent is therefore not a model, it is a model inside a control structure: durable execution, a bounded action set, and a human check at the step where an error becomes expensive. We laid that out in AI agents in production.
What does the accuracy-cost curve say about paying up?
The paper reports average tokens and estimated dollars per task at OpenRouter reference rates. Two findings follow.
Token count and cost are decoupled. Claude Opus 4.6 averaged 231k tokens per task and MiMo-V2.5 averaged 276k, yet MiMo cost roughly one seventy-first as much. A token budget is not a cost budget.
The two modalities have different frontier shapes. On text the frontier is nearly flat above two cents a task: MiMo-V2.5 at $0.006 scores 75.5, MiMo-V2.5-Pro at $0.012 scores 76.0, and GPT-5.5 at $0.155 scores 76.8. That last step is roughly thirteen times the cost for 0.8 points. On multimodal the frontier keeps climbing: MiMo-V2-Omni at $0.003 scores 46.0, MiMo-V2.5 at $0.005 scores 62.1, GPT-5.4 at $0.097 scores 63.5, Gemini 3.1 Pro at $0.114 scores 67.9, and Claude Opus 4.8 at $0.123 scores 71.1.
Put a volume on it. Take 20,000 agent tasks a month: roughly 100 users doing ten tasks a day over twenty working days. On text, moving from MiMo-V2.5 to GPT-5.5 takes the monthly bill from about $120 to about $3,100, and the 1.3 extra points buy about 260 additional correct runs, so each costs roughly $11.50. On multimodal at the same volume, moving from MiMo-V2-Omni to Claude Opus 4.8 takes the bill from about $60 to about $2,460, and the 25.1 extra points buy about 5,020 correct runs, at roughly $0.48 each.
The extra correct answer that paying up buys therefore costs about twenty-four times more on text than on multimodal here. Take the least expensive sensible multimodal baseline instead, MiMo-V2.5 at $0.005 and 62.1, and that extra answer costs about $1.31, still close to an order of magnitude under the text figure. The assumptions, plainly: these are the paper's reference rates rather than negotiated pricing, the task mix inside each modality is treated as uniform, and accuracy points as interchangeable when some failures cost far more than others. The direction survives all three. On text, paying up buys very little; on multimodal, paying up is most of the result. Route by workload rather than by provider and that becomes a configuration rather than a procurement round, the argument in model router architecture.
Does on-premise cost you more than fifteen accuracy points?
This is the part of the paper that should change a Dutch procurement conversation, and not the part that gets quoted. The quoted finding is a gap: Qwen3.6-27B, an Apache-2.0 dense model released in April 2026, was the strongest local model at 59.8 overall, roughly 15 points behind the frontier commercial models, with Qwen3.6-35B-A3B close behind at 56.6. Smaller 8B and 9B local models were unstable inside the full agent loop.
The audit finding is the one that matters. The meta-judge removed 143 positive scores as unsupported by execution evidence, and 124 of those 143 came from locally deployed models. Eighty-six were deadlock loops, where the agent cycles through materially equivalent actions without changing the workspace until it exhausts the 32K context, and the judge still awards credit. Twenty were pseudo-tool-call credulity cases, where the model emits text shaped like a tool call and the judge treats it as an executed one. Neither pattern appeared once in the API models.
So the on-premise choice does not only cost accuracy. It introduces a failure class in which the agent achieves nothing and your harness reports a pass. The paper is careful about scope and so should we be: these counts hold under one single-GPU profile at 32K context, and the authors explicitly do not claim the same rate at a larger serving configuration. Read it as a list of things to test for, not a fixed on-premise penalty.
The Dutch relevance is specific. Since the Cyberbeveiligingswet, the NIS2 implementation act, entered into force on 15 August 2026, the cyber regulation for the government sector refers directly to BIO2 version 1.3, which makes most of it a legal duty for the bodies it covers rather than guidance; a small set of BIO2 measures sits outside that act and stays mandatory self-regulation. Mid-market suppliers to those bodies inherit the requirement through their contracts. Data residency and ISMS scope are therefore real, non-technical reasons to put a model on your own hardware. The moment you make that choice for compliance, deadlock loops and pseudo-tool-calls move onto your risk register, and a judge's pass rate stops being evidence on its own. Trajectory-level instrumentation is the answer, which is what agent observability is for.
Who built this benchmark, and does that change how you read it?
Apply VECTOR Labs' independence test to the paper supplying these numbers. DAREBench comes from the Chinese Academy of Sciences, MiLM Plus at Xiaomi, and Brown University, and MiMo is Xiaomi's own model family, sitting on the Pareto frontier in both panels. That is a disclosure rather than a disqualification, and three things narrow it: every workload-level accuracy leader is a non-Xiaomi model, the cost advantage comes from published OpenRouter rates anyone can re-derive, and the tasks, runtime and audit rules are on GitHub. Take the accuracy ranking at face value, re-check the cost figures against today's rates, and reproduce the frontier on your own tasks.
What does a supplier acceptance test look like after this?
If you are the IT manager or CIO taking delivery of an agent from a specialist supplier while your ICT partner keeps the estate running, six things change.
- Name the workload group before the model. Write down the modality and the execution form of the process. Almost every real process is multi-step with tools, which tells you which column of any table you are allowed to read.
- Ask for the breakdown and refuse the average. A supplier who cannot say how the chosen model performs on your workload group is quoting marketing. The breakdown exists in most benchmark papers even when the headline does not.
- Score artifacts, not text. Acceptance criteria should name the output path, the validity condition and the time budget. A run that produces plausible prose and no file has failed.
- Put a second judge in the contract. If any part of your acceptance testing uses a language model as a judge, require a second model from a different family to re-check positive results against the execution trace. That one control removed 143 false passes here.
- Price per correct answer, not per token. Run the arithmetic above on your own volume before the model choice is locked, and record the assumptions next to the number.
- Test the deployment mode you will actually run. An accuracy figure from a hosted API does not transfer to the same weights on your own GPU at a smaller context. If compliance is pushing you on-premise, test for the loop-until-timeout pattern explicitly.
Where to start
Take the process you are most likely to automate next quarter and write one line: modality, execution form, monthly volume, and the cost of a wrong answer. That line tells you which column to read, whether your accuracy-cost frontier is flat or steep, and whether on-premise is a requirement or a preference.
We do this work alongside the ICT partner or internal IT team that already runs the estate, not instead of them. The same framing sits under our applied AI and ML work for mid-sized businesses and our AI implementation practice in the Netherlands.
Last updated: 25 September 2026.
We build custom AI and LLM systems that run in production: a clickable MVP by the second call, fixed steps, and you own the code.
AI development agency in the Netherlands →
