AI Agent Benchmarks: Why 24% Pass Means 97% Right

DAYJOB, published on 1 October 2026, scores an agent attempt as a pass only if it satisfies every binary criterion in an expert rubric that runs to roughly fifty items. The strongest model passes 24.7 percent of healthcare tasks. Divide that across the rubric and the implied per-criterion accuracy is about 97 percent. That second number, not the first, is the one that should decide how you scope an agent handoff.

What did DAYJOB actually measure?

DAYJOB is a benchmark from Surge AI, submitted to arXiv on 1 October 2026. It contains 130 tasks built by working professionals: 50 in healthcare and 80 in finance. The paper estimates a professional would need 13.6 hours on average for a healthcare task and 16.6 for a finance one. Surge's own write-up quotes higher averages, 19.6 and 21.6 hours; the paper's figures are used throughout here. Each task ships as a containerised Harbor environment with an expert rubric of binary criteria, a median of 47.5 per healthcare task and 57.5 per finance task, which an agentic judge applies to the files the agent delivers.

The scoring rule is the part worth reading twice. In the authors' words, "an attempt passes only if it meets every criterion." The same pass-everything rule drives Surge's earlier HANDBOOK.md policy-following benchmark. Across 30 model configurations from 13 developers, the strongest, Claude Opus 5.5, passes 24.7 percent of healthcare and 23.9 percent of finance attempts, and the median configuration passes 0.6 and 2.5 percent. The public harness runs five attempts per task under Apache-2.0 and grades with Claude Opus 4.8 as an agentic judge; a task's score is the mean reward over those five attempts. Surge released all healthcare tasks, 50 of the 80 finance tasks, the harness and the leaderboards. Note that the live healthcare leaderboard now lists 65 tasks, more than the 50 described in the paper, so treat the task count as the paper's snapshot rather than a fixed number.

What separates this from the benchmark most vendors quote is the shape of the prompt. Surge's own write-up puts DAYJOB prompts at 49 to 81 words against GDPval's 337, and input files at 19.8 to 25.7 against 1.2. GDPval, published in September 2025, covers 1,320 tasks across 44 occupations and grades by blind pairwise comparison against human deliverables. It asks whether the output is as good as a professional's. DAYJOB asks whether the agent can work out what the output is supposed to be. If your pilot prompt is long and your production traffic is short, you have been measuring the first and buying the second, and it is the same gap that makes a strong demo prove nothing in production.

Why does a 24 percent pass rate not mean 24 percent competence?

Because an all-or-nothing rubric compounds. If an attempt passes only when it clears all n binary criteria, and you treat the criteria as independent, then the all-pass rate p and the per-criterion accuracy a are related by p = a to the power n. Invert it: a equals p to the power one over n.

Run the numbers on the published results. Healthcare: 0.247 to the power 1/47.5 gives 97.1 percent per criterion, a miss rate of 2.9 percent. Finance: 0.239 to the power 1/57.5 gives 97.5 percent. The median configuration, the one that looks hopeless at 0.6 percent, comes out at 89.8 percent per criterion in healthcare and 93.8 percent in finance. GPT-6 Astra at 11.6 percent in healthcare implies 95.6 percent, and Gemini 4 Argon at 20.3 percent in finance implies 97.3 percent. Both sit on the public finance leaderboard.

That reframes the gap between models. On the headline number, Claude Opus 5.5 is more than twice GPT-6 Astra in healthcare. Per criterion, the difference is 97.1 against 95.6: a miss rate of roughly one in 34 against one in 22. Real, but not the chasm the leaderboard draws. The lesson generalises: an aggregate score is a reasonable filter and a poor selector, which is why a model choice should name its workload first.

The independence assumption has to be stated honestly, because it is doing work. Independence is the optimistic end of the range. If failures were perfectly correlated, a 24.7 percent all-pass rate would mean 24.7 percent per-criterion accuracy. The true figure therefore sits somewhere between 24.7 and 97.1 percent, and the calculation above gives the ceiling, not a point estimate. The headline numbers on their own support only the ceiling. Narrowing the band needs the distribution of misses per attempt rather than the pass rate, so read the paper's own threshold results before you quote 97 percent as a measurement.

I am not the first to notice that the rubric drives the score. InferenceWire made the same observation about the finance set, writing that agents must "get essentially all of them right to pass." What I have not seen anyone put a number on is the consequence.

How many criteria can one agent handoff actually carry?

Invert the formula again and ask what n you can afford. Holding the all-pass rate at 90 percent, which is roughly where a change board stops asking questions: at 97.1 percent per criterion you can afford 3.6 criteria. At 98 percent, 5.2. At 99 percent, 10.5. At 99.5 percent, 21.

Forward, from the same 97.1 percent: five criteria give an 86.3 percent all-pass rate, ten give 74.5 percent, fifteen give 64.3 percent, twenty give 55.5 percent.

So the scoping rule, with the independence assumption stated and therefore generous: an agent handoff you intend to accept without reading it should carry about four checkable criteria, not fifty. Every criterion you add to the acceptance contract multiplies rather than adds. This is the engineering content of a 24 percent score, and it is a design constraint rather than a verdict on the models.

Why doesn't a smaller deliverable fix it on its own?

Because the failures are not independent, and the paper says so. In its case studies, "agents accept premises that the record contradicts and carry wrong inputs through otherwise consistent analyses." That is correlated failure by construction: one wrong input fails every criterion downstream of it.

The distinction matters because it changes what splitting buys you. Under independence, halving the criterion count roughly squares your odds. Under clustering, splitting buys almost nothing, because the error sits upstream of every piece you split into. Four separate four-criterion handoffs fed the same wrong exchange rate fail four times.

Which leaves two levers, and you need both. Cut the criterion count per handoff, so the arithmetic is survivable. And pin the inputs, so the agent does not choose its own source of record, period, units, sign convention or counterparty. Pinning means those arrive as resolved parameters from a system you already run, before the agent starts reasoning.

Where should the human review actually sit?

Here I disagree with InferenceWire, which concludes from the same benchmark that you treat agent output as a rough first draft "that a qualified person still checks line by line." Two problems with that.

The first is arithmetic. Reviewing a fifty-criterion deliverable line by line consumes a meaningful fraction of the 13.6 hours the paper estimates the task takes a professional to do in the first place, so the agent has saved you the easy part and left the expensive part. The second is worse: the failure mode defeats that kind of review. The paper's finding is that the analysis downstream of a bad input is otherwise consistent. Internally coherent, plausibly formatted work resting on one wrong number is the hardest artefact there is to catch, and a reviewer reading for coherence will sign it off.

Put the gate upstream instead. Define a short list of invariants and check them before the agent runs and again at the handoff boundary: source of record, units and currency, period, sign convention, and whether the premise in the request survives contact with the record. InferenceWire's own case studies are all upstream errors of exactly this class, including a sign error that inflated a profit figure and a currency-unit mistake worth 25.7 million dollars that only two of 66 runs caught. Five checks on the inputs catch what fifty checks on the output will not.

What does this change for a Dutch mid-market IT estate?

For an informatiemanager or CIO somewhere between 250 and 5,000 FTE, the consequence is procedural rather than technical. The agent use cases that get through a change board are the ones with a small, nameable acceptance contract, and the instinct in most proposals runs the other way: the business case reads better the bigger the deliverable, so scope grows until the handoff has fifty criteria and nobody can accept it without reading all of it. The arithmetic above is the argument for cutting it back, and it is an argument a risk owner can follow. It is also the first question we ask in applied AI work at mid-sized firms: not which model, but how many criteria the handoff carries.

Two specifics for a Dutch estate. Your systems of record are already the pinning mechanism: AFAS, Dynamics 365 Business Central, SAP, your EPD or your TMS hold the authoritative period, unit and counterparty, so an agent that reads them rather than inferring them from an attachment removes most of the upstream error class for free. And in regulated work the review you design is the one an auditor will ask to see. A documented upstream invariant check is evidence; "a colleague read it over" is not.

The seam is worth naming. Your ICT partner runs the platform and keeps it up. What changes when a specialist AI supplier arrives is that somebody has to write the acceptance contract for each agent handoff and hold it to four lines, then prove it still holds after the next model swap. That is a different job from beheer, and in an AI implementation in the Netherlands it is the one that decides whether the agent reaches production.

How do you turn this into an acceptance test?

• Write the rubric before you build. If you cannot list the binary criteria a reviewer would tick, the handoff is not specified and no benchmark score will tell you whether it works.
• Count them. Above roughly eight, the handoff is too large to accept unreviewed. Split it, and accept that splitting only helps once the inputs are pinned.
• Pin the inputs. Source of record, period, units, sign convention and counterparty are parameters resolved by a system, never inferences the agent makes from an attachment.
• Measure all-pass, not average. Run the handoff five times, as the DAYJOB harness does, and report the share of runs that cleared every criterion. An average score hides exactly the behaviour production will show you.

None of this requires waiting for a better model. A 97 percent per-criterion agent is already useful on a four-criterion handoff with pinned inputs, and already unusable on a fifty-criterion one. Agent pilots that stall are stalled by scope rather than by what the model can do. The scope is the variable you control this quarter; the model is not.

Frequently asked questions