READY makes human review a measured cost, not a safety slogan
"A human checks the important cases" sounds reassuring. Then someone has to decide which cases are important, how many people must check them, and how often those reviewers are wrong.
Most agent evaluations end before that work begins. They give an agent a task set, score the answers, and report one accuracy number. A new open framework called READY begins with the operator's decision instead: can this agent reach the reliability required for one specific workflow without demanding more human review than the organization can provide?
In READY's clinical-audit case study, two systems had almost identical autonomous accuracy, 72.8% and 72.5%. To qualify at the same 76% reliability target, they needed human review on 39.2% and 29.6% of cases respectively. An ordinary score made them look tied. The operating plans differed by nearly a tenth of the workload.
The reviewer belongs inside the evaluation
READY stands for Reliable Enterprise Agent Deployment. It takes an agent, a workflow, and a set of candidate oversight policies. Each workflow keeps its own definition of a correct result. The framework finds the lowest-cost review policy that can meet the chosen reliability target, freezes that policy, and tests it on held-out cases.
The output is a deployment profile, not a general leaderboard position. It records the agent, workflow, reliability target, review burden, and operating cost supported by the test.
The paper divides the procedure into evaluation, policy optimization, and statistical qualification. First, the agent completes cases and reports confidence. Policy optimization decides which results can pass automatically and which should go to a person. Qualification applies the selected threshold to unseen cases.
Held-out qualification matters because threshold tuning can overfit. If a team adjusts escalation until one test set looks safe, then cites the same set as proof, it has measured how well the policy fits known examples. New cases provide the actual check.
A correct answer and a useful confidence signal are separate products
One agent may fail while expressing low confidence, giving the system a chance to escalate. Another may be confidently wrong and send the bad answer down the automatic path.
Across the 16 systems in READY's case study, autonomous accuracy and routing quality were essentially uncorrelated. The reported Pearson correlation was -0.12. Qwen-3.7-Plus shows the split clearly: it had one of the lower autonomous accuracies and the strongest routing signal among the tested systems.
That result does not establish Qwen-3.7-Plus as the best overall system. It shows that a deployment can gain value from knowing when a model needs help. A slightly more accurate model may cost more to operate if its mistakes are difficult to identify before they reach a user.
At the 76% target, the system with 72.8% autonomous accuracy needed review on 39.2% of cases. The system at 72.5% needed 29.6%. Procurement tables that stop at autonomous accuracy omit a substantial staffing difference.
Reviewers create a reliability ceiling
The clinical analysis assumes that a human reviewer succeeds on 90% of referred cases. Under that assumption, no oversight policy can exceed 90% reliability. As the target approaches roughly 85%, every tested system moves toward full human review.
The arithmetic corrects vague promises about keeping a person "in the loop." Review cannot guarantee perfection when reviewers have limited time, incomplete evidence, uneven expertise, or an interface that encourages approval of the agent's suggestion.
The 90% figure is a case-study assumption, not a universal estimate of clinical reviewer performance. A deployment has to measure its own reviewers on representative cases. Disagreement resolution, training, fatigue, and review time also belong in the cost model.
The READY benchmark site defines total completion cost as model tokens plus the human review needed to reach the target. Salary is only part of that cost. Queues introduce delay, and scarce experts may be pulled away from cases that require their full attention. Teams should report review in money, minutes, and capacity.
Design escalation before choosing the model
Teams can use the method without adopting every component of READY. They need to define the operating policy before treating an accuracy score as a deployment decision.
Five decisions make a useful starting point:
- Specify the workflow and the harm caused by a wrong result. A support refund, medical audit, and production deployment need different error budgets.
- Set the reliability target and say whether it applies per case, per customer, or across a time window.
- Define what reviewers can see, edit, reject, or return for more evidence.
- Measure reviewer accuracy and time on the same case distribution used for the agent trial.
- Freeze the escalation policy, then qualify it on unseen cases with confidence intervals.
READY's open testbed runs on the UK AI Security Institute's Inspect AI framework. Inspect represents evaluations as datasets, solvers, and scorers, with support for external agents and approval policies. Its approval documentation notes that asking a person to approve every tool call is tedious, so policies can route only selected calls for approval. Tool approval and final-output review address different risks. Both consume human time.
When several model APIs are under consideration, api.ish.chat can route a fixed case set through them while preserving the workflow and scoring rules. ish.chat can help reviewers compare how models express uncertainty and request escalation. Keep the harness, prompt, tools, and review interface fixed so model behavior is not confused with changes in the test.
BLOGish's article on CollabSkill explains why a human-agent score should not become an employee ranking. READY has a related boundary: its deployment profile is specific to one workflow and set of assumptions. It does not automatically transfer to a different organization, case mix, reviewer pool, or model version. The launch covers two healthcare benchmarks, with more industries planned.
The method is broader than the first result
The current case study covers 16 agent systems and 750 clinical-audit cases. Healthcare offers consequential decisions and structured review, but it does not stand in for software maintenance, finance, customer support, or education. An open testbed helps with reproduction. Every new workflow still needs domain experts, representative cases, a defensible scorer, and independent qualification data.
Before deployment, the record should show the frozen escalation threshold, held-out reliability interval, share of cases sent to people, reviewer accuracy and time, model cost, and the case distribution used for qualification. Without that record, "human oversight" describes an intention rather than an operating policy.
