Full AI automation is often the wrong starting point
Many AI plans begin with a question that is too broad: can we automate this entire process? That sends simple cases through unnecessarily expensive processing and floods human reviewers with work that conventional software could already handle reliably.
A more useful design question is: which part of the workload actually needs AI? Most case-based workflows contain three groups. Routine cases follow clear rules. Ambiguous cases contain enough context for a model but too much variation for simple software. True exceptions require human judgement, additional information or a decision from the process owner.
Value does not come from making one group as large as possible. It comes from routing each case to the least expensive reliable path.
A production case makes that allocation measurable
A new paper by Pinterest researchers describes a large-scale visual content-quality workflow. Existing lightweight models remained the first filter. A vision-language model was used only for the grey-zone cases left unresolved afterwards. Its output was not an autonomous final decision but a score used by the workflow to determine what could pass automatically and what needed human review.
After a three-week shadow deployment, the route was promoted to production. Within the grey-zone workload studied, the auto-pass rate increased from 20.48% to 41.99%, while the estimated false-negative rate fell slightly from 13.62% to 13.18%. The paper reports a 25.7% improvement in review productivity, a 16.2% reduction in normalized operating cost and a move from next-day to same-day final signal delivery.
Those figures come from one internal content workflow and cannot simply be transferred to logistics, manufacturing or business services. The pattern is broadly useful: let inexpensive, predictable components handle the easy work and reserve more capable AI for cases where it demonstrably adds value.
Design the allocation of work, not just the model
Model selection is only one part of this approach. The core is an explicit routing contract between software, AI and people.
- Routine: existing rules, search functions or specialised models handle cases that fall comfortably within known boundaries.
- Grey zone: AI combines context, recognises nuance and produces an outcome or supported proposal.
- Exception: a person receives the case, relevant context and the reason for escalation.
That contract should reflect the cost of errors in the process. A lower threshold may be acceptable for an internal classification. A payment, contract change or safety decision needs a different boundary. A language model's raw self-reported confidence is not enough; routing must be calibrated against real process outcomes.
Research on selective prediction and human collaboration also shows that deferral only works when the person receiving the case can actually assess it well. Escalation must not become a dumping ground for everything the model finds difficult.
Human capacity is part of the architecture
A workflow can route cases correctly and still stall if every difficult case lands with one specialist. Before building, establish how many cases the team can review each day, which information each handoff requires and how quickly a decision must be made.
The reason for escalation must be visible too. Without case context, relevant sources and a clear decision request, the reviewer has to repeat the investigation. AI then merely moves the queue.
The best route therefore does not minimise human activity at any cost. It maximises reliable flow: routine work without unnecessary friction, AI where variation justifies it and human attention where consequences or uncertainty demand it.
Measure the allocation as one operational system
A standalone model score says little about business value. Measure at least five outcomes together:
- the share of cases processed by each route;
- the number of relevant cases incorrectly passed automatically;
- the amount of work reviewers complete correctly per hour or day;
- the cost per completed case;
- the total time from arrival to usable outcome.
There is a blind spot to manage: automatically passed cases often receive no human review and therefore produce less feedback. The production case estimates its miss rate using audit samples and a correction for selective observation. Business workflows need periodic checks of the easy route too; otherwise the dashboard learns mainly from problems that were already visible.
Start with the grey zone
For a first operational AI workflow, the grey zone is often a better scope than the entire process. Choose one workload, map the current decision rules and exceptions, and measure how many cases can already be handled without specialist judgement. Then let AI compete only for the remaining time-consuming middle.
That makes the business case concrete. Not: how many tasks could the model theoretically replace? But: how much reliable capacity can we add without moving errors, costs or queues elsewhere?
That is how AI starts doing real work in operations: not as an expensive universal answer, but as one purposeful layer in a well-designed allocation of work.