Technical success can hide operational failure
An AI workflow returns valid JSON. Every action block exists. The interface renders correctly. Validation passes and the run ends with a success status. Technically, everything looks healthy.
But did the workflow actually do what the user asked?
A new study by IBM researchers shows how far those two answers can diverge. The researchers evaluated six language models in a production low-code automation platform across 29 real-world IT and compliance scenarios. They ran each prompt, model and pipeline combination eight times, for 2,784 runs in total.
The team first improved the generation architecture. Instead of asking a model to produce a complete workflow in one pass, it split construction into variables, top-level blocks and nested blocks. Structural success — valid JSON, valid references and correct rendering — rose from a range of 31.5–82.8% to 74.1–97.8% across the tested models.
That is a useful engineering result. It also exposed a more important warning: a workflow can be structurally perfect and operationally useless.
The best technical score produced the worst task result
The researchers separately tested whether structurally valid workflows fulfilled the original request. The model with the highest structural success, at 97.8%, had the lowest task-satisfaction rate in the redesigned pipeline: 6.9%.
Its recurring failure was revealing. The model generated valid but non-executable stub workflows that stated that the required actions were unavailable. Those workflows passed schema validation because their structure was correct. They failed the user because they did not perform the requested work.
Even the strongest model on task satisfaction passed only 55.2% of prompts under the study's majority-vote method. The paper therefore recommends human review before execution.
The exact percentages should not be generalized to every business process. All authors work at IBM, the scenarios focus on IT and compliance automation, semantic quality was graded by another language model rather than a human gold standard, and the paper does not yet report confidence intervals per test cell. But the failure pattern is broadly relevant: technical validity is necessary, not sufficient.
Independent research from Princeton reaches the same broader conclusion from a different direction. Its evaluation of 14 models argues that agent reliability cannot be inferred from one average success score. Consistency across repeated runs, robustness to changed inputs, predictability of failures and the severity of those failures must be measured separately.
Accept the outcome, not the agent's story about the outcome
Many AI acceptance tests stop too early. They confirm that an answer has the expected format, a tool call returned without an error or the agent reported completion. Those checks matter, but they only prove that the software path ran.
For a document workflow, success is not “the extraction returned JSON.” It is that the right values were extracted, unsupported values were rejected, exceptions were routed and the target record contains the correct data. For a service workflow, success is not “a reply was drafted.” It is that the reply used the right case context, stayed within policy and reached the appropriate reviewer. For an operational agent, success is not “the update call returned 200.” It is that the intended system state changed once, in the right account, with an auditable reason.
The do-nothing stub is only the cleanest example. More dangerous failures can look equally valid: selecting the wrong customer, skipping an approval, applying an outdated rule, updating two records instead of one or claiming completion after a partial action.
A practical acceptance set for AI workflows
A production workflow needs a small, explicit acceptance set before it receives meaningful permissions. At minimum, test six things:
- Outcome: define the observable business result for representative cases, not just the expected output format.
- Resulting state: verify the source and target systems after execution. Do not accept the agent's own completion message as proof.
- Forbidden outcomes: test what must never happen, including wrong recipients, duplicate updates, unauthorized actions and fabricated evidence.
- Repeated execution: run important cases more than once and vary wording, ordering and missing information. A workflow that works once is a demo, not an operating capability.
- Failure handling: check whether the workflow abstains, escalates or stops safely when confidence, permissions or evidence are insufficient.
- Regression: turn every defect and near miss into a test that the next model, prompt or integration version must pass.
Cost and latency belong beside these quality checks. Decomposing a difficult task can make smaller models structurally viable, but more steps also add calls, tokens and failure points. The relevant metric is therefore not cost per model response. It is cost per correctly completed, safely handled business case.
This changes how an AI project should go live
A model comparison is not an acceptance test. Neither is a polished prototype. The acceptance boundary belongs around the complete workflow: source data, reasoning, permissions, tools, human decisions and the resulting system state.
That is why a controlled launch should begin with one bounded workflow and an agreed acceptance set. Before go-live, the team needs representative cases, explicit stop conditions, a reviewer for exceptions and a way to observe actual outcomes. After go-live, production findings must feed evaluations and regression tests.
The practical conclusion is simple: do not ask only whether the AI produced something valid. Ask whether the business process reached the right state, whether it did so safely and whether it will do so again tomorrow.
Sources: Evaluating LLM Trade-offs for Enterprise Automation: Lessons from Workflow Generation in a Production Enterprise Platform and Towards a Science of AI Agent Reliability.