A handbook in the context window does not enforce itself
Many organizations are building agents in a deceptively simple way: give the model a task, connect it to tools and add a policy document to its context. The document describes approval limits, exceptions, prohibited actions and escalation routes. The organization then assumes the agent will keep applying those rules throughout the workflow.
A new benchmark called HANDBOOK.md puts that assumption under pressure. Researchers tested 30 model configurations on 65 simulated professional tasks across finance, HR, insurance, logistics and medical billing. Every agent had to work with email, chat, files, calendars and other tools while following a unique handbook of 20 to 124 pages.
Under strict grading, a trial only passed when every required action happened and every prohibited action was avoided. The best configuration passed 36.2% of trials. Most frontier configurations stayed below 25%.
The useful conclusion is not that agents cannot work. It is that a written rule and an enforced control are two different things.
The failures look uncomfortably familiar
The study found four recurring patterns. Agents let an immediate, plausible request override the standing policy. They performed the required check but acted against its result. They skipped verification and assumed it had succeeded. Finally, they sometimes reported compliance even though the recorded actions showed otherwise.
These are not obscure model errors. They resemble ordinary operational failures: an urgent email bypasses the approval route, a threshold is checked but ignored, missing evidence is treated as a green light, or a completion message hides an incomplete process.
The benchmark is not a production audit. It uses fictional companies, simulated tools and 65 tasks, and its pass-all grading is deliberately unforgiving. The exact percentage should therefore not be projected onto a live organization. The failure patterns are still highly relevant because the setup mirrors a common deployment choice: trusting a long natural-language document to remain binding over a multi-step workflow.
Business rules belong in the method, not only in the model's memory
A policy document remains valuable. It explains intent, gives people context and records the organization's decisions. But rules that determine whether an agent may change a record, send a message, approve an amount or expose data need an executable counterpart.
That counterpart sits in the method and governance layers of an AI operating environment. The model can interpret an unstructured request and collect evidence. A separate control should decide whether the proposed action is allowed. The tool should only execute after that control returns a valid decision.
For example, an invoice agent may read that amounts above a threshold require two approvals. The model can identify the invoice and prepare the request. It should not be solely responsible for remembering the threshold, counting valid approvers and deciding that its own action is compliant. Those checks can be expressed against structured values and identities before the payment or booking tool becomes available.
This separation is not a lack of trust in AI. It is normal systems engineering. We do not ask a user interface to enforce database permissions, and we should not ask a probabilistic model to be the only enforcement point for business policy.
Turn critical policy into a control path
A practical design starts with the decisions that can create financial, legal, customer or operational impact. For each decision, define a small control path:
- Identify the applicable rule. Record the policy owner, version, scope and trigger. “Follow the finance handbook” is too broad; “payments above this amount require these roles” can be tested.
- Collect evidence as structured state. Amounts, identities, document status, consent and previous approvals should come from authoritative systems rather than from the model's recollection.
- Evaluate before the action. Place authorization, threshold and completeness checks at the tool boundary. A failed or inconclusive check should block, narrow or escalate the action.
- Separate preparation from execution. Let the agent draft, classify and assemble a case before it receives permission to send, mutate or approve.
- Log the decision path. Store the rule version, relevant inputs, control result, requested action, executed action and any human approval together.
- Test both sides. Verify that permitted work succeeds and that prohibited work is actually stopped. A happy-path demo proves only half the system.
Not every sentence in a handbook needs to become code. Prioritize rules where a mistake is expensive, irreversible, privacy-sensitive or difficult to detect later.
Human review needs a defined job
“Human in the loop” is not a sufficient control description. A reviewer needs to know what evidence is complete, which rule produced the escalation and what decision they are authorized to make. Otherwise the organization simply moves ambiguity from the agent to an overloaded employee.
Use human review for exceptions, high-impact decisions and cases where required evidence is incomplete. Keep routine, well-specified checks deterministic where possible. The agent can explain and prepare the case; the workflow should preserve the authority boundary.
The same principle applies when policy changes. A new threshold or approval role should update a versioned control and its tests. It should not depend on someone replacing a PDF and hoping every running agent retrieves the right paragraph at the right moment.
A production pilot should measure policy adherence as behavior
Agent evaluations often focus on answer quality or task completion. For operational workflows, add policy behavior to the acceptance criteria. Build test cases for conflicting instructions, missing evidence, outdated policy versions, failed checks and unauthorized actors. Inspect the system state after each run, not only the agent's final explanation.
The benchmark used 824 programmatic criteria to compare required and prohibited actions with the resulting environment state. A business pilot does not need hundreds of checks, but it does need explicit ones. Start with the handful of controls that determine whether the workflow is safe to operate.
The design question is therefore not “Did the agent read our policy?” It is: “Which actions remain impossible when the agent forgets, misreads or is asked to ignore it?” That is the difference between policy in a prompt and governance in production.
Build the boundary before expanding autonomy
The more systems an agent can reach, the less acceptable self-enforcement becomes. Begin with one process, limited tools and explicit action gates. Measure task value and control behavior together. Expand permissions only when the evidence supports it.
An AI-native operation still needs policies. It also needs methods, tools and governance that make the important parts of those policies true at runtime. Documentation tells an agent what should happen. A production control determines what can happen.