The latest system state is not always the truth
Imagine that an incident lead asks an AI agent for an update at 19:05. At that moment, the incident has been acknowledged, the timeline contains three events and the root cause is still unknown. By 20:55, the incident is resolved, the timeline contains ten events and the root cause has been documented.
If the acceptance test uses only the final system state, it produces a strange result. The agent that correctly says at 19:05 that the incident is still in progress fails. The agent that answers with information from the future appears to perform well.
That example comes from new research into time-aware evaluation of enterprise AI agents. Its central point extends far beyond incident management: an answer is only correct relative to what existed at that moment and what that particular user was allowed to see. A current database snapshot is therefore not automatically valid ground truth for a historical or ongoing workflow.
Future information also lives inside existing records
This cannot be fixed by simply filtering out records with a later creation date. Business records evolve internally. A ticket gains comments, an order changes status, a case receives a decision, an email thread grows and a maintenance report later receives a cause and closure date.
A record may already have existed at 19:05 while the version in the test environment contains information added hours or days later. Without point-in-time reconstruction, the test does not establish whether the agent could have answered correctly then. It only tests whether the agent can repeat the latest version now.
The same distinction applies to actions. An agent that correctly refuses to release an order at 14:00 because a check is missing should not fail because that check was completed at 16:00. Otherwise, evaluation rewards precisely the wrong behaviour: acting on knowledge that was unavailable during the real execution.
Permissions belong in the expected answer
Time is only half the problem. Two employees can have different valid views of the world at the same moment. A service employee may see a customer case and public incident status, while an engineer can also access internal error logs and security details.
An answer can be factually true and still unacceptable if the agent used a source that the person asking the question was not permitted to see. An acceptance scenario therefore needs more than a question and reference answer. At minimum, it needs a persona, timestamp, visible sources and permitted actions.
This makes permissions part of the test data rather than a separate security check afterwards. It also exposes an important distinction: an agent can have insufficient information without being broken. The correct response may be to say that evidence is missing or escalation is required. Pretending to be certain is the failure.
A useful acceptance set needs a scenario clock
For document-heavy workflows spanning several systems, an acceptance set must do more than replay isolated prompts. Build scenarios around real process events: a request arrives, a check is added, someone updates a case, an exception is approved and the target system changes state.
For each relevant moment, record:
- which records existed and which version of each record was valid;
- which employee or role asked the question;
- which sources and tools that role could access;
- which facts were still unknown or unconfirmed;
- which action, system change or escalation counted as success;
- which information the agent was explicitly prohibited from using.
Then test more than the wording of the answer. Inspect the sources used, tool calls made, approval gates passed and final state in the CRM, ERP, ticketing system or case file. An agent that writes a convincing completion message but updates the wrong record has not succeeded operationally.
Make the test reproducible before comparing models
The paper describes an approach that reconstructs changing records in advance for relevant points in time. Mechanical changes — removing later comments, blanking future timestamped fields and trimming growing lists — are handled with deterministic rules. A language model is used only where interpretation is required, such as free text or a derived lifecycle status. Those outputs are precomputed, so no additional model can make the score fluctuate during the actual evaluation.
The authors report that one authored episode produced dozens of per-person, point-in-time evaluations, onboarding a new application fell from days to minutes and batch runs were exactly reproducible. These are early experiences with the proposed architecture, not an independent agent comparison or proof that every reconstructed historical state is perfect. The authors identify validation of that reconstructed past as an open question.
The design rule is still immediately useful: stabilise the test world first, then compare models, prompts or agent architectures. If the expected reality changes between runs, you cannot tell whether the agent improved or the measuring stick merely moved.
The go-live question is not only: does it work now?
A good first workflow has an owner, a bounded outcome and an acceptance set. For processes with changing data, that adds another question: does the agent work at every decision point using the information and authority valid at that time?
Acceptance should therefore include scenarios from the middle of the process, not just tidy end states. Show what the agent does before information is complete, after a status changes, when two roles have different access and when a previously correct answer becomes outdated. Turn every discovered failure into a regression test.
A static snapshot proves that an agent can handle one frozen world. An operational workflow demands more: correct action while that world changes.
Sources: What Could the Agent See at 19:05? (Sahu and Arora, 2026); Reproducible Hybrid Time-Travel Retrieval in Evolving Corpora (Staudinger, Piroi and Rauber, 2024).