Operational knowledge is rarely in one place
Engineers and operators do not work from a neat document library. The answer to an operational question may be spread across technical manuals, maintenance records, internal wikis, chat messages, logbooks and live system data. Finding the right evidence quickly affects downtime, consistency and whether knowledge survives when experienced people leave.
A new paper from researchers at Argonne National Laboratory describes a deployed knowledge platform for the Advanced Photon Source, a large scientific facility in the United States. The system gives staff one natural-language interface across nine operational data sources. More importantly, the researchers evaluated which architectural components actually improved grounded answers.
The facility is unusually technical, but the knowledge problem is familiar to manufacturers, engineering firms, logistics operators and business-service organisations: information exists, yet the people doing the work cannot reliably retrieve the right fragment at the right moment.
The strongest result was not the most agentic component
The researchers built a 50-question benchmark with auditable reference answers. A basic keyword-search baseline achieved 63.8% strict recall of vital answer elements. The full system, combining multiple retrieval methods, a knowledge graph and a corrective agent loop, reached 70.3%.
That improvement matters, but the ablation results are more useful for buyers. Removing the dedicated cross-encoder reranker and asking the language model itself to judge passage relevance reduced strict vital recall by 32.8%, according to the paper. By contrast, the knowledge-graph channel and corrective agent loop contributed positively, but their gains were marginal.
In plain language: selecting and ordering the evidence well had more impact than adding another reasoning loop. A fluent model cannot recover facts that the retrieval layer failed to put in front of it.
A knowledge agent is only as good as its evidence pipeline
Many knowledge-agent projects start at the visible end: choose a model, build a chat interface and add an agent framework. The less visible work is more decisive. Which systems are authoritative? How is content kept current? Which source may each user access? How are exact terms, semantic similarity and relationships combined? Which passages deserve to reach the model?
A robust knowledge layer separates these concerns. Data sources supply the raw record. Methods define retrieval, ranking, permissions and freshness. Tools expose controlled access. Governance determines provenance and review. Channels place answers where work happens. Only then should an agent reason over that evidence or prepare an action.
This is not an argument against graphs or corrective agents. They can help on questions that require relationships, decomposition or recovery from weak initial retrieval. It is an argument about sequence: do not add orchestration to compensate for an unmeasured evidence pipeline.
Evaluate the workflow in layers
A practical pilot should test at least four things separately:
- Retrieval: did the system find the passages containing the essential facts?
- Ranking: did the best evidence reach the model before irrelevant or conflicting material?
- Faithfulness: is every material claim supported by retrieved evidence?
- Operational fit: are permissions, freshness, citations and escalation usable in the real workflow?
Only after those layers are measurable should a team test whether query routing, graph traversal or a corrective loop improves a defined failure case enough to justify its cost and maintenance burden.
This also makes model evaluation more honest. A model can appear weak because retrieval supplied the wrong context. Another can sound excellent while inventing unsupported detail. Measuring retrieval and generation separately shows which component needs work and prevents expensive model upgrades from becoming the default answer to every quality problem.
The paper also shows why evidence must remain inspectable
The reported system is promising, not perfect. Citation precision in the benchmark was about 73%, meaning retrieved citations still included a substantial share of material outside the reference evidence pool. The authors also note that the benchmark covers one facility, uses automatically generated questions and relies partly on language-model judging. Human expert validation remains the definitive standard.
Those limitations strengthen the operational lesson. A single aggregate accuracy score is not enough. Users need source links, claim-level traceability and a clear route to human review when the evidence is incomplete or consequential decisions are involved.
Start with one costly knowledge bottleneck
For a company, the useful starting point is not “build an enterprise chatbot.” Choose one recurring question or document-heavy workflow where poor findability creates measurable delay: troubleshooting equipment, checking a dossier, answering a service question, preparing a maintenance decision or locating the latest procedure.
Build a small evaluation set from real questions and approved answers. Measure the baseline. Improve retrieval and ranking. Add citations and access controls. Then decide whether extra agentic behaviour earns its place.
The business outcome is not a more sophisticated diagram. It is less time searching, fewer unsupported answers, faster handovers and better retention of operational knowledge. Complexity is justified only when it moves one of those outcomes.