Practicality of Copilot agents in OneDrive

—

coffemug wiht black coffee and with ai on a desk with d5eae1fa 31bc 44b8 b621 377d1b7f1da0 3

How can we extract information from a collection of documents we have gathered as quickly and comprehensively as possible? This question arose during a research project in which we searched and analysed several hundred PDF documents.

Microsoft Copilot offers an AI agent that can access a defined set of sources and be customised with specific instructions depending on the intended purpose. These can include seemingly simple instructions such as “If you cannot find any information, please do not make anything up” to reduce the risk of hallucinations.

In practice, repeated searches produced different results depending on how precisely the agent had been configured or briefed beforehand. One drawback was that, as is generally the case with LLMs, the same prompt could produce different results on different days, making reproducibility more difficult. On the positive side, however, the agent made it possible to carry out much more complex and content-specific queries with relatively little effort compared with conventional search functions in SharePoint or File Explorer.

In our case, we first asked the agent to list all articles dealing with the topic in question. In the first attempt, it only returned documents in which the exact term appeared. We then added an instruction telling the agent to automatically consider synonyms and thematically related terms from the same field. The agent was able to follow this instruction and returned additional relevant documents.

More in-depth analysis was also possible, for example compiling information on manufacturers and the characteristics of different variants of the product being researched. The agent was able to extract this information from all of the documents, combine it with source references and present it clearly. It was also helpful that the agent sometimes proactively suggested ways of presenting the data, such as in tables or visualisations.

Our conclusion: Despite the agent’s intelligent adaptation mechanisms and ability to anticipate aspects of the task, successful use still depends heavily on the quality of the prompt. Copilot agents should therefore always be tailored specifically to the relevant use case. Their limited reproducibility also restricts where they can be used effectively, and users should remain aware that results may vary.

By Frederic Simon