Because language models are known to hallucinate information, we deliberately tested how different AI chat tools deal with (1) conflicting information and (2) situations where the requested information is not available online. For reliable research results, it is important that contradictions are made explicit and that tools are equally clear when a question cannot be answered.
Case 1: conflicting information is available online
For the first test, we asked for a list of branches belonging to a retail brand in Munich. The company had only one branch in the city centre, which had closed at the end of 2023. However, there was only one online reference to the closure, while most other sources still contained the outdated information that the branch remained open.
ChatGPT initially returned only the outdated information and identified the one link pointing to the closure only after further prompting. Perplexity listed some locations belonging to other affiliated retailers operating under different brand names and, even after follow-up questions, did not arrive at the correct conclusion that the branch no longer existed. Microsoft Copilot gave the correct answer immediately. However, there were problems with source traceability, as the links provided referred to the closure of a branch belonging to another brand within the same group.
Case 2: no information is available online
For the second test, we asked whether companies X and Y work with Content5 or are among our clients. For company X, there were some indications in public sources, but no direct confirmation. For company Y, there was no publicly available information at all.
ChatGPT performed best in this case. It made accurate statements about both companies. For company X, it identified the relevant sources and inferred a business relationship from the wording used in them. Its phrasing was cautious and the answer nuanced. Copilot made an incorrect assumption about company X. Perplexity correctly identified the sources containing indirect evidence and reached the right conclusion, but its answer was less nuanced than ChatGPT’s. Google’s AI mode behaved similarly. For company Y, all models gave the correct answer: there was no evidence of a collaboration or business relationship.
Our conclusion: Manual research remains essential, particularly for questions where conflicting information may exist. None of the AI models pointed out the contradictory information in Case 1 without additional prompting. By contrast, in Case 2, the absence of information about company Y was handled correctly, and none of the models invented a relationship.
By Maria Kleiner
