Limits of AI chatbots in structured bulk data processing

—

© 𝘈𝘯𝘥𝘳𝘦𝘢𝘴 𝘖𝘩𝘳𝘦𝘯𝘴𝘤𝘩𝘢𝘭𝘭

How well suited are language models to the automated processing of large datasets, for example matching, cleaning or enriching hundreds or thousands of entries with additional data? We wanted to find out whether LLMs could be used alongside paid web-crawling software to retrieve larger volumes of information from publicly available sources. With specialised AI and software solutions now available for almost every use case, it makes sense to assess whether an existing LLM can take on some of these tasks and thereby reduce complexity, costs and the number of tools required.

In practice, the AI chatbots we tested quickly ran into fundamental limitations. A table containing several hundred or even thousands of rows, whether listing companies, products or addresses, is not always handled by the model as a structured matrix but can instead be processed more like a continuous stream of text. Precise row-by-row matching can then break down, increasing the risk of mix-ups and omissions.

We tested this use case by asking ChatGPT, Perplexity and Claude to find relevant additional information, such as web addresses or categories, for entries in a list. We explicitly instructed the models to rely only on publicly available data. The results were disappointing: not only were some of the research results incorrect or hallucinated, but the technical handling of the data was also problematic.

When we tried to process an Excel list containing around 2,000 rows, the language models indicated that a list of this size could not be handled in full. One suggested workaround was to process the list in batches of ten, requiring the user to enter “Continue” or a similar instruction after each batch of research results before ChatGPT or Perplexity would proceed. Claude behaved in a similar way. We also found that the information returned in this process tended to be incomplete.

Adjusting the prompts did not overcome this technical limitation. Instead, the chatbots repeatedly suggested using other solutions such as web crawlers, which are specifically designed for this kind of task.

Our conclusion: AI chatbots are well suited to exploratory analysis and text-based tasks, but they quickly reach structural limits when it comes to processing large volumes of structured data.

By Frederic Simon