How well do language models perform when searching for quantitative data relating to future periods?
For the current edition of the newsletter, we tested how AI tools handle quantitative forecasts. Our starting question was how Germany’s unemployment rate might develop over the next 10 to 25 years. We conducted the research using several AI tools, including:
- ChatGPT
- Google AI
- Google AI Overviews
The number of genuinely relevant and robust sources turned out to be very limited. This made it all the more interesting to see how the different AI systems dealt with the challenge.
ChatGPT produced the most useful results. Among other things, it pointed to a European Commission report containing tables with long-term projections for unemployment rates across all EU countries. A comparison with manual research confirmed that this was the best source overall – particularly noteworthy because it had not appeared prominently in a conventional search.
As the second most relevant source, ChatGPT identified a Bundestag document containing specific, albeit more limited, forecast figures. Both sources covered the long-term outlook up to 2070. When we asked again for forecasts covering shorter periods, such as 2035 to 2040, the results were weaker. However, this reflected the limited availability of suitable sources.
Google AI and AI Overviews proved less helpful by comparison. They frequently cited sources that either contained no specific figures for the unemployment rate or were of questionable reliability. In contrast to ChatGPT, they also did not indicate where exactly the relevant information could be found within the documents. This meant that significantly more manual effort was required for further research.
Our conclusion: ChatGPT performed well in identifying relevant sources for this type of quantitative forecasting research. Particularly useful were its prioritisation of sources and its indication of where specific metrics could be found within the respective documents. This made subsequent in-depth research considerably easier. For future-oriented questions, such as market or trend analyses, this can provide tangible added value by identifying sources that may be difficult to discover through manual searching alone.
By Maria Kleiner
