New general publication: ‘RAG for Historians: A Radically New Way of Interacting with Multilingual Collections?’

A new article has been published on LinkedIn.

Retrieval-augmented generation (RAG) is a radically new way of interacting with historical documents. It essentially allows researchers to talk to entire collections, ask questions and get answers grounded in your own data – across languages – without perfect keywords, irrespective of the language of the original records. For work with highly multilingual archives, this is a game-changer – but not without issues.

In this article, I summarize my experiences developing and working with a RAG system for historical research. I document the process of tailoring the RAG app AnythingLLM to the rigors of scientific work and provide pointers and warnings for humanities scholars and GLAM practitioners wanting to experiment with RAG and vector embedding, including strategies for avoiding data loss and hallucinations.