109 DESIDOC Journal of Library & Information Technology, Vol. 45, No. 2, March 2025, pp.109-115, DOI : 10.14429/djlit.20206  2025, DESIDOC Received : 11 May 2024, Revised : 28 September 2024 Accepted : 12 December 2024, Online published : 27 February 2025 Designing Conversational Search for Libraries: Retrieval Augmented Generation through Open Source Large Language Models Parthasarathi Mukhopadhyay Department of Library and Information Science, Kalyani University, Kalyani - 741 235 West Bengal, India Email: psm@klyuniv.ac.in ABSTRACT Large language models (LLMs) from the commercial domain like BERT and GPT have made machine learning technologies accessible to everyone. On the other hand, the open-source LLMs like Llama, Mistral, and Orca are equally effective and are now widely available. Librarians and information professionals around the world are exploring how to use these models to improve library systems, particularly in the area of searching and finding information, and in building question-answer based search systems. This research study aims to use open-source large language models to develop a conversational search system that can answer questions in natural language on the basis of a given set of documents. The system is based on a Retrieval Augmented Generation (RAG) pipeline, which helps to overcome two major issues with large language models: providing false or imaginary information (hallucination) and giving outdated or unrelated answers. Through two case studies, this research demonstrates that using a RAG-based approach can effectively address these issues and provide more accurate and relevant results. The study proves that an open-source RAG framework can be used to incorporate large language models into library search systems. This integration allows users to receive direct answers to their questions, rather than just a list of potentially relevant documents. In the coming future, the conversational search system can be designed to work in Indian languages, allowing users to ask questions and receive answers in their preferred language. Keywords: RAG (Retrieval Augmented Generation); LLM (Large Language Model); Generative AI; Conversational search; Library retrieval 1. INTRODUCTION Large Language Models (LLMs) from commercial providers (Anthropic [Claude], Google [Gemini], and OpenAI[GPT]and so on..) – all are very costly for use programmatically for large- scale projects) as well as from open-source domains (Llama series, Orca, Mistral, etc.) have made significant advancements in generating human-like text but still face challenges such as outdated information and hallucinations (feature of LLMs to produce coherent and grammatically correct text but factually incorrect or ludicrous). To address these issues, two main approaches are employed: “fine-tuning” and “Retrieval-Augmented Generation (RAG).” Fine-tuning involves re-training LLMs to enhance their understanding of specific topics, but at a high cost in terms of resources and expertise 1 . On the other hand, RAG uses relevant content (retrieved through semantic matching against a query) to improve response accuracy without extensive training 2 . While RAG excels at quickly generating reliable answers based on provided dataset 3 , fine-tuning is more suitable for specialised tasks and creative writing 4 , despite potential transparency and accuracy measurement concerns 5 . In the given context, this research aims to investigate the potential of an open-source RAG pipeline in libraries for developing a conversational search system that delivers accurate and relevant answers to user queries, thereby addressing the limitations inherent in large language models. Historically, library professionals have been early adopters of technological innovations right from the 1970s. However, the widespread adoption of LLM technologies in libraries has been hindered by the tendency of these models to generate responses that are often hallucinated, outdated, or out of context. By examining the feasibility of an open-source RAG pipeline, this research seeks to bridge the gap between the eagerness to embrace new technologies and the practical challenges associated with deploying LLMs in a library setting. 2. RAG IN LIBRARIES The introduction of LLMs like ChatGPT in November 2022 has sparked interest in generative AI within the library community. Phil Bradley 6 and Kent Fitch 7 foresee the possible integration of conversational AI models like ChatGPT in library search systems, marking a new era in information retrieval with both opportunities and challenges for professionals and users. Kent Fitch developed a prototype search interface in 2023 as a proof-of-concept for transforming library search systems using LLMs. The prototype focuses on improving keyword indexing and text retrieval and implementing summarisation and a “chat” interface to enhance the user experience, proving that RAG has the potential to revolutionise library services with the assistance of LLMs. It could enhance information retrieval