Proposal for industry RAG evaluation: Generative Universal Evaluation of LLMs and Information retrieval

This thesis reports my internship experience at Bitapp, a software development company located in Bologna. The primary focus of my work involved the design and implementation of a chatbot utilising Retrieval-Augmented Generation (RAG) for an e-learning platform. My main objective was to develop a virtual teacher capable of providing contextually relevant responses to user queries. However, the RAG system comprises a multitude of parameters, such as chunk size and embedding model, which may not be universally applicable across all use cases. Therefore, it is essential to conduct an evaluation of the RAG system in order to identify the most appropriate parameters. In particular, the creation of benchmarks presented a significant challenge, as existing academic benchmarks do not cover the private data involved. To address these issues, a novel approach for generating benchmarks was developed under critical condition to the business. This approach included the evaluation of the retrieval system and an analysis of the relationship between chunk size and embedding models, employing hit rate metrics for assessment. The results indicated that the optimal configuration for managing private business data consisted of a chunk size of 500, utilizing the paraphrase-multilingual-mpnet-based-v2 as the embedding model and gemma2 9b-instruct-q3_K_M as the language model. The work establishes a foundation upon which a framework capable of generating benchmarks on one's own private data may be constructed in the future.