An Investigation on Generative-AI and RAG Support to Question Answering on Scientific Papers

Retrieval-Augmented Generation (RAG) has become the prominent architectural framework in order to automate the answer and assessment of open-ended questions (OEQs) in educational environments. However, this process’s effectiveness at each stage of the retrieval process is influenced by various architectural design decisions, such as document segmentation, choice of embedding models, and similarity-based retrieval. This thesis aims to empirically investigate in a controlled procedure, based on the retrieval pipeline stages, the notion of retrieval quality in the context of OEQ-oriented RAG systems. The corpus has been carefully selected and composed of 22 scientific documents and 19 OEQs, with ground-truth relevance labels for each document-question pair, annotated manually. Two chunking strategies were examined: fixed-length chunking and structure-aware chunking, combined with four selected Sentencetransformer embedding models, namely the all-mpnet-base-v2, all-MiniLM-L12 -v2, all-MiniLM-L6-v2, and all-DistilRoBERTa-v1. Precision, recall, and F1- score metrics have been used for Retrieval-performance assessment at a osinesimilarity, similarity of text-question, threshold of 0.5.. The results show that structure-aware chunking is superior to fixed-length chunking. Among the experimented embedding models, all-mpnet-base-v2 performed in a superior way regarding the retrieval results. The findings further highlight that question formulation quality and the presence of bibliographic noise are significant sources of false positives, underscoring the mportance of an adequate pre-processing stage and question designs, especially in automated assessment pipelines.