Investigating Contextual Memorization in Large Language Models through Prompting Techniques

Large Language Models are central to advancements in Natural Language Processing and human-computer interaction, yet their increasing capabilities raise ethical concerns, such as memorization of sensitive or copyrighted information. This thesis introduces the concept of contextual memorization, exploring how models store context-specific data. Using poems by renowned authors as a case study, the extent of memorization was analyzed based on elements like author and title. The study employs techniques such as Prompt Tuning, applied to both Supervised Fine-Tuning and Reinforcement Learning, to create soft prompts that enhance the retrieval of memorized content. Results demonstrate that soft prompts effectively increase reproduction rates, highlighting implications for model training and ethical considerations.