Distillation Knowledge applied on Pegasus for Summarization

In the scope of Natural Language Processing one of the most intricate tasks is Text Summarization, in human terms: writing an essay. Something that we learn in primary school is yet very difficult to reproduce for a machine, it was almost impossible before the advent of Deep Learning. The trending technology to face up Summarization - and every task that involves generating text - is the Transformer. This thesis aims to experiment what entails reducing the complexity of Pegasus, a huge state-of-the-art model based on Transformers. Through a technique called Knowledge Distillation the original model can be compressed in a smaller one transferring the knowledge, but without losing much efficiency. For the experimentation part the distilled replicas were varied in size and their performance assessed evaluating some suitable metrics. Reducing the computational power needed by the models is crucial to deploy such technologies in devices with poor capabilities and a not reliable enough internet connection to use cloud computing, like mobile devices.