Neural-Symbolic Learning: challenges and benchmarks

Building systems capable of integrating learning, reasoning and optimisation is one of the main goals of the field of artificial intelligence, and in particular of its dedicated sub-field now known as Neural-Symbolic Artificial Intelligence (NeSy AI). However, the evaluation of new ideas in this context poses a significant challenge due to the lack of adequate benchmarks. This severely limits the overall progress of NeSy AI. Now is the time to consider new challenges, which can guide the development of new hybrid systems. This thesis analyses the current state-of-the-art (SOTA) regarding existing datasets and corpora, studying their limitations and analysing existing systems that have been applied to such data. It then provides a list of desiderata that new benchmarks for NeSy AI systems should include in terms of tasks, data, metrics, dimensions, domains and proposes new ideas for evaluating and comparing different approaches. As a result of the analysis, Visual Nonogram is proposed as a new challenge to test the capabilities and limitations of existing learning and reasoning systems. We provide a proof-of-concept demonstration of the benchmark by testing two state-of-the-art NeSy AI systems (DeepProbLog and NeurASP). Our experimental analysis suggests that the benchmark is a suitable candidate for future development of NeSy AI solutions. Indeed, we observe that the tested systems are not even able to solve the simplest version of the task, thereby highlighting the need for new ways to integrate neural and symbolic approaches. Additionally, the benchmark includes a curriculum of tasks aiming at testing the scalability of future systems. Finally, we believe that the modularity and simplicity criteria used to design this benchmark are key assets for future extensions and the development of new tasks.