Large Language Model Understanding of Graph Structures via Graph Embeddings

Large Language Models (LLMs) have demonstrated remarkable success in processing textual data and have been extended to handle multimodal inputs such as images, audio, and video. However, understanding graph-structured data remains a significant challenge due to the non-Euclidean nature of graphs and the difficulty in encoding their complex relationships in a format suitable for LLMs. This thesis explores the integration of graph-structured data with LLMs, focusing on the development of graph representations that enable LLMs to understand graph structure effectively. We propose the use of Graph Projectors — neural networks trained to project graphs into sequences of embeddings compatible with LLM input spaces — and compare different architectures, including Graph Neural Networks (GNNs) and simpler Multi-Layer Perceptrons (MLPs), for this purpose. To evaluate these methods, we conduct experiments on two datasets: the GraphQA benchmark and a synthetic dataset designed to mimic real-world complex networks. We assess the performance of LLMs on structural and metric-based graph tasks, comparing results across different graph representations. Additionally, we investigate the interpretability of these learned embeddings and analyze the extent to which they capture graph structure. Our findings demonstrate that LLMs, when provided with suitable graph representations, can effectively answer graph-related queries. Notably, simpler representations such as edge and node sequences offer competitive performance compared to more complex GNN-based approaches, suggesting that LLMs may not require sophisticated graph encoders to understand graph structure. This opens promising avenues for leveraging LLMs in graph-related tasks while reducing architectural complexity.