Comparing Large Language Models on Unfair Clause Detection in Terms of Services

The detection of unfair clauses in Terms of Service (ToS) is a crucial AI application for consumer protection, enabling users and regulatory bodies to identify potentially harmful contractual terms. While previous research has focused on fine-tuned transformer-based models for this task, the advent of Large Language Models (LLMs) has introduced new possibilities for automated legal text analysis. Unlike traditional models, LLMs can leverage in-context learning, reducing the need for extensive retraining when legislation changes. This study systematically evaluates the effectiveness of LLMs in unfair clause detection by comparing different prompting strategies against fine-tuned classifiers. Through an extensive set of experiments, including 62 model-setting combinations, this work explores various prompting techniques, such as zero-shot, few-shot, multi-prompt and pipeline approaches. The results indicate that while LLMs provide flexibility and adaptability, they still struggle to surpass supervised models in classification performance. Furthermore, this study prioritizes open-source models to ensure accessibility, transparency, and feasibility in low-resource environments. As an additional contribution, a lightweight evaluation framework was developed to facilitate systematic testing of LLMs on the CLAUDETTE dataset.