Leveraging Fine-Tuned LLMs for Policy Reasoning in Identity and Access Management

Identity and Access Management (IAM) systems are critical for enforcing access control and compliance in modern enterprises, yet their interfaces are often excessively complex. To simplify interaction, natural language reasoning layers are increasingly used, but they typically rely on commercial APIs, creating dependencies that limit transparency, control, and adaptability. This thesis explores the viability of using task-adapted open-source language models for reasoning over IAM policies as an alternative to API-based solutions. It presents three core contributions: an end-to-end synthetic data generation pipeline with semi-automatic quality assurance ensuring logical consistency; systematic fine-tuning and performance benchmarking across the Qwen3 model family (0.6B–32B parameters); and a deployment study assessing the operational trade-offs of self-hosted inference. Results reveal complex scaling dynamics, where mid-sized models (4B–8B) achieve substantial accuracy gains from fine-tuning—often reaching peak performance with only fractions of the training dataset—while larger models (14B–32B) exhibit diminishing returns, suggesting that further performance gains may depend on more diverse and complex data rather than model scale alone. Benchmark comparisons indicate that open-weight models perform competitively on standard queries but lag behind commercial APIs on complex, multi-user reasoning. Fine-tuning improves response clarity and conciseness across models of all sizes, though it can reduce performance on tasks outside the target domain. Although self-hosting enhances control and data governance, it incurs up to 16 times higher latency and roughly 9 times greater cost than API-based systems. These findings highlight both the potential and limitations of open-source language models in enterprise IAM, outlining a clear path toward reducing vendor dependency while maintaining accuracy and governance integrity.