Obduracy: Fine-Tuning at What Cost? Evaluating Specialization Rigidity, Catastrophic Forgetting and Decoding Variability in Large Language Models

Fine-tuning Large Language Models to specialize them for downstream tasks is a standard practice, but it often leads to catastrophic forgetting, a significant degradation of their general knowledge. Existing evaluation methods typically quantify this phenomenon by measuring performance drops on academic benchmarks, an approach that captures the symptoms but not the underlying cause. These metrics often fail to characterize a more fundamental behavioral shift: an increase in specialization rigidity, where the model loses its generative flexibility and becomes overly deterministic. To address this evaluation gap, we introduce Obduracy, a novel composite metric designed to provide a holistic measure of this brittleness. Our approach integrates multiple dimensions of model behavior into a single score. It combines established continual learning metrics that quantify catastrophic forgetting (Backward Transfer) and performance degradation (Forgetting Measure) with a direct measure of behavioral rigidity. This behavioral component, which we term Format Spread, assesses the model's sensitivity to variations in prompt formatting. We apply this composite metric to analyze models fine-tuned with various state-of-the-art strategies, including Supervised Fine-Tuning and preference-based trainers like Generalized Ratio Policy Optimization and Generalized Sigmoid Policy Optimization. We show that a high Obduracy score, indicating high overall brittleness, correlates with more severe performance drops on general knowledge benchmarks. These results confirm that Obduracy offers a more nuanced understanding of the true costs of fine-tuning, providing a valuable tool for researchers to evaluate the multi-faceted impact of different specialization techniques.