Personality Alignment in Large Language Models: Analysis and Fine-Tuning through Preference Optimization

As Large Language Models (LLMs) become increasingly embedded in interactive applications, the ability to personalize their behavior—particularly through personality traits—has emerged as a key goal. Personality shaping can enhance user engagement, trust, and effectiveness, but current methods often rely on large-scale models and manually labeled data, limiting scalability and accessibility. This work presents a fully automated pipeline for aligning small-scale Large Language Models with the Big Five personality traits without relying on human-labeled data. The proposed method leverages LLMs themselves at every stage of the process: generating personality-variant responses, evaluating those responses through preference ratings, training a reward model, and fine-tuning the base model via Supervised Fine-Tuning and Direct Preference Optimization. Special attention is given to addressing challenges specific to smaller models, including prompt-level positional biases—mitigated through systematic inversion of input order—and limited representational capacity, addressed by decomposing complex tasks into simpler sub-tasks across the pipeline. To reduce bias in the evaluation phase, rating tasks are reframed as binary preference comparisons rather than Likert-scale scoring. While the effectiveness of the alignment remains difficult to conclusively measure, the fine-tuned models exhibit consistent patterns in self-assessed personality expression over time. The findings suggest that personality shaping is possible even in low-resource, fully autonomous settings, though the work also highlights the need for improved evaluation methods—such as testing the aligned LLM on specific benchmarks or conducting human-centered assessments—to better capture the nuances of personality in LLM behavior.