Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs

Ao, S, Dong, Y ORCID: 0000-0003-3047-7777, Hu, J ORCID: 0009-0008-5261-211X and Ramchurn, SD (2025) Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs Transactions of the Association for Computational Linguistics, 13. pp. 1474-1487. ISSN 2307-387X, 2307-387X

Access the full-text of this item by clicking on the Open Access link.

Abstract

Fine-tuning Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) enhances adaptability while reducing computational costs. However, fine-tuning can compromise safety alignment, even with benign data, increasing susceptibility to harmful outputs. Existing safety alignment methods struggle to capture complex parameter shifts, leading to suboptimal safety-utility trade-offs. To address this issue, we propose Safe Pruning LoRA (SPLoRA), a novel pruning-based approach that selectively removes LoRA layers that weaken safety alignment, improving safety while preserving performance. At its core, we introduce Empirical-DIEM (E-DIEM), a dimension-insensitive similarity metric that effectively detects safety misalignment in LoRA-adapted models. We conduct extensive experiments on LLMs fine-tuned with mixed of benign and malicious data, and purely benign datasets, evaluating SPLoRA across utility, safety, and reliability metrics. Results demonstrate that SPLoRA outperforms state-of-the-art safety alignment techniques, significantly reducing safety risks while maintaining or improving model performance and reliability. Additionally, SPLoRA reduces inference overhead, making it a scalable and efficient solution for deploying safer and more reliable LLMs. The code is available at https://github.com/AoShuang92/SPLoRA.

Item Type:	Article
Uncontrolled Keywords:	4606 Distributed Computing and Systems Software, 46 Information and Computing Sciences, Bioengineering
Divisions:	Faculty of Science & Engineering Faculty of Science & Engineering > School of Computer Science & Informatics Faculty of Science & Engineering > School of Computer Science & Informatics > Artificial Intelligence
Depositing User:	Symplectic Admin
Date Deposited:	01 Dec 2025 09:55
Last Modified:	23 Jan 2026 04:11
DOI:	10.1162/TACL.a.44
Open Access URL:	https://doi.org/10.1162/tacl.a.44
Related Websites:	Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3195745
Disclaimer:	The University of Liverpool is not responsible for content contained on other websites from links within repository metadata. Please contact us if you notice anything that appears incorrect or inappropriate.