Clinical Prompt Learning With Frozen Language Models

Taylor, Niall, Zhang, Yi, Joyce, Dan W ORCID: 0000-0002-9433-5340, Gao, Ziming, Kormilitzin, Andrey and Nevado-Holgado, Alejo (2023) Clinical Prompt Learning With Frozen Language Models. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, PP (99). pp. 1-11.

Text
ieee_2022_clinical_prompt_learning.pdf - Author Accepted Manuscript
Download (681kB) | Preview

Official URL: http://dx.doi.org/10.1109/tnnls.2023.3294633

Abstract

When the first transformer-based language models were published in the late 2010s, pretraining with general text and then fine-tuning the model on a task-specific dataset often achieved the state-of-the-art performance. However, more recent work suggests that for some tasks, directly prompting the pretrained model matches or surpasses fine-tuning in performance with few or no model parameter updates required. The use of prompts with language models for natural language processing (NLP) tasks is known as prompt learning. We investigated the viability of prompt learning on clinically meaningful decision tasks and directly compared this with more traditional fine-tuning methods. Results show that prompt learning methods were able to match or surpass the performance of traditional fine-tuning with up to 1000 times fewer trainable parameters, less training time, less training data, and lower computation resource requirements. We argue that these characteristics make prompt learning a very desirable alternative to traditional fine-tuning for clinical tasks, where the computational resources of public health providers are limited, and where data can often not be made available or not be used for fine-tuning due to patient privacy concerns. The complementary code to reproduce the experiments presented in this work can be found at https://github.com/NtaylorOX/Public_Clinical_Prompt.

Item Type:	Article
Uncontrolled Keywords:	Clinical decision support, few-shot learning, pretrained language models (PLMs), prompt learning
Divisions:	Faculty of Health and Life Sciences Faculty of Health and Life Sciences > Institute of Population Health
Depositing User:	Symplectic Admin
Date Deposited:	04 Apr 2024 13:22
Last Modified:	04 Apr 2024 13:22
DOI:	10.1109/TNNLS.2023.3294633
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3180042