Position-based Prompting for Health Outcome Generation



Abaho, Micheal, Bollegala, Danushka ORCID: 0000-0003-4476-7003, Williamson, Paula R ORCID: 0000-0001-9802-6636 and Dodd, Susanna ORCID: 0000-0003-2851-3337
(2022) Position-based Prompting for Health Outcome Generation. In: Proceedings of the 21st Workshop on Biomedical Language Processing, 2022-5 - 2022-5.

Access the full-text of this item by clicking on the Open Access link.

Abstract

Probing factual knowledge in Pre-trained Language Models (PLMs) using prompts has indirectly implied that language models (LMs) can be treated as knowledge bases. To this end, this phenomena has been effective, especially when these LMs are fine-tuned towards not just data, but also to the style or linguistic pattern of the prompts themselves. We observe that, satisfying a particular linguistic pattern in prompts is an unsustainable, time-consuming constraint in the probing task, especially because, they are often manually designed and the range of possible prompt template patterns can vary depending on the prompting task. To alleviate this constraint, we propose using a position-attention mechanism to capture positional information of each word in a prompt relative to the mask to be filled, hence avoiding the need to re-construct prompts when the prompts' linguistic pattern changes. Using our approach, we demonstrate the ability of eliciting answers (in a case study on health outcome generation) to not only common prompt templates like Cloze and Prefix, but also rare ones too, such as Postfix and Mixed patterns whose masks are respectively at the start and in multiple random places of the prompt. More so, using various biomedical PLMs, our approach consistently outperforms a baseline in which the default PLMs representation is used to predict masked tokens.

Item Type: Conference or Workshop Item (Unspecified)
Divisions: Faculty of Health and Life Sciences
Faculty of Health and Life Sciences > Institute of Population Health
Depositing User: Symplectic Admin
Date Deposited: 20 Oct 2022 08:14
Last Modified: 15 Mar 2024 02:25
DOI: 10.18653/v1/2022.bionlp-1.3
Open Access URL: https://arxiv.org/abs/2204.03489
Related URLs:
URI: https://livrepository.liverpool.ac.uk/id/eprint/3165670