Position-based Prompting for Health Outcome Generation



Abaho, M, Bollegala, D ORCID: 0000-0003-4476-7003, Williamson, P ORCID: 0000-0001-9802-6636 and Dodd, S
(2022) Position-based Prompting for Health Outcome Generation. .

[img] Text
Prefix_Cloze_Postfix__DB_.pdf - Accepted Version

Download (481kB) | Preview

Abstract

Probing Pre-trained Language Models (PLMs) using prompts has indirectly implied that language models (LMs) can be treated as knowledge bases. To this end, this phenomena has been effective especially when these LMs are fine-tuned towards not just data of a specific domain, but also to the style or linguistic pattern of the prompts themselves. We observe that, satisfying a particular linguistic pattern in prompts is an unsustainable constraint that unnecessarily lengthens the probing task, especially because, they are often manually designed and the range of possible prompt template patterns can vary depending on the prompting objective and domain. We therefore explore an idea of using a position-attention mechanism to capture positional information of each word in a prompt relative to the mask to be filled, hence avoiding the need to re-construct prompts when the prompts linguistic pattern changes. Using our approach, we demonstrate the ability of eliciting answers to rare prompt templates (in a case study on health outcome generation) such as Postfix and Mixed patterns whose missing information is respectively at the start and in multiple random places of the prompt. More so, using various biomedical PLMs, our approach consistently outperforms a baseline in which the default mask language model (MLM) representation is used to predict masked tokens.

Item Type: Conference or Workshop Item (Unspecified)
Uncontrolled Keywords: cs.CL, cs.CL, cs.LG
Divisions: Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User: Symplectic Admin
Date Deposited: 31 Mar 2022 09:27
Last Modified: 24 May 2022 00:10
Related URLs:
URI: https://livrepository.liverpool.ac.uk/id/eprint/3151816