Assessment of contextualised representations in detecting outcome phrases in clinical trials

Abaho, Micheal, Bollegala, Danushka ORCID: 0000-0003-4476-7003, Williamson, Paula R ORCID: 0000-0001-9802-6636 and Dodd, Susanna ORCID: 0000-0003-2851-3337 (2022) Assessment of contextualised representations in detecting outcome phrases in clinical trials [Preprint]

	Text 2203.03547v2.pdf - Submitted version Download (449kB) | Preview
	Text 2203.03547v2.pdf - Submitted version Download (449kB) | Preview

Abstract

Automating the recognition of outcomes reported in clinical trials using machine learning has a huge potential of speeding up access to evidence necessary in healthcare decision-making. Prior research has however acknowledged inadequate training corpora as a challenge for the Outcome detection (OD) task. Additionally, several contextualized representations like BERT and ELMO have achieved unparalleled success in detecting various diseases, genes, proteins, and chemicals, however, the same cannot be emphatically stated for outcomes, because these models have been relatively under-tested and studied for the OD task. We introduce "EBM-COMET", a dataset in which 300 PubMed abstracts are expertly annotated for clinical outcomes. Unlike prior related datasets that use arbitrary outcome classifications, we use labels from a taxonomy recently published to standardize outcome classifications. To extract outcomes, we fine-tune a variety of pre-trained contextualized representations, additionally, we use frozen contextualized and context-independent representations in our custom neural model augmented with clinically informed Part-Of-Speech embeddings and a cost-sensitive loss function. We adopt strict evaluation for the trained models by rewarding them for correctly identifying full outcome phrases rather than words within the entities i.e. given an outcome "systolic blood pressure", the models are rewarded a classification score only when they predict all 3 words in sequence, otherwise, they are not rewarded. We observe our best model (BioBERT) achieve 81.5\% F1, 81.3\% sensitivity and 98.0\% specificity. We reach a consensus on which contextualized representations are best suited for detecting outcomes from clinical-trial abstracts. Furthermore, our best model outperforms scores published on the original EBM-NLP dataset leader-board scores.

Item Type:	Preprint
Uncontrolled Keywords:	cs.CL, cs.CL, cs.LG
Divisions:	Faculty of Health & Life Sciences Faculty of Health & Life Sciences > Inst. Population Health Faculty of Science & Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User:	Symplectic Admin
Date Deposited:	29 Mar 2022 10:17
Last Modified:	21 Apr 2026 19:59
DOI:	10.24105/ejbi.2021.17.9.53-65
Related Websites:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3151001
Disclaimer:	The University of Liverpool is not responsible for content contained on other websites from links within repository metadata. Please contact us if you notice anything that appears incorrect or inappropriate.