Exploring the impact of design criteria for reference sets on performance evaluation of signal detection algorithms: the case of drug-drug interactions.

Kontsioti, Elpida ORCID: 0000-0002-4053-0220, Maskell, Simon ORCID: 0000-0003-1917-2913 and Pirmohamed, Munir ORCID: 0000-0002-7534-7266 (2023) Exploring the impact of design criteria for reference sets on performance evaluation of signal detection algorithms: the case of drug-drug interactions. Pharmacoepidemiology and drug safety, 32 (8). pp. 832-844.

Access the full-text of this item by clicking on the Open Access link.

Text
Pharmacoepidemiology and Drug - 2023 - Kontsioti - Exploring the impact of design criteria for reference sets on.pdf - Author Accepted Manuscript
Download (1MB) | Preview

Official URL: http://dx.doi.org/10.1002/pds.5609

Abstract

<h4>Purpose</h4>To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug-drug interactions (DDIs).<h4>Methods</h4>Starting from a large and diversified reference set for two-way DDIs, we generated custom-made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three different SDAs when applied to FDA Adverse Event Reporting System (FAERS) data.<h4>Results</h4>For some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on AUC and PPV estimates.<h4>Conclusions</h4>The relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by "favouring" some algorithms while penalising others. This article is protected by copyright. All rights reserved.

Item Type:	Article
Uncontrolled Keywords:	adverse events, drug-drug interactions, performance metrics, pharmacovigilance, postmarketing surveillance, signal detection, spontaneous reports data
Divisions:	Faculty of Health and Life Sciences Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science Faculty of Health and Life Sciences > Institute of Systems, Molecular and Integrative Biology
Depositing User:	Symplectic Admin
Date Deposited:	20 Mar 2023 09:00
Last Modified:	10 Jul 2023 02:52
DOI:	10.1002/pds.5609
Open Access URL:	https://onlinelibrary.wiley.com/doi/10.1002/pds.56...
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3169167