External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb

Snell, Kym IE, Archer, Lucinda, Ensor, Joie, Bonnett, Laura J ORCID: 0000-0002-6981-9212, Debray, Thomas PA, Phillips, Bob, Collins, Gary S and Riley, Richard D (2021) External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb. JOURNAL OF CLINICAL EPIDEMIOLOGY, 135. pp. 79-89.

Text Sample size for validation revised SUBMITTED clean.docx - Author Accepted Manuscript Download (1MB)

Abstract

<h4>Introduction</h4>Sample size "rules-of-thumb" for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach.<h4>Methods</h4>Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility.<h4>Results</h4>Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model.<h4>Conclusion</h4>Where researchers can anticipate the distribution of the model's LP (eg, based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.

Item Type:	Article
Uncontrolled Keywords:	Sample size, External validation, Clinical prediction model, Calibration and discrimination, Net benefit, Simulation
Depositing User:	Symplectic Admin
Date Deposited:	08 Mar 2021 14:44
Last Modified:	18 Jan 2023 22:57
DOI:	10.1016/j.jclinepi.2021.02.011
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3116766