Using video-based examiner score comparison and adjustment (VESCA) to compare the influence of examiners at different sites in a distributed objective structured clinical exam (OSCE).

Yeates, Peter ORCID: 0000-0001-6316-4051, Maluf, Adriano ORCID: 0000-0001-8375-0533, Cope, Natalie ORCID: 0000-0003-2053-0474, McCray, Gareth, McBain, Stuart, Beardow, Dominic, Fuller, Richard ORCID: 0000-0001-7965-4864 and McKinley, Robert Bob ORCID: 0000-0002-3684-3435 (2023) Using video-based examiner score comparison and adjustment (VESCA) to compare the influence of examiners at different sites in a distributed objective structured clinical exam (OSCE). BMC medical education, 23 (1). 803-.

Access the full-text of this item by clicking on the Open Access link.

PDF Using video-based examiner score comparison and adjustment (VESCA) to compare the influence of examiners at different sites .pdf - Open Access published version Download (1MB) | Preview

Abstract

<h4>Purpose</h4>Ensuring equivalence of examiners' judgements within distributed objective structured clinical exams (OSCEs) is key to both fairness and validity but is hampered by lack of cross-over in the performances which different groups of examiners observe. This study develops a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA) using it to compare examiners scoring from different OSCE sites for the first time.<h4>Materials/ methods</h4>Within a summative 16 station OSCE, volunteer students were videoed on each station and all examiners invited to score station-specific comparator videos in addition to usual student scoring. Linkage provided through the video-scores enabled use of Many Facet Rasch Modelling (MFRM) to compare 1/ examiner-cohort and 2/ site effects on students' scores.<h4>Results</h4>Examiner-cohorts varied by 6.9% in the overall score allocated to students of the same ability. Whilst only a tiny difference was apparent between sites, examiner-cohort variability was greater in one site than the other. Adjusting student scores produced a median change in rank position of 6 places (0.48 deciles), however 26.9% of students changed their rank position by at least 1 decile. By contrast, only 1 student's pass/fail classification was altered by score adjustment.<h4>Conclusions</h4>Whilst comparatively limited examiner participation rates may limit interpretation of score adjustment in this instance, this study demonstrates the feasibility of using VESCA for quality assurance purposes in large scale distributed OSCEs.

Item Type:	Article
Uncontrolled Keywords:	Humans, Educational Measurement, Clinical Competence, Students, Medical
Divisions:	Faculty of Health and Life Sciences Faculty of Health and Life Sciences > Institute of Life Courses and Medical Sciences
Depositing User:	Symplectic Admin
Date Deposited:	21 Dec 2023 16:45
Last Modified:	21 Dec 2023 16:45
DOI:	10.1186/s12909-023-04774-4
Open Access URL:	https://doi.org/10.1186/s12909-023-04774-4
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3177620