Shining a spotlight on scoring in the OSCE: Checklists and item weighting


Homer, Matt, Fuller, Richard ORCID: 0000-0001-7965-4864, Hallam, Jennifer and Pell, Godfrey (2020) Shining a spotlight on scoring in the OSCE: Checklists and item weighting. Medical Teacher, 42 (9). pp. 1037-1042.

Access the full-text of this item by clicking on the Open Access link.
[img] Text
ShiningASpotlightOnOSCEScoringMain.docx - Author Accepted Manuscript
Download (84kB)

Abstract

<b>Introduction:</b> There has been a long-running debate about the validity of item-based checklist scoring of performance assessments like OSCEs. In recent years, the conception of a checklist has developed from its dichotomous inception into a more 'key-features' and/or chunked approach, where 'items' have the potential to become weighted differently, but the literature does not always reflect these broader conceptions.<b>Methods:</b> We consider theoretical, design and (clinically trained) assessor issues related to differential item weighting in checklist scoring of OSCEs stations. Using empirical evidence, this work also compares candidate decisions and psychometric quality of different item-weighting approaches (i.e. a simple 'unweighted' scheme versus a differentially weighted one).<b>Results:</b> The impact of different weighting schemes affect approximately 30% of the key borderline group of candidates, and 3% of candidates overall. We also find that measures of overall assessment quality are a little better under the differentially weighted scoring system.<b>Discussion and conclusion:</b> Differentially weighted modern checklists can contribute to valid assessment outcomes, and bring a range of additional benefits to the assessment. Judgment about weighting of particular items should be considered a key design consideration during station development and must align to clinical assessor expectations of the relative importance of sub-tasks.

Item Type: Article
Uncontrolled Keywords: OSCE scoring, checklist design, item weighting, assessment quality
Depositing User: Symplectic Admin
Date Deposited: 13 Aug 2020 07:26
Last Modified: 18 Jan 2023 23:37
DOI: 10.1080/0142159X.2020.1781072
Open Access URL: https://eprints.whiterose.ac.uk/161400/
Related URLs:
URI: https://livrepository.liverpool.ac.uk/id/eprint/3097401
Dimensions
Altmetric
CORE (COnnecting REpositories)