Measurement precision at the cut score in medical multiple choice exams: Theory matters

Lahner, Felicitas-Maria; Schauber, Stefan; Lörwald, Andrea Carolin; Kropf, Roger; Guttormsen, Sissel; Fischer, Martin R.; Huwendiek, Sören (2020). Measurement precision at the cut score in medical multiple choice exams: Theory matters. Perspectives on medical education, 9(4), pp. 220-228. Springer 10.1007/s40037-020-00586-0

[img]
Preview
Text
Lahner2020_Article_MeasurementPrecisionAtTheCutSc.pdf - Published Version
Available under License Creative Commons: Attribution (CC-BY).

Download (445kB) | Preview

Introduction: In high-stakes assessment, the measurement precision of pass-fail decisions is of great importance. A concept for analyzing the measurement precision at the cut score is conditional reliability, which describes measurement precision for every score achieved in an exam. We compared conditional reliabilities in Classical Test Theory (CTT) and Item Response Theory (IRT) with a special focus on the cut score and potential factors influencing conditional reliability at the cut score.

Methods: We analyzed 32 multiple-choice exams from three Swiss medical schools comparing conditional reliability at the cut score in IRT and CCT. Additionally, we analyzed potential influencing factors such as the range of examinees' performance, year of study, and number of items using multiple regression.

Results: In CTT, conditional reliability was highest for very low and very high scores, whereas examinees with medium scores showed low conditional reliabilities. In IRT, the maximum conditional reliability was in the middle of the scale. Therefore, conditional reliability at the cut score was significantly higher in IRT compared with CTT. It was influenced by the range of examinees' performance and number of items. This influence was more pronounced in CTT.

Discussion: We found that conditional reliability shows inverse distributions and conclusions regarding the measurement precision at the cut score depending on the theory used. As the use of IRT seems to be more appropriate for criterion-oriented standard setting in the framework of competency-based medical education, our findings might have practical implications for the design and quality assurance of medical education assessments.

Item Type:

Journal Article (Original Article)

Division/Institute:

04 Faculty of Medicine > Medical Education > Institute for Medical Education > Assessment and Evaluation Unit (AAE)
04 Faculty of Medicine > Medical Education > Institute for Medical Education

UniBE Contributor:

Lahner, Felicitas-Maria; Lörwald, Andrea Carolin; Guttormsen, Sissel and Huwendiek, Sören

Subjects:

600 Technology > 610 Medicine & health

ISSN:

2212-277X

Publisher:

Springer

Language:

English

Submitter:

Eveline Götschmann-Meile

Date Deposited:

10 Aug 2020 17:08

Last Modified:

01 Sep 2020 01:33

Publisher DOI:

10.1007/s40037-020-00586-0

PubMed ID:

32468274

BORIS DOI:

10.7892/boris.145667

URI:

https://boris.unibe.ch/id/eprint/145667

Actions (login required)

Edit item Edit item
Provide Feedback