Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools.

Appelhaus, Stefan; Werner, Susanne; Grosse, Pascal; Kämmer, Juliane E (2023). Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools. Medical Education Online, 28(1), p. 2143298. Taylor & Francis 10.1080/10872981.2022.2143298

Preview

Text Feedback_fairness_and_validity_effects_of_disclosing_and_reusing_multiple_choice_questions_in_medical_schools.pdf - Published Version Available under License Creative Commons: Attribution (CC-BY). Download (1MB) | Preview

BACKGROUND

Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive.

METHODS

We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as 'new'; 'reused, not disclosed'; or 'reused, disclosed'. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests.

RESULTS

The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, ηp2 = 0.011. On average, reused, disclosed items were significantly easier (M = 0.83) than reused, not disclosed items (M = 0.71) and entirely new items (M = 0.66), P < .001, ηp2 = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, ηp2 = 0.002.

CONCLUSIONS

Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items.

Interest & Impact

Downloads

134 since deposited on 10 Nov 2022
45 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item