Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study.

Da Costa, Bruno R; Beckett, Brooke; Diaz, Alison; Resta, Nina M; Johnston, Bradley C; Egger, Matthias; Jüni, Peter; Armijo-Olivo, Susan (2017). Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Systematic Reviews, 6(1), p. 44. BioMed Central 10.1186/s13643-017-0441-7

[img]
Preview
Text
DaCosta SystRev 2017.pdf - Published Version
Available under License Creative Commons: Attribution (CC-BY).

Download (476kB) | Preview

BACKGROUND The Cochrane risk of bias tool is commonly criticized for having a low reliability. We aimed to investigate whether training of raters, with objective and standardized instructions on how to assess risk of bias, can improve the reliability of the Cochrane risk of bias tool. METHODS In this pilot study, four raters inexperienced in risk of bias assessment were randomly allocated to minimal or intensive standardized training for risk of bias assessment of randomized trials of physical therapy treatments for patients with knee osteoarthritis pain. Two raters were experienced risk of bias assessors who served as reference. The primary outcome of our study was between-group reliability, defined as the agreement of the risk of bias assessments of inexperienced raters with the reference assessments of experienced raters. Consensus-based assessments were used for this purpose. The secondary outcome was within-group reliability, defined as the agreement of assessments within pairs of inexperienced raters. We calculated the chance-corrected weighted Kappa to quantify agreement within and between groups of raters for each of the domains of the risk of bias tool. RESULTS A total of 56 trials were included in our analysis. The Kappa for the agreement of inexperienced raters with reference across items of the risk of bias tool ranged from 0.10 to 0.81 for the minimal training group and from 0.41 to 0.90 for the standardized training group. The Kappa values for the agreement within pairs of inexperienced raters across the items of the risk of bias tool ranged from 0 to 0.38 for the minimal training group and from 0.93 to 1 for the standardized training group. Between-group differences in Kappa for the agreement of inexperienced raters with reference always favored the standardized training group and was most pronounced for incomplete outcome data (difference in Kappa 0.52, p < 0.001) and allocation concealment (difference in Kappa 0.30, p = 0.004). CONCLUSIONS Intensive, standardized training on risk of bias assessment may significantly improve the reliability of the Cochrane risk of bias tool.

Item Type:

Journal Article (Original Article)

Division/Institute:

04 Faculty of Medicine > Pre-clinic Human Medicine > Institute of Social and Preventive Medicine
04 Faculty of Medicine > Medical Education > Institute of General Practice and Primary Care (BIHAM)

UniBE Contributor:

Da Costa, Bruno and Egger, Matthias

Subjects:

600 Technology > 610 Medicine & health
300 Social sciences, sociology & anthropology > 360 Social problems & social services

ISSN:

2046-4053

Publisher:

BioMed Central

Language:

English

Submitter:

Doris Kopp Heim

Date Deposited:

25 Apr 2017 15:39

Last Modified:

08 Aug 2017 15:56

Publisher DOI:

10.1186/s13643-017-0441-7

PubMed ID:

28253938

BORIS DOI:

10.7892/boris.99501

URI:

https://boris.unibe.ch/id/eprint/99501

Actions (login required)

Edit item Edit item
Provide Feedback