Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol

Da Costa, Bruno R.; Resta, Nina M.; Beckett, Brooke; Israel-Stahre, Nicholas; Diaz, Alison; Johnston, Bradley C.; Egger, Matthias; Jüni, Peter; Armijo-Olivo, Susan (2014). Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol. Systematic Reviews, 3(1), p. 144. BioMed Central 10.1186/2046-4053-3-144

daCosta SystRev 2014.pdf - Published Version
Available under License Creative Commons: Attribution (CC-BY).

Download (248kB) | Preview

BACKGROUND The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers. METHODS/DESIGN This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals. DISCUSSION This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

Item Type:

Journal Article (Original Article)


04 Faculty of Medicine > Pre-clinic Human Medicine > Institute of Social and Preventive Medicine (ISPM)
04 Faculty of Medicine > Pre-clinic Human Medicine > CTU Bern

UniBE Contributor:

Da Costa, Bruno; Egger, Matthias and Jüni, Peter


600 Technology > 610 Medicine & health
300 Social sciences, sociology & anthropology > 360 Social problems & social services




BioMed Central




Doris Kopp Heim

Date Deposited:

05 Jan 2015 16:34

Last Modified:

19 Sep 2017 07:47

Publisher DOI:


PubMed ID:


Uncontrolled Keywords:

Systematic review, Meta-analysis, Risk of bias, RCT, Cochrane





Actions (login required)

Edit item Edit item
Provide Feedback