ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis.

Chiocchia, Virginia; Nikolakopoulou, Adriani; Higgins, Julian P T; Page, Matthew J; Papakonstantinou, Theodoros; Cipriani, Andrea; Furukawa, Toshi A; Siontis, George C.M.; Egger, Matthias; Salanti, Georgia (2021). ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis. BMC medicine, 19(1), p. 304. BioMed Central 10.1186/s12916-021-02166-3

Chiocchia_BMCMed_2021.pdf - Published Version
Available under License Creative Commons: Attribution (CC-BY).

Download (2MB) | Preview


Selective outcome reporting and publication bias threaten the validity of systematic reviews and meta-analyses and can affect clinical decision-making. A rigorous method to evaluate the impact of this bias on the results of network meta-analyses of interventions is lacking. We present a tool to assess the Risk Of Bias due to Missing Evidence in Network meta-analysis (ROB-MEN).


ROB-MEN first evaluates the risk of bias due to missing evidence for each of the possible pairwise comparison that can be made between the interventions in the network. This step considers possible bias due to the presence of studies with unavailable results (within-study assessment of bias) and the potential for unpublished studies (across-study assessment of bias). The second step combines the judgements about the risk of bias due to missing evidence in pairwise comparisons with (i) the contribution of direct comparisons to the network meta-analysis estimates, (ii) possible small-study effects evaluated by network meta-regression, and (iii) any bias from unobserved comparisons. Then, a level of "low risk", "some concerns", or "high risk" for the bias due to missing evidence is assigned to each estimate, which is our tool's final output.


We describe the methodology of ROB-MEN step-by-step using an illustrative example from a published NMA of non-diagnostic modalities for the detection of coronary artery disease in patients with low risk acute coronary syndrome. We also report a full application of the tool on a larger and more complex published network of 18 drugs from head-to-head studies for the acute treatment of adults with major depressive disorder.


ROB-MEN is the first tool for evaluating the risk of bias due to missing evidence in network meta-analysis and applies to networks of all sizes and geometry. The use of ROB-MEN is facilitated by an R Shiny web application that produces the Pairwise Comparisons and ROB-MEN Table and is incorporated in the reporting bias domain of the CINeMA framework and software.

Item Type:

Journal Article (Original Article)


04 Faculty of Medicine > Pre-clinic Human Medicine > Institute of Social and Preventive Medicine (ISPM)
04 Faculty of Medicine > Department of Cardiovascular Disorders (DHGE) > Clinic of Cardiology

Graduate School:

Graduate School for Health Sciences (GHS)

UniBE Contributor:

Chiocchia, Virginia, Nikolakopoulou, Adriani, Papakonstantinou, Theodoros, Siontis, Georgios, Egger, Matthias, Salanti, Georgia


600 Technology > 610 Medicine & health
300 Social sciences, sociology & anthropology > 360 Social problems & social services




BioMed Central


[4] Swiss National Science Foundation




Andrea Flükiger-Flückiger

Date Deposited:

09 Dec 2021 12:56

Last Modified:

03 Jan 2023 11:33

Publisher DOI:


PubMed ID:


Uncontrolled Keywords:

Evidence synthesis Missing evidence Network meta-analysis Publication bias Reporting bias Risk of bias Selective outcome reporting




Actions (login required)

Edit item Edit item
Provide Feedback