High certainty evidence is stable and trustworthy, whereas evidence of moderate or lower certainty may be equally prone to being unstable.

Djulbegovic, Benjamin; Koletsi, Despina; Hozo, Iztok; Price, Amy; Martimbianco, Ana Luiza Cabrera; Riera, Rachel; Nadanovsky, Paulo; Dos Santos, Ana Paula Pires; Pandis, Nikolaos; Pacheco, Rafael Leite; Fontes, Luis Eduardo; Seehra, Jadbinder; Ahmed, Muneeb; Yao, Liang; Nunan, David; Hemkens, Lars G (2024). High certainty evidence is stable and trustworthy, whereas evidence of moderate or lower certainty may be equally prone to being unstable. Journal of clinical epidemiology, 171, p. 111392. Elsevier 10.1016/j.jclinepi.2024.111392

Text 1-s2.0-S0895435624001471-main.pdf - Accepted Version Restricted to registered users only Available under License Publisher holds Copyright. Download (788kB)

OBJECTIVE

To assess to what extent the overall quality of evidence indicates changes to observed intervention effect estimates when new data become available.

STUDY DESIGN AND SETTING

We conducted a meta- epidemiological study. We obtained evidence from meta-analyses of randomized trials of Cochrane reviews addressing the same healthcare question that was updated with inclusion of additional data between January 2016 and May 2021.

METHODS

We extracted the reported effect estimates with 95% confidence intervals from meta-analyses and corresponding GRADE (Grading of Recommendations Assessment, Development, and Evaluation) assessments of any intervention comparison for the primary outcome in the first and the last updated review version. We considered the reported overall quality (certainty) of evidence (CoE) and specific evidence limitations (no, serious or very serious for risk of bias, imprecision, inconsistency, and/or indirectness). We assessed the change in pooled effect estimates between the original and updated evidence using the ratio of odds ratio (ROR), absolute ROR (aROR), ratio of standard errors (RoSE), direction of effects, and level of statistical significance.

RESULTS

High CoE without limitations characterized 19.3% (n=29) out of 150 included original Cochrane reviews. The update with additional data did not systematically change the effect estimates (mean ROR 1.00; 95%CI 0.99-1.02), which deviated 1.06-fold from the older estimates (median aROR; IQR: 1.01-1.15), gained precision (median RoSE 0.87; IQR 0.76-1.00), and maintained the same direction with the same level of statistical significance in 93% (27 of 29) of cases. Lower CoE with limitations characterized 121 original reviews and graded as moderate CoE in 30.0% (45 of 150), low CoE in 32.0% (48 of 150), and very low CoE in 18.7% (28 of 150) reviews. Their update had larger absolute deviations (median aROR 1.12 to 1.33) and larger gains in precision (median RoSE 0.78 to 0.86) without clear and consistent differences between these categories of CoE. Changes in effect direction or statistical significance were also more common in the lower quality evidence, again with a similar extent across categories (without change in 75.6%, 64.6%, and 75.0% for moderate, low, very low CoE). As limitations increased, effect estimates deviated more (aROR 1.05 with zero, 1.11 with one, 1.25 with two, 1.24 with three limitations) and changes in direction or significance became more frequent (93.2% stable with no limitations, 74.5% with one, 68.2% with two, and 61.5% with three limitations).

CONCLUSIONS

High-quality evidence without methodological deficiencies is trustworthy and stable, providing reliable intervention effect estimates when updated with new data. Evidence of moderate and lower quality may be equally prone to being unstable and cannot indicate if available effect estimates are true, exaggerated, or underestimated.

Interest & Impact

Downloads

0 since deposited on 14 May 2024
0 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item