Influence of focusing on dominant first order loop when assessing the certainty of evidence of network meta-analysis: a case study

OBJECTIVES: Assessing the certainty of evidence (COE) of both direct and indirect evidence of is crucial for enhancing the understanding of network meta-analysis (NMA) findings and drawing appropriate conclusions. To make the certainty rating of indirect evidence feasible, Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidance suggests focusing on the dominant first order loop that typically contributes the most information to the indirect estimate. This approach, however, can raise concerns that failure to consider most or all loops will provide inaccurate COE of indirect evidence. STUDY DESIGN AND SETTING: We investigated six NMA publications and compared indirect COE ratings considering only the dominant first order loop vs ratings considering most or all loops. RESULTS: Across 103 indirect comparisons in the six NMAs, 15% (15 out of 103) comparisons did not have a first order loop, and 47% (42 out of 88) comparisons with dominant first order loops contributed a weight smaller than 50% of the indirect evidence. We identified a total of 6% (6 out of 103) indirect comparisons in which ratings based on the dominant first order loop resulted in different COE ratings from considering most or all loops, with two shifting from moderate to low COE of NMA. CONCLUSION: Our findings suggest that using only the dominant first order loop vs considering most or all loops is very seldom misleading. Further research is necessary to replicate or refute our findings in other network meta-analyses and assess the implications for clinical decision-making. PLAIN LANGUAGE SUMMARY: NMA allows researchers to compare the relative effectiveness of multiple treatments. The GRADE working group provides guidance to establish the confidence in treatment effects (how likely they are to be true) by evaluating the COE to direct, indirect and NMA estimates, which can be high, moderate, low or very low. We investigated the influences of using two different approaches (dominant first order loop vs consideration of most or all loops) for rating the certainty of NMA. We found similar certainty ratings of indirect evidence and NMA between the two approaches. Therefore, systematic review authors can continue using the simpler approach.