Paper 2: a semi-automated approach facilitated the assessment of the certainty of evidence for in a network meta-analysis: part 2 – indirect and mixed comparisons

OBJECTIVES: To implement a semiautomated approach to facilitate rating the Grading, Recommendation, Assessment, Development and Evaluation certainty of evidence (CoE) for indirect and network meta-analysis (NMA) estimates. METHODS: We developed and implemented algorithms for generating automated ratings for the CoE for indirect and network estimates in two living NMAs of rheumatoid arthritis treatment. At the indirect stage, inputs included CoE ratings for direct estimates and the contribution matrix. Intransitivity ratings were assigned based on the indirectness ratings of the two direct estimates with the highest percent contribution. An online tool (customized to our project) facilitated assessment of imprecision on the network estimate. Automated ratings were reviewed by two independent experts. RESULTS: Across 1306 indirect comparisons, the contribution matrix identified the dominant branches of evidence regardless of whether a single first order loop was present (80%) or not. The reviewers agreed with all automated CoE ratings for incoherence (n = 34), network estimates (n = 34) and imprecision (n = 1447). They agreed with the automated intransitivity algorithm except when the total contribution of the top-two direct estimates was low (eg, <50%, which occurred in 38% of the estimates). CONCLUSION: Automated approaches facilitated CoE ratings for indirect and network estimates. Further work is required to define appropriate algorithms for intransitivity.