The impact of an online course on agreement rates of the certainty of evidence assessment using Grading of Recommendations, Assessment, Development, and Evaluation Approach: a before-and-after study

BACKGROUND AND OBJECTIVE: The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach is a systematic method for assessing the certainty of evidence (CoE) and strength of recommendations in health care. We aimed to verify the effects of an online-based GRADE course on multirater consistency in the evaluation of the CoE in systematic reviews (SRs) analysis. STUDY DESIGN AND SETTINGS: Sixty-five Brazilian methodologists and researchers participated in an online course over 8 weeks. Asynchronous lessons and weekly synchronous meetings addressed the GRADE system in the context of CoE assessment. We asked participants to evaluate the CoE of random SRs (two before and another two after the course). Analyzes focused on the multirater agreement with a standard response, in the interrater agreement, and before-after changes in the proportion of participants that rated down the domains. RESULTS: 48 individuals completed the course. Participants presented improvements in the raters' assessment of the CoE using the GRADE approach after the course. The multirater consistency of indirectness, imprecision, and the overall CoE increased after the course, as well as the agreement between raters and the standard response. Furthermore, interrater reliability increased for risk of bias, inconsistency, indirectness, publication bias, and overall CoE, indicating progress in between-raters consistency. After the course, approximately 78% of individuals rated down the overall CoE to a low/very low degree, and participants presented more explanations for the judgment of each domain. CONCLUSION: An online GRADE course improved the consistency and agreement of the CoE assessment by Brazilian researchers. Online training courses have the potential to improve skills in guideline methodology development.