Structured decision-making drives guidelines panels' recommendations “for” but not “against” health interventions

BACKGROUND AND OBJECTIVES: The determinants of guideline panels' recommendations remain uncertain. The objective of this study was to investigate factors considered by members of 8 panels convened by the American Society of Hematology (ASH) to develop guidelines using Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system. STUDY DESIGN AND SETTING: Web-based survey of the participants in the ASH guidelines panels. ANALYSIS: two-level hierarchical, random-effect, multivariable regression analysis to explore the relation between GRADE and non-GRADE factors and strength of recommendations (SOR). RESULTS: In the primary analysis, certainty in evidence [OR = 1.83; (95CI% 1.45-2.31)], balance of benefits and harms [OR = 1.49 (95CI% 1.30-1.69)] and variability in patients' values and preferences [OR = 1.47 (95CI% 1.15-1.88)] proved the strongest predictors of SOR. In a secondary analysis, certainty of evidence was associated with a strong recommendation [OR = 3.60 (95% CI 2.16-6.00)] when panel members recommended "for" interventions but not when they made recommendations "against" interventions [OR = 0.98 (95%CI: 0.57-1.8)] consistent with "yes" bias. Agreement between individual members and the group in rating SOR varied (kappa ranged from -0.01 to 0.64). CONCLUSION: GRADE's conceptual framework proved, in general, to be highly associated with SOR. Failure of certainty of evidence to be associated with SOR against an intervention, suggest the need for improvements in the process.