Comparison of evidence of treatment effects in randomized and nonrandomized studies on allergen immunotherapy

Nonrandomized studies (NRS) on allergen immunotherapy (AIT) particularly lend themselves to evaluate outcomes that are insufficiently addressed in randomized controlled studies (RCTs). However, NRS are prone to several sources of bias, which limit their validity. We aimed at comparing AIT effects between RCTs and NRS and evaluate the reasons for discrepancies in study results. In this analysis, we compared NRS on AIT (including subcutaneous and sublingual immunotherapy, SCIT and SLIT, respectively) with SLIT and SCIT RCTs from published meta-analyses, assessing the Risk of Bias (RoB) for each study and the certainty of evidence from NRS and RCTs using the GRADE approach. We found: (1) very serious RoB in the 7 NRS included in the meta-analysis showing a large difference between AIT and controls (standardized mean difference [SMD] for symptom score [SS], -1.77; 95% CI, -2.30, -1.24; p < .001; I²  = 95%) with very low certainty evidence; (2) serious RoB in the 13 SCIT-RCTs reporting a moderate-to-high difference between SCIT and controls (SMD for SS, -0.81; 95% CI, -1.12, -0.49; p < .001; I²  = 88%) with moderate certainty evidence; (3) low RoB in the 13 SLIT-RCTs reporting a small benefit (SMD for SS, -0.28; 95% CI, -0.37, -0.19; p < .001; I²  = 54.2%) with high certainty evidence. Similar results were reported for medication score. Our evidence is sufficient to conclude that the magnitude of effect estimates of NRS and RCTs directly correlate with the degree of RoB and inversely with the overall evidence certainty. NRS, which are more affected than RCTs by bias resulting in low certainty evidence, showed the largest effect size. Sound NRS are needed to complement RCTs.