Errors in the Conduct of Systematic Reviews of Pharmacological Interventions for Irritable Bowel Syndrome

OBJECTIVES: Systematic reviews and meta-analyses are integral to evidence-based clinical decision making. Although flawed systematic reviews could compromise optimal decision making, their accuracy has received limited investigation. We assessed conduct of systematic reviews of pharmaceutical interventions for irritable bowel syndrome (IBS). METHODS: We searched MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews (up to June 2008) to identify and replicate all published systematic reviews and meta-analyses that examined pharmacological interventions for IBS. We identified trials appropriately and inappropriately included according to the investigators' own eligibility criteria and eligible trials the investigators failed to include, and assessed the accuracy of dichotomous data extraction from all truly eligible trials. We conducted meta-analyses of accurate data from all truly eligible trials, and examined the differences between these accurate estimates and those reported by the authors. RESULTS: The search strategy identified 120 citations, and 13 appeared to be relevant. Five systematic reviews did not extract dichotomous data, leaving eight eligible for inclusion. In five of the eight meta-analyses 13-29% of included trials were ineligible according to investigators' criteria, constituting 8-26% of included patients. Six of the meta-analyses missed 17 separate published eligible trials; 3-11% of eligible patients were, as a result, not included. All eight meta-analyses contained errors in dichotomous data extraction, in 29-100% of truly eligible trials, leading to errors in 15 of 16 reported pooled treatment effects. There was a > or =10% relative difference in treatment effects between the reported and recalculated summary statistic in five (31%) cases, and a change in the statistical significance of the recalculated summary statistic in a further four (25%) cases. CONCLUSIONS: We found many errors in both application of eligibility criteria and dichotomous data extraction in the eight meta-analyses studied. Independent verification of systematic reviews and meta-analyses may be required for full confidence in their results.