The Importance of Quality of Primary Studies in Producing Unbiased Systematic Reviews

BACKGROUND: Traditional and largely qualitative reviews of evidence are now giving way to much more structured systematic overviews that use a quantitative method to calculate the overall effect of treatment. The latter approach is dependent on the quality of primary studies, which may introduce bias if they are of poor methodologic quality. OBJECTIVE: To test the hypothesis that the inclusion of poor-quality trials in meta-analyses would bias the conclusions and produce incorrect estimates of treatment effect. METHODS: An overview of randomized trials of antiestrogen therapy in subfertile men with oligospermia was performed to test the hypothesis. Data sources included online searching of MEDLINE and Science Citation Index databases between 1966 and 1994, scanning the bibliography of known primary studies and review articles, and contacting experts in the field. After independent, blind assessment, nine of 149 originally identified studies met the inclusion criteria and were selected. We assessed study quality independently. Outcome data from each study were pooled and statistically summarized. RESULTS: There was a marginal improvement in pregnancy rate with antiestrogen treatment (odds ratio, 1.6; 95% confidence interval, 0.9 to 2.6). Sensitivity analyses on the basis of methodologic quality demonstrated that poor-quality studies produced a positive effect with treatment, whereas no benefit was observed with high-quality studies. CONCLUSION: The results of a meta-analysis are influenced by the quality of the primary studies included. Methodologically, poor studies tend to exaggerate the overall estimate of treatment effect and may lead to incorrect inferences.