Conclusions from surveys may not consider important biases: a systematic survey of surveys

OBJECTIVES: Surveys can inform important health care questions. However, should decision-makers believe strong conclusions that authors of surveys report? Objectives of this systematic survey of surveys are to describe the characteristics and conduct of surveys and investigate the association of the conduct with the conclusions. STUDY DESIGN AND SETTING: We randomly selected 180 surveys published over 1 year in core clinical journals that included a self-report instrument to elicit knowledge, attitudes, behaviors, and experiences regarding health. Study selection and abstraction was independent and in duplicate. We calculated frequencies of descriptive data. We conducted multivariable logistic regression analyses to assess the association of strong or weak conclusions with survey methods. RESULTS: Our results suggest that authors who validate questions in their survey make strong conclusions. We found that strong conclusions may not be associated with response rates or number of respondents. However, it did not appear that journal impact factor was related to strong conclusions. CONCLUSION: Our results suggest that users of surveys should not rely on the conclusions of authors. A critical appraisal tool for users of surveys and guidance for authors about factors to consider when making conclusions would be helpful.