An index of scientific quality for health reports in the lay press

Although the quality of health reporting has been criticized for being unscientific, evaluations of health care reporting have been limited by the lack of a reliable and credible measure of scientific quality. We developed an index of scientific quality (ISQ) for health-related news reports and tested its reliability and sensibility. Items were generated from a survey of the literature and experts in research methodology. Items that were unclear, confusing or discriminated poorly between articles of high and low scientific quality were revised or deleted in an iterative process wherein potential criteria were independently applied to samples of 5 to 15 articles by 6 raters. To test the reliability of the final criteria 60 articles were drawn from three sampling frames: newspapers, magazines, and professional journals. Articles were intentionally selected to obtain a wide range of quality and topics. Two categories of raters were used: research assistants and physicians with research training. All 6 raters assessed all 60 articles. The sensibility of the index was tested by a questionnaire with 13 items related to face validity and content validity as well as other aspects of sensibility. The questionnaire was completed by 20 researchers and 13 health and science writers. The final ISQ includes 7 items that address the extent to which a report allows readers to draw conclusions about the applicability, validity and practical importance of the information that is reported. Chance corrected agreement (kappa) among all 6 raters for overall scientific quality was 0.62 (SE 0.02). The index was found to be sensible with only one major problem, the need for judgment in making ratings. While some degree of subjectivity appears to be inevitable in rating the scientific quality of health reports, the ISQ is acceptable reliable and credible and should be useful for evaluating and improving the scientific quality of health reporting.