An evidence rating service provided valid correlates of the clinical importance of medical articles and journals

OBJECTIVES: The objective of this study was to determine reliability and validity of McMaster PLUS measures of scientific merit and clinical importance of articles in medical journals. STUDY DESIGN AND SETTING: Analytic survey of peer-reviewed medical journals was carried out. Articles were qualified for inclusion by meeting (1) scientific criteria and (2) a clinical importance rating threshold. Included articles were sent as e-mail alerts to physicians according to their clinical interests. Internal measures included the number of high-quality, clinically important studies published in source journals and response to alerts. For external validation, we correlated internal measures with the Journal Impact Factor (JIF) and citation in DynaMed Plus (DMP). RESULTS: We evaluated 34,232 articles from 57 journals. Inclusion criteria were met by 2,638 articles (7.71%). The number of qualifying articles per journal was correlated with the number of articles with high clinical importance ratings (r 0.96, P < 0.001), article alert clicks (r 0.86, P < 0.001), and DMP citations (r 0.99, P < 0.001). Correlation was lower with the JIF (r 0.68, P < 0.01). CONCLUSIONS: Measures of scientific merit and clinical importance of medical journal articles were strongly correlated with each other, less so with JIFs. Journals varied widely by these measures but, generally, few articles were both scientifically sound and clinically important.