Home
Scholarly Works
Are higher quality papers cited more often?
Journal article

Are higher quality papers cited more often?

Abstract

Most researchers are aware of the increasing emphasis placed on citations as an indicator of quality by their colleagues, as well as by administrators and librarians. But is it true that high quality papers are cited more often? In what we believe to be the only previous publication to examine this question, West and McIlwaine (2002) selected 79 papers that had been published in the journal Addiction in 1997. These papers were rated by an assistant editor of the journal on a five-level quality scale (ranging from 'among the top 10%' to 'should not have been published') as well as a second 'blind' rater. The authors found that the number of citations to each paper from 1997 to 2000 did not correlate with the mean score of the two reviewers, the score assigned by the editors, or the score assigned by the blind referees. The peer-review system used by Physics in Medicine and Biology (PMB) facilitates a larger, prospective study with much less effort. In general, two reviewers are asked to assign each paper a score of 1 to 10 in each of three categories: originality, soundness and significance. The total scores (out of 30) are averaged and on this basis manuscripts are given a quality rating from Q1 (highest quality) to Q5. In a paper recently published in Scientometrics (Patterson and Harris 2009), we compared the number of citations to papers published in PMB in 2003, 2004 and 2005 to the mean quality rating assigned by the reviewers. In total, data were analyzed for 1095 published papers. For each year we found that there was a low but statistically robust correlation between citations and quality rating: for 2003, -0.227 (p < 0.001); for 2004, -0.238 (p < 0.001), and for 2005, -0.154 (p < 0.01). The low correlation means that it is not possible to predict the citation frequency of an individual paper with accuracy but, as a group, the highest ranked papers were cited about twice as often as the average for all published papers. We also examined the data retrospectively by dividing the papers published in each year into five citation quintiles. A paper of the highest quality (Q1) was about ten times more likely to be found in the most-cited quintile than in the least-cited. While it is reassuring to find that the best papers in PMB are, indeed, cited more often on average, we cannot discount the hypothesis that both measures might be influenced by an extrinsic factor, such as the reputation of the authors. We suggest that a study similar to ours be performed for a journal that utilizes a system of double-blind peer review. Michael S Patterson Editorial Board Member Simon Harris Publisher References West R and McIlwaine A 2002 What do citation counts count for in the field of addiction? An empirical evaluation of citation counts and their link with peer ratings of quality Addiction 97 501–4 Patterson M S and Harris S 2009 The relationship between reviewers' quality-scores and number of citations for papers published in the journal Physics in Medicine and Biology from 2003–2005 Scientometrics 80 343–9

Authors

Patterson MS; Harris S

Journal

Physics in Medicine and Biology, Vol. 54, No. 17,

Publisher

IOP Publishing

Publication Date

September 7, 2009

DOI

10.1088/0031-9155/54/17/e01

ISSN

0031-9155

Contact the Experts team