A Citation Analysis of the Impact of Blinded Peer Review
(JAMA. 1994;272:147-149)
David N. Laband, PhD, Michael J. Piette, PhD
Objective.--To determine whether articles published in
journals using blinded peer review receive significantly more or fewer
citations than those published in journals using nonblinded peer
review.
Design.--Drawing from a sample of 1051 full articles
published in 28 economics journals during 1984, we used nonlinear
regression and ordered probit techniques to estimate the impact of
blinded peer review on citations of these articles in 1985 through
1989.
Outcomes.--Citations of articles.
Results.--Articles published in journals using blinded peer
review were cited significantly more than articles published in
journals using nonblinded peer review, controlling for a variety of
author, article, and journal attributes.
Conclusions.--Nonblinded peer review apparently suffers from
type I error to a greater extent than blinded peer review. That is,
journals using nonblinded peer review publish a larger fraction of
papers that should not have been published than do journals using
blinded peer review. When reviewers know the identity of the author(s)
of an article, they are able to (and evidently do) substitute
particularistic criteria for universalistic criteria in their
evaluative process.
(JAMA. 1994;272:147-149)
In this article, we use citation
analysis to shed empirical light on the relative efficiency of
double-blind vs single-blind refereeing of journal articles. A
single-blind review occurs when the reviewer(s) knows the identity of
the author(s), but the latter do not know the identities of the former.
With a double-blind review, neither reviewers' nor authors'
identities are revealed to the other party. We view the double-blind vs
single-blind review process debate in terms of the relative incidence
of type I errors and type II errors. Do editors using a single-blind
review process systematically fail to reject papers that have little or
no impact on the profession (type I error) and/or fail to accept more
truly good papers (type II error) than do editors of double-blind
journals? Both types of errors might characterize a single-blind review
process for at least two reasons. First, reviewers with knowledge of
the author's identity might economize on their refereeing costs by
substituting the already-revealed value of the author's average
contribution in previous articles for their evaluation and forecast of
the marginal contribution contained in the manuscript under
consideration. Second, personal characteristics of the author (eg,
gender, institutional affiliation, friendship with the reviewer, race,
and intellectual conformity with the reviewer) may weigh more heavily
in a reviewer's evaluation of the publishability of a manuscript than
the reviewer's forecast of the marginal contribution contained
therein.
To gauge the severity of the type I and type II error problem that
arguably plagues single-blind reviewing but not double-blind reviewing,
one must have information regarding how the marketplace for scientific
ideas responds to published articles. Without knowing the fate of
manuscripts submitted but rejected for publication, one simply cannot
determine the severity of the type II error problem. However, drawing
from a large sample of articles published in 28 top economics journals
in 1984, we were able to investigate the degree to which journals using
a single-blind review process publish articles that are revealed not to
have the impact that might reasonably have been expected.
DATA
We compiled information on 1051 articles published in 28 top
economics journals in 1984. Specifically, we identified citations to
each article, as listed in the Social Sciences Citation
Index[1] for the 5 years following publication
(1985 through 1989); length in American Economic Review-equivalent
pages, a relative quality rating of the journal in which each article
was published[2] ; and the review process used by each journal. (Our sample of 1051 articles consists only of those articles
for which complete information on all variables was available. Thus,
for example, if we were unable to locate an author in the American
Economic Review directory in an effort to obtain age and
affiliation information, the article [co]written by the individual
was dropped from the data set. We concede that this may ultimately bias
our results, but it is unclear a priori in which direction any such
biases, if present, would run.)
Articles published in the 13 journals using double-blind review
received an average of 6.7 citations per article during the immediate
5-year period following publication, while articles published in the 15
journals using single-blind review received an average of 7.3 citations
per article during that same time period. Articles published in the
former journals were, on average, shorter than articles published in
the latter. The average quality rating (developed by Liebowitz and
Palmer [2]) assigned to the set of journals using
double-blind review was 38.5 (on a scale from 0 to 100), while the
average rating of the journals using single-blind review was 58.6.
METHODS
We estimated the impact of the type of review process on
citations to published articles, controlling for article- and
journal-specific characteristics that might influence citations (Table 1). This method permitted us to evaluate whether one
type of review process was superior to the other in terms of either (1)
identifying articles that will attract more citations than would be
predicted by the characteristics or (2) failing to identify articles
that will be cited less than would be predicted by these
characteristics.
The distribution of citations to articles is nonnormal. Most scholarly
articles in economics are cited infrequently, if at all[3]; this finding holds across scientific disciplines
generally.[4] [5] A relatively small number of articles and
authors are truly influential; most offer marginal contributions to the
stock of scientific knowledge. To account for this acknowledged
skewness in the distribution of citations, we estimated our model using
the ordered probit nonlinear regression method (LIMDEP software
package, Version 6.0, Econometric Software Inc, Bellport, NY). Our
dependent variable, citations in 1985 through 1989 to an article
published in 1984, was distributed in such a manner that quintiles were
easy to identify. Finer analysis by decile was impossible because some
21% of all articles received no citations at all during the 1985
through 1989 period.
We expected journals using a single-blind review process to be plagued
by both type I and type II errors to a greater extent than journals
using a double-blind review process. Thus, we expected articles
published in the latter group to attract more citations than articles
published in the former group. Citations should be a positive function
of article length--more substantive scientific contributions will
plausibly require greater elucidation than less substantive
contributions. Citations also are likely to be influenced by journal of
publication. Readers are attracted to a scholarly journal, at least in
part, by the expected value of articles published. A
citations-based rating of the relative impact of journals, like the one
developed by Liebowitz and Palmer,[2] provides relevant
information about the actual value of a journal's past contributions.
Since the value of past contributions is probably used implicitly by
readers in forecasting the expected value of current contributions in a
scholarly journal, we expected citations of current articles to be
influenced in a positive manner by the relative quality rating of the
journal in which the article was published.
RESULTS
The estimated relationship between the dependent citations variable and
the explanatory variables (double-blind review, article length, and
journal quality), with asymptotic t values reported in
parentheses, follows:
CITATIONS 1985-1989 =
-0.840 (-6.471) + 0.261 (3.442)
DOUBLE-BLIND + 0.087 (16.840)
length + 0.015 (9.789)
JOURNAL QUALITY
The coefficient estimate of the double-blind review dummy
variable reveals what the residual impact of the review process is on
citations. Articles published in journals using the double-blind review
process have a significantly greater probability of receiving more
citations than would be predicted on the basis of length and journal
quality than articles reviewed single-blind. As expected, article
length and relative quality of the publishing journal both demonstrate
positive and statistically significant explanatory power with respect
to the probability that an article achieves specific citation levels.
In previous work, Laband[3] argued that the
distinguishing feature of top journals is their differential ability to
"capture" the few high-impact articles written over time. This
suggests that our understanding of differences between double-blind and
single-blind reviewing might be further extended by examining the tail
ends of the distribution of residuals. That is to say, does either
review process outperform the other in identifying the articles that
are much better than would be predicted by length, quality of journal,
and citations per article? Conversely, does either review process
consistently pick "losers," in the sense that the articles selected
for publication do not live up to their expected impact, measured by
subsequent citations? To investigate this, we report in
Table 2 the actual and predicted
probabilities of articles receiving citations in each of the quintiles, by review
process.
The findings reported in Table 2 are revealing. Articles reviewed
single-blind are less likely than those reviewed double-blind to be
identified correctly as the highest-impact articles (those with nine or
more citations in the ensuing 5 years). By the same token, articles
reviewed single-blind are more likely than those reviewed double-blind
to be misidentified as the lowest-impact papers (those with no
citations in the ensuing 5 years).
CONCLUSIONS
Our findings with respect to 28 top economics journals suggest
that the double-blind review process outperforms the single-blind
review process. Specifically, citations to articles published in
journals using a double-blind review process exceeded citations of
articles published in journals using a single-blind review process,
controlling for article length and the quality of the publishing
journal. We conclude that the single-blind review process apparently
suffers from a type I error bias to a greater extent than the
double-blind review process. Can our findings be generalized to
scholarly journals in the biomedical literature? We suspect that they
can; our study offers a methodological basis for shedding empirical
light on that question.
From the Department of Economics and Finance, Salisbury (Md) State
University (Dr Laband) and Economic Research Services, Tallahassee, Fla
(Dr Piette). Dr Laband is now with the Department of Economics, Auburn
(Ala) University.
Presented in part at the Second International Congress on Peer Review
in Biomedical Publication, Chicago, Ill, September 11, 1993.
Address correspondence to Department of Economics, Auburn University,
Auburn, AL 36849 (Dr Laband).
References
1. Institute for Scientific Information. Social
Sciences Citation Index. Philadelphia, Pa: Institute for
Scientific Information; annual volumes 1985-1989.
2. Liebowitz SJ, Palmer JP. Assessing the relative impacts
of economics journals. J Econ Lit. 1984;22:77-88.
3. Laband DN. Article popularity. Econ Inquiry. 1986;24:173-180.
4. Hamilton DP. Research papers: who's uncited now? Science. 1991;251:25.
5. Hamilton DP. Publishing by--and for?--the numbers. Science. 1990;250:1331-1332.
Table of Contents
