Peer Review Is an Effective Screening Process to Evaluate Medical
Manuscripts
(JAMA. 1994;272:105-107)
Margaret Abby; Michael D. Massey, MCM; Susan Galandiuk, MD;
Hiram C. Polk, Jr, MD
Objective.--To measure the effectiveness of peer review
as a screening process to evaluate medical manuscripts.
Design.--Retrospective.
Setting.--The editorial office of the American Journal
of Surgery (AJS ).
Method.--A MEDLINE search was conducted of publications from
1984 to 1992 for manuscripts that were identical or similar to those
rejected by AJS between January and December 1989. Manuscripts
that were submitted to AJS by foreign authors were excluded
because of the presumed difficulty in tracking foreign-language
publications.
Main Outcome Measures.--The percentage of manuscripts
rejected by AJS that were subsequently published in journals
indexed by MEDLINE, the time from rejection to ultimate publication,
and the journal of publication. The reasons for rejection were also
documented. We assumed that the majority of rejected manuscripts would
be published within 3 years after rejection.
Results.--One hundred twenty-five manuscripts submitted by
North American authors were rejected by AJS in 1989, and 62%
were not subsequently published in another core medical journal during
the study period. The average duration between rejection and later
publication was 17 months. Of those manuscripts subsequently published,
54% appeared in general surgical journals, including 12% that were
revised, reevaluated, and later accepted by AJS. Twenty-nine
percent of the rejected manuscripts were published in specialty medical
journals, 10% in state and local journals, and the remainder in
general medical journals. Twenty-eight percent of the authors of
rejected manuscripts had previously and subsequently published
manuscripts on very similar subjects.
Conclusions.--Our data indicate that the review process
serves as a sieve and influences whether manuscripts are published in
core medical journals. This was demonstrated by the fact that rejected
manuscripts often were not published in other indexed medical journals.
(JAMA. 1994;272:105-107)
Little sound scientific research has
been published on the effectiveness of the review process.[1]
Skeptics of the process believe that peer review exercises little
control over what is published. Some even speculate that persistent
authors forward their manuscripts from journal to journal in a random
fashion until eventual publication. Our study was designed to measure
the effectiveness of the peer review process by determining what
percentage of manuscripts rejected by the American Journal of
Surgery (AJS) were ultimately published or eliminated from
the core medical literature. We also reviewed the frequency with which
an author had published manuscripts on very similar subjects to the one
rejected by AJS.
METHODS
We retrospectively studied the bibliographic fate and frequency of
similar publications of all freestanding manuscripts that were
submitted to AJS by North American authors but were rejected
by AJS between January and December 1989. A freestanding
manuscript was one that had not been submitted to AJS through
one of the journal-affiliated societies. Relevant data regarding
rejected manuscripts were extracted from editorial office records.
Foreign manuscripts were excluded from the study because of the
presumed difficulty in tracking overseas manuscripts and
non-English-language publications.
The AJS is published monthly by Reed Elsevier Medical
Publishers, Belle Meade, NJ, with Hiram C. Polk, Jr, MD, University of
Louisville (Ky), serving as editor in chief. The AJS publishes
about 1226 pages of original articles per year and has approximately
17,000 subscribers.
To determine the outcome of each rejected manuscript, the scope of the
data collection process was defined beforehand.[2] Pertinent
information regarding only those manuscripts rejected during the year
1989 was retrieved from the AJS tracking files. These files
contained the letter of rejection and the reviewer critiques for each
manuscript. The main database of the National Library of Medicine
(MEDLINE) on CD-ROM (SilverPlatter Information Retrieval System Version
3.1 [SPIRS 3.11], SilverPlatter Information Inc, Norwood, Mass) was
searched for a listing of the principal author's publications from
1989 to 1992. MEDLINE is automated to retrieve all publications of the
principal author, regardless of whether that author is the primary
author. Manuscripts in which the principal author was subsequently
deleted may have been missed by this process.
A finding from the search was only defined as a "subsequent
publication" when (1) it contained a title as well as a list of
authors identical to those of the rejected manuscript or (2) there was
a slight variation in the authorship but the MEDLINE abstract contained
results and conclusions identical to those discussed in the reviewer
critiques. The rejected manuscript could not be compared with the
subsequently published manuscript because of the journal policy of
returning all rejected manuscripts and artwork to the corresponding
author with the letter of rejection. A comparison of the MEDLINE
abstract with data from the tracking file resolved questions regarding
the identity of the rejected manuscript with the one found in the
search.
The type of journal in which the manuscript was ultimately published
was recorded, as were the circulation numbers for each journal and the
Science Citation Index rankings for the journals that published the
highest number of rejected AJS manuscripts.[3] [4]
Data were then entered into a computerized database (Paradox 3.0,
Borland International, Scotts Valley, Calif) for further assessment of
the results. The delay to publication was calculated by the computer as
the interval between the date of rejection and the month and year of
subsequent publication.
As an additional assessment to determine how effective the review
process was in screening orginality, we determined the highest
occurrence of similar publications, ie, whether an author had published
similar subjects previously and subsequently to the one rejected by
AJS. The data were analyzed in the same fashion as were
"subsequent publications" but also arbitrarily included data 5
years before AJS submission as well as 3 years after
AJS rejection (1984 to 1992). Briefly, we carefully reviewed
the authors and titles of the MEDLINE search results manually and/or
with the computer for any publication that appeared to report a similar
subject as the one rejected by AJS. We were unable to compare
the actual publications line by line, except for one case with nearly
identical titles. A publication was only defined as "similar" when
the author appeared to publish the same or nearly identical information
(Bruce Squires, MD, PhD. CBE Views. 1993;16:98). The
AJS editorial office did not retain copies of the manuscripts
after rejection, so no comparison of manuscripts with those
subsequently published was possible other than for the title and
authors. Nearly identical titles and authors were presumed to represent
similar manuscripts.
To determine the reasons that each manuscript was deemed unacceptable
for publication in AJS, each rejection letter was reviewed.
All AJS reviewer critiques were summarized in the letter of
rejection from the editor. The AJS has access to a 58-member
editorial advisory board and nearly 200 ad hoc reviewers. Each negative
comment made by the reviewer or editor was defined as a reason for
rejection and was categorized as shown in the Table.
A rejection by AJS implied that the quality of the manuscript
was lower than that of accepted manuscripts, except for case reports,
which were rejected by AJS because of journal policy. We
assumed that a proportion of the manuscripts rejected by AJS
in 1989 could have also been rejected elsewhere. However, the previous
submission data were not available from authors or other publishers.
Each rejected AJS manuscript, except for case reports,
underwent peer review evaluation by three to five reviewers. The review
process was defined as any assessment of manuscript quality by an
assigned reviewer or the journal editor. The AJS reviewers
were aware of the identity of contributing authors, but contributing
authors were not aware of the identity of AJS reviewers.
RESULTS
A total of 347 freestanding manuscripts were submitted to AJS
in 1989, of which only 99 (29%) were accepted and 248 (71%) were
rejected. Of those rejected, 125 were classified as free-standing
manuscripts submitted by North American authors, and 77 (62%) of those
125 were not published by a core medical journal within 3 years
following rejection by AJS. Only 48 (38%) of the rejected
manuscripts submitted by North American authors were subsequently
published. Twenty-eight (58%) of the 48 were published within 1 year
of rejection, 15 (31%) within 2 years of rejection, and five (5%)
within 3 years of rejection. The mean delay to subsequent publication
was 17 months (range, 6 to 37 months). Approximately half (52%) of the
published manuscripts appeared in print with identical manuscript
titles; the remaining titles had changed. The sequence of authors had
changed in 12 (25%) of the manuscripts. The subject matter of the
rejected manuscript (eg, clinical or basic research reports) did not
influence the rate of subsequent publication, except in the case of
case reports. Twenty-one of the rejected manuscripts were case reports,
and more than half were published elsewhere. Thirty-eight percent
(27/72) of rejected clinical reports and 31% (10/32) of the remaining
rejected manuscripts (eg, experimental reports, operative techniques)
were published elsewhere.
The 48 subsequently published manuscripts appeared in 28 different
medical journals. More than half (52%) of the 48 manuscripts were
subsequently published in journals with a lower circulation than
AJS.[3] The journal circulation rates for another
eight manuscripts were unavailable. Six manuscripts (13%) were
extensively revised, reevaluated, and later accepted by AJS.
As stated previously, we also determined the Science Citation Index
journal rankings for the journals that published the highest number of
rejected AJS manuscripts. Eight (16%) of the 48 manuscripts
were published by a single general surgical journal that had a much
lower journal ranking than AJS (2286th vs 946th).[4]
Another 10 rejected manuscripts appeared in journals that were not
ranked. When categorized according to the type of journal, 26 (54%) of
the subsequently published manuscripts appeared in general surgical
journals, 14 (29%) appeared in specialty journals, five (10%)
appeared in state and local journals, and the remainder appeared in
general medical journals. Publication of manuscripts by surgical
journals was associated with an additional 7-month mean delay to
publication compared with manuscripts published by state and local
journals (Figure).
Thirty-five (28%) of the 125 authors of rejected manuscripts had
published articles very similar to the one rejected by AJS in
1989. Based on a percentage of the total number of articles on similar
subjects, 48% were published before 1989 and 52% were published
during and after 1989. Six authors in particular had a total of 139
similar publications during the 8-year period examined. The rejected
manuscripts of the three authors with the highest number of similar
publications were not subsequently published. One of these three
authors had 83 publications in the literature that expressed the same
theme 83 different ways. In three of the 83 publications, the titles
were so similar to the manuscript rejected by AJS that only
one word in the 18-word title was different, and four of the six
original authors were the same as for the rejected manuscript. These
three published articles were compared word for word. Although the
abstract, introduction, and discussion sections of each manuscript
varied slightly, the methods, results, tables, and figures were
virtually identical. There was also no reference to the similar works.
COMMENT
Peer review is an effective screening process to evaluate medical
manuscripts. This was demonstrated by the fact that the larger
percentage (62%) of manuscripts rejected by AJS remained
unpublished in the core medical literature. This finding is unique in
that it reflects more favorably on the review process than have prior
studies.[5] [6]
Prior studies of well-known and very well-respected medical journals
reported dramatically different rates. The Journal of Clinical
Investigation[5] and the New England Journal of
Medicine[6] both reported an 85% rate of subsequent
publication, and the British Medical Journal[7]
reported a 68% rate for manuscripts rejected in the 1970s. The
American Journal of Roentgenology[8] conducted a
similar MEDLINE search for manuscripts rejected in 1986 and reported a
64% subsequent publication rate. The differences between these reports
and our report may be attributed to the fact that two of these
studies[5,8] included manuscripts published within 4 to 6
years following rejection rather than the 3-year interval reported
herein. We believe that the rate of subsequent publication of
manuscripts rejected by AJS is not likely to increase
dramatically 4 to 6 years following rejection, as there appeared to be
a steady annual decline in the number of manuscripts published after
initial rejection.
It is possible that we underestimated our subsequent publication rate
because publication in journals not indexed by MEDLINE were excluded
from our study. However, higher publication rates reported by other
studies were similarly based on MEDLINE searches.[8] [9]
According to the National Library of Medicine,[10] those
journals that are included in Index Medicus or MEDLINE are
considered "the highest quality and most important biomedical and
health sciences journals published throughout the world." Because our
goal was to measure the effectiveness of the review process, our report
emphasized core medical journals with national exposure.
The prestige of the journal to which a biomedical manuscript is
originally submitted may account for the different rates of subsequent
publication between journals within the medical field. Our study showed
that more than half of the ultimately published manuscripts appeared in
journals with a lower journal circulation rate than AJS and
that the journal with the highest accumulation of subsequent
publications also has a much lower Science Citation Index journal
ranking than does AJS. The hierarchy of journals in medicine
lends itself to the hypothesis that many rejected manuscripts are
submitted to the next-lower-level journal until
acceptance.[11] In a recent report,[12] the
New England Journal of Medicine had the lowest number of prior
rejections of nine journals studied. Had we studied the previous
rejection pattern of manuscripts prior to their being submitted to
AJS, we may have found a high number of manuscripts that had
been rejected by prestigious journals.[13]
The initial acceptance rates of journals of various disciplines are
lower overall in the fields of human sciences (eg, medicine,
psychology, sociology) than in physical sciences (eg, astronomy,
physics, chemistry).[11] Differences between disciplines are
more thoroughly discussed elsewhere.[14]
Another outcome that we attempted to measure was similar publication
before and after AJS rejection. The fact that three of six
authors with the highest number of similar publications did not find
another journal willing to publish that rejected manuscript within 3
years of rejection suggests that the review process screened
originality of published data somewhat but still permitted similar
manuscripts to be published. A recent study of the British Medical
Journal[15] reported that duplicate publication was
increasing each year. One method to help prevent duplication is for
editors to perform a MEDLINE search on each title and author
tentatively accepted for publication. Many editors agree, however, that
the expense of such a procedure would be prohibitive and time-consuming
and that authors should be held accountable for the manuscripts they
submit for publication.
When a manuscript is accepted for publication following the review
process, it may not appear in the literature for several months. This
delay from submission to publication may prevent important scientific
research from reaching clinicians in a timely fashion and is partly due
to the structure of the review process, which involves additional time
for reviewers to assess manuscripts before publication. Geographically
restricted journals (ie, state and local) generally do not have
extensive review processes. This probably explains why the publication
delay in our study was shorter for state and local journals.
We conclude that the review process serves as a sieve and influences
whether manuscripts are published in core medical journals. The overall
implication is that the review process effectively prevents many
lower-quality or duplicate manuscripts from being published in the
indexed medical literature.
From the Department of Surgery, University of Louisville (Ky) School
of Medicine, and
The American Journal of Surgery, Reed
Elsevier Medical Publishers, USA, Belle Mead, NJ.
Presented at the Second International Congress on Peer Review in
Biomedical Publication, Chicago, Ill, September 9, 1993.
Reprint requests to the Department of Surgery, University of
Louisville, Louisville, KY 40292 (Dr Polk).
References
1. Bailar JC III, Patterson K. Journal peer review: the
need for a research agenda. N Engl J Med. 1985;310:654-657.
2. Williamson JW, Goldschmidt PG, Colton T. The quality of
medical literature: an analysis of validation assessments. In: Bailar
JC III, Mosteller F, eds. Medical Uses of Statistics.
Waltham, Mass: NEJM Books; 1986:370-391.
3. Ulrich's International Periodicals Directory
1992-1993. 31st ed. New York, NY: RR Bowker; 1993.
4. Garfield E. SCI Journal Citation Reports: A
Bibliographic Analysis of Science Journals in the ISI Data Base.
Philadelphia, Pa: Institute for Scientific Information; 1988;19.
5. Wilson JD. Peer review and publication: presidential
address before the 70th Annual Meeting of the American Society for
Clinical Investigation. J Clin Invest. 1978;58:1697-1701.
6. Relman AS. Are journals really quality filters? In:
Goffman W, Bruer JT, Warren KS, eds. Coping With the Biomedical
Literature Explosion: A Qualitative Approach. New York, NY: The
Rockefeller Foundation; 1978:54-60.
7. Lock S. A Difficult Balance: Editorial Peer Review
in Medicine. London, England: Nuffield Provincial Hospitals Trust;
1985.
8. Chew FS. Fate of manuscripts rejected for publication in
the AJR. AJR Am J Roentgenol. 1991;156:627-632.
9. Yentis SM, Lerman J. Publication of abstracts presented
at anaesthesia meetings. Can J Anaesth. 1993;40:632-634.
10. National Library of Medicine. List of Journals
Indexed in Index Medicus 1993. Bethesda, Md: US Dept of
Health and Human Services; 1993.
11. Abt HA. What happens to rejected astronomical papers?
Publ Astronomical Soc Pacific. 1988;100:506-508.
12. Whitman N, Eyre S. The pattern of publishing previously
rejected articles in selected journals. Fam Med.
1985;17:26-28.
13. Stossel TP. Beyond rejection: a user's view of peer
review. In: Bailar JC III, Angell M, Boots S, eds. Ethics and
Policy in Scientific Publication. Bethesda, Md: Council of Biology
Editors; 1990:268-272.
14. Garvey WD, Lin N, Nelson CE. Communication in the
physical and social sciences. In: Garvey WD, ed. Communication:
The Essence of Science. Elmsford, NY: Pergamon Press Inc;
1979:280-299.
15. Waldron T. Is duplicate publishing on the increase?
BMJ. 1992;304:1029.
Table of Contents
