How Well Does a Journal's Peer Review Process Function?
A Survey of Authors' Opinions
(JAMA. 1994;272:152-153)
Bobbie Jean Sweitzer, MD, David J. Cullen, MD, MS
Objective.--To evaluate the authors'
satisfaction or dissatisfaction with the peer review process of the
Journal of Clinical Anesthesia.
Design.--Anonymous questionnaires were sent to authors to
survey their opinions about specific aspects of the peer review
process. Authors were grouped by status of their manuscripts: AR
(accept with revision), RR (reject but may resubmit), and RO (reject
outright).
Participants.--Authors of unsolicited manuscripts submitted
in 1991 to 1992.
Main Outcome Measures.--Factors that determine authors'
satisfaction with the relevancy and benefit of peer review of their
manuscript.
Results.--Significantly more authors of AR manuscripts
responded to our survey than did authors of rejected manuscripts and
viewed the review process more favorably. Authors of AR manuscripts
were more satisfied with specific aspects of the review process, which
led to improvement in their manuscripts. More authors of RR manuscripts
believed that our review process improved subsequent manuscript
preparation than did authors of accepted manuscripts.
Conclusions.--The surveying of authors, important clients of
the peer review process, should guide change necessary to better serve
our authors and improve peer review.
(JAMA. 1994;272:152-153)
Although some knowledge, much
interest, and many publications about the peer review process are
accumulating, to our knowledge, the authors' evaluation of the process
has not been studied to any extent.[1] [2] In this climate of
total quality improvement, input from authors, important clients of
journals, should be valuable to editors and reviewers to improve the
review process. We solicited authors' opinions of the editorial,
review, and administrative processes of the Journal of Clinical
Anesthesia (JCA).
METHODS
Anonymous questionnaires were sent to authors of unsolicited
manuscripts submitted to the JCA in 1991 to 1992 for which the
review process was complete. Authors of letters to the editor, review
articles, editorials, and solicited manuscripts were excluded. We tried
to ascertain factors that contributed to the authors' satisfaction or
dissatisfaction with the review process by evaluating the six
characteristics in Table 1 using a
five-point ordinal scale. A rating of 1 was positive with a gradation to 5 being
negative, and a mean score was calculated. Authors chose from a list of
seven factors that contributed to their satisfaction or dissatisfaction
with the review process. Results are reported as the percentage of
responding authors selecting each comment. Authors were asked to
compare experiences with other journals' reviews to JCA's
process and whether our initial review process influenced the
preparation of subsequent journal articles.
All JCA manuscripts are assigned to one of four
categories: (1) accept outright (AO), (2) accept with revision (AR),
(3) reject but may resubmit after revision (RR), and (4) reject
outright (RO). The rejection rate is 70%. Although responding authors
remained anonymous, the questionnaires were coded by category of
manuscript disposition, enabling us to compare the responses between
authors of accepted and rejected manuscripts. Statistical analyses were
conducted using chi2 analysis and two-way analysis of
variance.
RESULTS
Of 209 authors polled, 95 (45%) returned the questionnaire. No
manuscripts from authors who participated in the survey were AO. Of 69
AR manuscripts, 67% of authors responded; 43% of authors of RR
manuscripts responded, but only 30% of authors of RO manuscripts
responded. Significantly more authors of AR manuscripts responded to
our survey (P<.001).
There was a trend toward favorable ratings by authors of
accepted manuscripts, but the differences were not statistically
significant. Authors of RO manuscripts were less likely to modify their
articles or find improvement in their manuscript as a result of the
review process. The best score was awarded by the authors of AR
manuscripts concerning the reviewers' specificity of comments. The
worst score was given by authors of RO manuscripts for modification of
their manuscript based on the review. The authors gave the manuscript
review process slightly positive to neutral ratings in all categories.
Authors of AR manuscripts chose more satisfactory comments than
other authors (Table 2). Eighty-nine
percent of authors of AR manuscripts, chose more than two characteristics that
satisfied their needs compared with 65% of authors of RR manuscripts
and 62% of authors of RO manuscripts. Twenty-seven percent of authors
of RO manuscripts chose no satisfactory factors, whereas all authors of
AR and RR manuscripts obtained some satisfaction from the review.
Negative comments were chosen by 46% of authors of AR
manuscripts compared with 74% of authors of RR manuscripts and 62% of
authors of RO manuscripts (Table 3).
Only 8% of authors of AR manuscripts noted more than two unsatisfactory
factors compared with 26% of authors of RR manuscripts and 19% of authors of
RO manuscripts. Authors were most unhappy about discrepancies between
reviewers and possible misinterpretation of material. Half of the
authors of AR manuscripts and 66% of the authors of RR manuscripts
believed that the process helped with subsequent manuscript
preparation. Only 20% of the authors of RO manuscripts believed that
they were helped by the review process in developing future
manuscripts.
COMMENT
This survey was designed to obtain information to improve our
manuscript peer review process. The primary purpose of peer review
should be to improve manuscripts and facilitate the dissemination of
accurate and valid knowledge for the ultimate benefit of patients.
Horrobin[3] stated that "most authors of articles
on the subject assume that the purpose of peer review is quality control. The
fundamental purpose must be consistent with that of medicine itself, to
cure sometimes, to relieve often, to comfort always." Editors can
poll their readership to determine whether journals are meeting the
readers' goals, a survey of one group of clients. Other clients are
the authors whose work is subjected to scrutiny by a process that is
under intense examination. Authors must be served by the process, and
editors must learn whether they are meeting the authors' needs.
Several conclusions derive from the survey. The administrative
and editorial assistance of the journal staff were highly regarded, by
authors of both rejected and accepted manuscripts. Reviewers'
guidelines that the review is a means of improving the manuscript and
more than just a grading effort may need additional emphasis. Should
reviewers be given more instructions? Editors have expressed concern
that this approach may limit comments, resulting in a narrow and
focused review. In our survey, widely divergent opinions in a review
were most dissatisfying to authors, although some editors prefer such
critiques so that all aspects of the manuscript are
considered.[4] [5] The editor or another reviewer acts as
"tie-breaker" after assessing contrary opinions.
Aspects of the process may be viewed differently by authors than by
reviewers or editors. The review turnaround time, often cited as a
major cause of dissatisfaction,[6] received above-average
ratings by all categories of authors. Only half of all authors believed
that their manuscript had received a careful evaluation, which differed
from the opinions of the editor and reviewers.
Authors of AR manuscripts ranked specificity of comments by the
reviewers most favorably, perhaps because a detailed critique of the
manuscript improved it for publication. Authors of RO manuscripts
reported the highest percentage of unaltered manuscripts being
resubmitted to other journals, but the results are unknown. Authors of
RO manuscripts rated manuscript improvement unfavorably, consistent
with the findings of Lock and Smith[7] that only 20% of the
136 manuscripts rejected by the British Medical Journal and
published elsewhere were altered prior to publication.
Limitations of the study mandate that caution be exercised when
interpreting the results. There was a delay between the time of
manuscript review and survey, necessary because we surveyed authors
whose work had completed the review process. Authors' memories and
opinions may have changed with time. We did not send separate
questionnaires for each of two reviews; therefore, the respondents may
have been biased toward a single review while offering no insight on
the companion review. Because the surveys guaranteed anonymity, we
could not compare the actual review with the author's perception of
that review nor could we evaluate individual reviewers based on the
satisfaction or dissatisfaction of the author. It would have been most
interesting to compare the editor's opinions of reviewers' work with
those of the authors. This was impossible given the anonymity of the
questionnaire.
As shown by Garfunkel et al,[2] the rate of returned
questionnaires is related to disposition of the manuscript. Authors of
AR manuscripts were more responsive than authors of RR manuscripts, who
responded more than authors of RO manuscripts, probably because the
experience of the authors of AR manuscripts was more positive. The
response rate of 45% may have resulted from the delay between the
review and the survey.
Until the 18th century, peer review of scientific articles did
not exist. Even well into the 20th century, peer review as we know it
today was the exception rather than the rule.[8] Journals
often followed the newspaper format of carrying news as well as
opinions and employing correspondents to cover medicine. There was no
discernable movement to establish editorial peer review, contrasted
with the history of peer reviewing grant applications, which developed
very carefully.[8] Peer review of manuscripts became
specialized when editors lacked the expertise to make decisions in very
specific fields.[8] [9]
The initial aim was to improve manuscripts, yet peer review is often
perceived as a "hurdle to get over" rather than an opportunity
to obtain advice and assistance from colleagues. Few studies have
looked at the process of peer review of medical journals.
Academic advancement, research funding, and institutional reputations
depend on peer reviewed publications as a barometer of quality
work.[4,6] Perhaps the current application of peer review
has outstripped its initial aims and abilities.
From the Department of Anesthesia, Massachusetts General Hospital,
Boston.
Presented in part at the Second International Congress on Peer Review
in Biomedical Publication, Chicago, Ill, September 10, 1993.
Address correspondence to the Department of Anesthesia, Massachusetts
General Hospital, Fruit Street, Boston, MA 02114 (Dr Cullen).
References
1. Guarding the guardians: research on
editorial peer review: selected proceedings from the First
International Congress on Peer Review in Biomedical Publication.
JAMA. 1990;263:1317-1441.
2. Garfunkel JM,
Lawson EE, Hamrick HJ, Ulshen MH. Effect of acceptance or rejection on
the author's evaluation of peer review of medical manuscripts.
JAMA. 1990;263:1376-1378.
3. Horrobin DF. The
philosophical basis of peer review and the suppression of innovation.
JAMA. 1990;263:1438-1441.
4. Bailar JC,
Patterson K. Journal peer review: the need for a research agenda.
N Engl J Med. 1986;312:654-657.
5. Scott WA.
Interreferee agreement of some characteristics of manuscripts submitted
to the journal of personality and social psychology. Am
Psychol. 1974;29:698-702.
6. Rennie D, Knoll E.
Investigating peer review. Ann Intern Med.
1988;109:181.
7. Lock S, Smith J. Peer review
at work. Scholarly Pub. 1986;17:302-314.
8. Burnham JC, Patterson K. The evolution of editorial peer review. JAMA. 1990;263:1323-1329.
9. Ingelfinger FJ. Peer review in
biomedical publication. Am J Med. 1974;56:686-692.
Table of Contents
