Is There Gender Bias in JAMA's Peer Review Process?
(JAMA. 1994;272:139-142)
Julie R. Gilbert; Elaine S. Williams; George D. Lundberg, MD
Objective.--To assess whether manuscripts received by
JAMA in 1991 possessed differing peer review and manuscript
processing characteristics, or had a variable chance of acceptance,
associated with the gender of the participants in the peer review
process.
Design.--Retrospective cohort study of 1851 research
articles.
Setting.--JAMA editorial office.
Participants.--Eight male and five female JAMA
editors, 2452 male and 930 female reviewers, and 1698 male and 462
female authors.
Main Outcome Measure.--Statistically significant gender
bias.
Results.--Female editors were assigned manuscripts from
female corresponding authors more often than were male editors
(P<.001). Female editors used more reviewers per manuscript
if sent for other review. Male reviewers assisted male editors more
often than female editors, and male reviewers took longer to return
manuscripts than did their female counterparts (median, 25 vs 22 days).
Content reviewer recommendations were independent of corresponding
author and review gender, while male statistical reviewers
recommended the highest and lowest categories more frequently than
did female statistical reviewers (P<.001). Manuscripts
handled by female editors were rejected summarily at higher rates
(P<.001). Articles submitted to JAMA in 1991 were
not accepted at significantly different rates based on the gender of
the corresponding author or the assigned editor
(P >.4).
Conclusions.--Gender differences exist in editor and
reviewer characteristics at JAMA with no apparent effect on
the final outcome of the peer review process or acceptance for
publication.
(JAMA. 1994;272:139-142)
PEER REVIEW of scientific journals would appear to
be about three centuries old,[1] yet there is great variation
and a lack of uniformity in the ways in which the process has become
institutionalized at various journals. In A Difficult Balance:
Editorial Peer Review in Medicine, Steven Lock[2] offers
the first comprehensive critique of the peer review process and raises
the awareness of the potential biases embedded in the system. Many
attending the first Peer Review Congress came in search of consensus
statements and peer review procedure guidelines to reduce systematic
bias.[3]
A review of the literature provides evidence that publication bias
exists in many forms. The biomedical peer review literature focuses on
the tendency to publish based on the direction or strength of research
findings.[4] Chalmers et al[5] have identified
publication bias in three stages of the peer review process. Gender
bias, the differential handling of a manuscript based on the gender of
the author, editor, or reviewer, may be a component of the second, or
publication, phase. Conventional wisdom maintains that gender bias has
been common in many aspects of professional and scientific life during
this century. In response, several humanities and social science
journals have altered their peer review policies to blind reviewers to
authors' identities.[6] [7] While blinding is feasible in
biomedical publication,[8] some believe it may inflict a cost
by replacing individuality with uniformity and by showing a distrust of
reviewer impartiality.[9]
Scientific-based gender studies have covered various aspects of the
peer review process. These studies have focused on the disproportionate
use of male reviewers,[10] collaborative patterns among
authors,[11] and the association between reviewer gender and
reviewer recommendations.[12] [13] Others have cited variation
among reviewer recommendations without considering reviewers' gender.
Another line of research examines the collaborative patterns of authors
by gender.[14]
We are not certain when JAMA began peer review on a regular
basis, but we know that it was several decades ago. Our current
policies and practices have evolved from trial and error, from adopting
other journals' procedures, such as those of the International
Committee of Medical Journal Editors, and from the 1989 Peer Review
Congress. We were interested in finding out whether gender differences
exist among the various participants in our peer review process and, if
so, the nature and effect of the differences so that a proper remedy
could be formulated. Thus, we set out to investigate whether the gender
of the corresponding author, the assigned editor, or the reviewers had
an effect on, or association with, the peer review, manuscript
processing, or acceptance rates of research articles submitted to
JAMA in 1991.
SUBJECTS AND METHODS
Subjects
Using the JAMA manuscript tracking database system, we
conducted a retrospective cohort study of all Original Contributions,
Preliminary Communications, Clinical Investigations, and Brief Reports
submitted for publication in JAMA in 1991. By the time of the
study, the manuscripts had been processed and final decisions made.
We hypothesized that all participants in JAMA's peer review
process would act in a gender-neutral way. This study was designed to
test whether the gender of the corresponding author, the assigned
editor, or the reviewers was associated with (1) manuscript assignment
to the editors, (2) the editor's ratings of reviewer performance, (3)
the editor's selection of reviewers, (4) content reviewer
recommendations, (5) statistical reviewer recommendations, (6) reviewer
turnaround times, (7) manuscript rejection rates without other review,
and (8) manuscript acceptance rates.
The following data were extracted from the database and tabulated by
gender, when applicable: research articles received in 1991; the
corresponding author; the assigned editor; reviewer(s), if any,
requested; reviewer recommendations; turnaround times from the date the
manuscript was sent for review until the requested review was received;
editor ratings of reviewer performance; and final decisions (acceptance
or rejection). Genders were determined for the corresponding author,
the editor handling the manuscript, and reviewers based on their first
names or through personal knowledge. Ambiguities in gender assignment
were resolved by contacting the authors or their staff by telephone.
Reviewer recommendations were recorded in one of the following
categories: accept as is, accept if suitably revised, revise and
reconsider, or reject. Borderline recommendations were assigned a
single response through a randomized process. Editors rated
reviewers on a scale of 1 through 5 (with 1 indicating poor; 2, fair;
3, good; 4, very good; and 5, excellent).
Corresponding authors were studied because we assumed that, in addition
to general differential treatment based on the authors' sex, gender
bias was more likely to occur in situations in which contact, verbal or
written, had occurred among the participants in the peer review
process. Corresponding authors served as a good proxy for lead authors,
as in many cases they were the same.
Statistical Analysis
Hypothesis testing was conducted with computer (Microsoft Excel 4.0 for
Windows) software. Reviewer recommendations and manuscript acceptance
rates were analyzed with use of chi2 tests for
independence. Manuscript assignments, rejection without other review,
and editor's selection of reviewers were analyzed with use of
z tests for proportional data. Average reviewer ratings and
turnaround times were assessed with use of two-sided t tests.
Conventional levels of significance (P<.05) were used in all
hypothesis testing.
RESULTS
Editor Characteristics
Table 1 shows the characteristics of editors who
handled research articles at JAMA in 1991. On average, male
editors were 12 years older than female editors and had 2 years
additional work experience at JAMA. Although most male editors
were employed full-time and females were employed part-time, female
editors handled almost twice the number of manuscripts as their male
counterparts (median, 200 vs 126 articles). Articles received by
JAMA were assigned by the deputy editor to an editor based on
the subject matter and the editor's area of expertise. Male editors
were assigned 800 manuscripts (80%) from male corresponding authors
and 204 manuscripts (20%) from female corresponding authors, while
female editors were assigned 605 (71%) and 242 (29%) manuscripts,
respectively. Female editors were assigned manuscripts with female
corresponding authors significantly more often than were male editors
(P<.001). On average, male and female editors used similar
numbers of reviewers per manuscript; however, of manuscripts sent out
for other review, those handled by female editors were seen by more
reviewers (mean, 3.3 vs 2.9 reviews).
Reviewer Characteristics
Table 2 shows the characteristics of reviewers
providing guidance on manuscripts considered in this study. Male
reviewers assisted male editors more often than female editors
(P<.001), while female reviewers were as likely to review
manuscripts handled by a male or female editor. Of 3326 editor ratings
of reviewers' performance, 3285 (99%) were available for study. Data
loss occurred when the editor did not explicitly give an assessment or
when the reviewer failed to return the manuscript or returned it
unreviewed. Under some circumstances, a reviewer may have critiqued a
manuscript on multiple occasions but received an overall rating on a
given manuscript; thus, the total number of editor ratings (n=3326) is
slightly less than the number of manuscript reviews (n=3382). Female
editors rated reviewers lower than did male editors, irrespective of
reviewer gender (P<.001). Male reviewers took significantly
longer to return manuscripts than did their female counterparts
(P<.001).
Of 3382 reviewer recommendations, 3188 (94%) were available for
study. Data loss occurred because some reviews were not on file at
the time of study and some reviewers did not explicitly state a
recommendation. We assume that these data are independent of gender.
The content reviewer recommendations by author and reviewer gender are
shown in Table 3. For content reviews, the gender of
the reviewers and the corresponding authors had no significant
association with the recommendation provided by the reviewer
(P >.05). Content reviewers had a higher rejection
rate with authors of the opposite sex. The statistical reviewers'
recommendations by author and reviewer gender are shown in Table 4. Unlike the content reviewers, the
recommendations given by the statistical reviewers differed according to
reviewer gender (P<.001). Male statistical reviewers recommended the
highest and lowest categories, accept as is and reject, at higher rates
than their female counterparts (P<.001).
Manuscript Characteristics
Table 5 traces the fate of the
1851 research articles submitted for publication in JAMA in 1991.
Manuscripts with male corresponding authors fared worse initially by
editors of both sex, although not significantly so (P=.055).
Female editors rejected manuscripts summarily at higher rates than did
male editors (P<.001). Manuscripts receiving further review
experienced similar acceptance and rejection rates. No significant
difference in overall acceptance rates was exhibited across author
gender and editor gender combinations (P>.40).
COMMENT
In this study, we set out to determine whether gender bias exists in
JAMA's peer review process. Specifically, were manuscripts
treated differently in the publication process based on the gender of
the corresponding author, the assigned editor, or the reviewers?
We found that male editors handling research articles submitted to
JAMA in 1991 were older, were employed full-time, and had more
years of experience yet carried a lighter manuscript caseload than the
female editors. Owing to the small sample size, it is difficult to
attribute these differences in caseload to gender, age, years of
experience, employment status, or competition from other assignments at
JAMA.
Because the editor handling the manuscript has sole power to reject at
any time, manuscript assignment by the deputy editor serves as a
critical step in getting an article to publication. The editor handling
the manuscript decides whether to reject the manuscript summarily,
chooses the reviewers, interprets the reviewers' comments, and
presents their views at a manuscript meeting. Each of these variables
leaves room for subjective judgments and variable treatment of a given
manuscript. Our study showed that female editors were assigned
manuscripts with female corresponding authors at higher rates than were
male editors (P<.001). Possible confounding factors may be
the fact that gender ratios differ according to specialties within the
research community. For example, women account for a larger percentage
of the nursing and pediatrics workforce, and the JAMA editors
handling manuscripts from these fields were also female. One can assume
that the greater representation of women in these fields implies that
they submit manuscripts at higher rates than in other medical fields.
Male editors utilized male reviewers more often than female reviewers,
while female editors used reviewers of both genders at equal rates.
This result, as in the initial manuscript assignments to the editors,
may be an artifact of the differences in gender ratios among
specialties in the research community. Female editors rated reviewers
lower than did male editors, irrespective of reviewer gender
(P<.001). Remember that the sample sizes involved were
small; the results may simply reflect different working styles of the
editors, as no standard criteria were used to assess reviewer
performance, and the editor's judgment determined the rating given.
Male reviewers took significantly longer to return manuscripts than did
female reviewers (P<.001).
The recommendations provided by reviewers play an important role in
manuscript selection, as they provide the editor with an additional
source of evaluation in the decision whether to continue support for a
given manuscript. For content reviews, the gender of the reviewers and
the corresponding authors had no significant association with the
recommendation provided by the reviewer (P >.05). A
key argument raised by proponents of a blind peer review process is
that the disclosure of the author's identity will bias or influence
the reviewer's assessment of the manuscript in what is ideally a
purely objective process. We found no evidence of gender bias at this
critical stage in the peer review process.
Statistical reviewers showed significant differences in recommendations
provided. Male reviewers recommended the highest and lowest categories
(accept as is and reject) at higher rates than the female reviewers,
while female reviewers more frequently recommended the two categories
of revision. Any interpretation of these results should be tempered by
the small number of purely statistical reviewers (n=6) used to evaluate
manuscripts received in 1991. Because of the large volume of
manuscripts reviewed by the statistical reviewers, editors are familiar
with each reviewer's working style. In this case, it may be argued
that an open review process serves as a valuable source of information
by providing a context for the reviewer's recommendations.
This study shows that gender differences exist in editor and reviewer
characteristics. The essential question is whether the process affects
the outcome: Does the publication process at JAMA influence
the fate of manuscripts according to the gender of the corresponding
author, the editor, or the reviewers?
We found that female editors rejected manuscripts summarily at higher
rates than did male editors (P<.001). Without a gold
standard of manuscript quality, it is difficult to draw definitive
conclusions; however, the evidence suggests that either female editors
are assigned manuscripts of lower quality, irrespective of the
corresponding author's gender, or female editors' higher rejection
rates without other review shows a difference in the level of criticism
between male and female editors. The larger number of reviewers used
per manuscript for those receiving other review (3.3 vs 2.9 for female
and male editors, respectively) indicates that both factors may play a
role. Final manuscript acceptance rates did not differ across author
gender and editor gender combinations (P >.40).
We found no evidence of gender bias in content reviewer recommendations
or final acceptance rates. However, many factors that may influence the
final decision about a manuscript required data that are beyond the
scope of this study. These data include manuscript load at the time of
the final decision, priority or originality of the topic, length of the
manuscript, institutions with which the authors are affiliated, whether
the study found negative or positive results, study sample size,
whether the manuscript was solicited, and inventory level for that
category at the time of presentation at a manuscript meeting.
From the Departments of Public Policy and Biological Sciences,
Stanford University, Palo Alto, Calif (Ms Gilbert), and
JAMA,
Chicago, Ill (Ms Williams and Dr Lundberg).
Presented in part at the Second International Congress on Peer Review
in Biomedical Publication, Chicago, Ill, September 11, 1993.
Reprint requests to JAMA, 515 N State St, Chicago, IL 60610
(Dr Lundberg).
References
1. Kronick DA. Peer review in 18th-century scientific
journalism. JAMA. 1990;263:1321-1322.
2. Lock S. A Difficult Balance: Editorial Peer Review
in Medicine. Philadelphia, Pa: ISI Press; 1986.
3. Rennie D. Editorial peer review in biomedical
publication: the first international congress. JAMA.
1990;263:1317.
4. Dickersin K. The existence of publication bias and risk
factors for its occurrence. JAMA. 1990;263:1385-1389.
5. Chalmers TC, Frank CS, Reitman D. Minimizing the three
stages of publication bias. JAMA. 1990;263:1392-1395.
6. Kronik JW. Editor's column. Publications of the
Modern Language Association of America. 1990;105:192-196.
7. Gannon L, Luchetta T, Rhodes K, Pardie L, Segrist D. Sex
bias in psychological research: progress or complacency? Am
Psychol. 1992;47:389-396.
8. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The
effects of blinding on the quality of peer review: a randomized trial.
JAMA. 1990;263:1371-1376.
9. Sharp DW. What can and should be done to reduce
publication bias? the perspective of an editor. JAMA.
1990;263:1390-1391.
10. Sojka RE, Mayland HF, Gbur EE. American Society of
Agronomy member experiences and perceptions of the peer
reviewing-editing process. In: Kral DM, ed. Research Ethics,
Manuscript Review and Journal Quality. Madison, Wis: ACS
Miscellaneous Publication; 1992:35-53.
11. Knoppers A. Productivity and collaborative patterns of
physical educators. Res Q Exerc Sport. 1989;60:159-165.
12. Hicks C. Of sex and status: a study of the effects of
gender and occupation on nurses' evaluations of nursing research.
J Adv Nurs. 1992;17:1343-1349.
13. Lloyd ME. Gender factors in reviewer recommendations
for manuscript publication. J Appl Behav Anal.
1990;23:539-543.
14. Siegelman SS. Assassins and zealots: variations in peer
review. Radiology. 1991;178:637-642.
Table of Contents
