Effects of Peer Review and Editing on the Readability
of Articles Published in Annals of Internal Medicine
(JAMA. 1994;272:119-121)
John C. Roberts, MD; Robert H. Fletcher, MD; Suzanne W.
Fletcher, MD
Objective.--To measure the effect of the peer review
and editorial processes on the readability of original articles.
Design.--Comparison of manuscripts before and after the peer
review and editorial processes.
Setting.--Annals of Internal Medicine between March 1
and November 30, 1992.
Manuscripts.--One hundred one consecutive manuscripts
reporting original research.
Measurements.--Assessment of readability by means of two
previously validated indexes: the Gunning fog index (units of
readability in the fog index roughly correlate to years of education)
and the Flesch reading ease score. Each manuscript was analyzed for
readability and length on receipt and after it had passed through the
peer review and editorial processes. Text and abstracts were analyzed
similarly but separately. Mean readability scores were compared by
two-tailed t tests for paired observations.
Results.--Mean (+/-SD) initial readability scores of
manuscripts and abstracts by the Gunning fog index were 17.16+/-1.55
and 16.65+/- 2.80, respectively. At publication, scores were
16.85+/-1.42 and 15.64+/- 2.42 (P=.0005 and
P<.0001 for before-after differences, respectively). By
comparison, studies of other print media showed scores of about 11 for
the New York Times editorial page and about 18 for a typical
legal contract. Similar changes were found for the Flesch scores. The
median length of the manuscripts increased by 2.6% and that of the
abstracts by 4.2% during the processes.
Conclusions.--The peer review and editorial processes
slightly improved the readability of original articles and their
abstracts, but both remained difficult to read at publication. Better
readability scores may improve readership.
(JAMA. 1994;272:119-121)
PEER-REVIEWED general medical journals are
intended mainly for use by clinicians, who must read and understand
journal articles if they are to use the information to improve the care
of patients.
Clinicians who read reports of original research must overcome many
obstacles.[1] With increasing specialization in medicine and research, it is difficult for researchers to communicate with a broad,
clinical audience. The concepts and language of modern clinical
research may be unfamiliar. Also, articles may not be directly relevant
to the reader's practice. In any case, readership of even the most
respected journals is not high--20% to 40% of subscribers--and may be
declining.[2]
One barrier to reading that is potentially improvable is the
readability of articles. We therefore studied the readability of
original research articles published in a large, peer-reviewed clinical
medical journal, Annals of Internal Medicine, to answer two
main questions. First, how readable are the articles submitted to
Annals? Second, do peer review and editing improve the
readability of these articles?
MATERIALS AND METHODS
This study was carried out at Annals of Internal Medicine, a
semimonthly clinical journal published by the American College of
Physicians. Its circulation is approximately 100,000, and it
receives about 2400 manuscripts a year, of which it publishes about
16%, all original research articles.
The peer review and editorial processes of Annals are similar
to those of most major journals. By the time a manuscript reaches
publication, it has been critiqued by at least one editor-in-chief, a
deputy editor, at least one associate editor, at least two reviewers, a
statistician, and at least one copy editor. Authors also revise their
manuscripts at least once in the process.
Readability is defined as "the ease of understanding or comprehension
due to the style of writing."[3] Quantitative measurements
of readability began in the 1920s, and readability formulas have been
applied to newspapers, magazines, textbooks, grant applications, and
other printed media.[3] [4]
The most commonly used and well-validated readability formulas
are those of Gunning[5] and Flesch[6] (Table 1). Both indexes are based on word and sentence
length. The Gunning fog index[5] is calculated as follows:
reading grade level=0.4x(mean sentence length+ percentage of words
more than two syllables). The Gunning fog index is scaled to reflect
the years of education of a reader who can "easily read and
comprehend the piece of writing." The Flesch reading ease
index[6] is calculated as follows: reading ease
score=206.835-[0.846x(number of syllables per 100
words)]-[1.015x(average number of words per sentence)]. The
Flesch score uses a 100-point scale, with 100 representing easiest
readability and 0, most difficult.
Although neither index is an ideal measure of readability,
comprehension, or readership, both have been studied many times for
reliability and validity (using measures of reading speed, expert
judgment, readership, and comprehension, with correlations ranging from
.62 to .90).[3,4] [7] Little research has been done regarding the validity of readability indexes in technical writing outside the
field of psychology.[4]
From March 1, 1992, to March 1, 1993, for all manuscripts of original
research ultimately published in the journal, computer disks of the
original submitted versions were obtained from the authors. After the
manuscripts passed through the entire peer review and editorial
processes, the published versions also were entered on computer disks.
Original and final versions were matched. Although associate editors,
copy editors, reviewers, and authors were aware that research studies
were being carried out, only the editors-in-chief were aware of the
goals of this study.
Using a computer program (RightWriter 4.0),[8] we compared readability scores of the original and final versions. In addition, all
versions were analyzed for average sentence and word lengths. Abstracts
and manuscripts were analyzed separately, and all bibliographies and
appendixes were removed before analysis. Data were collected on
Epi-Info 5.0 software[9] and were analyzed by
means of descriptive statistics and paired t tests on SAS, release
6.03.[10] Data are reported as means (+/-SD) except for manuscript lengths, which are reported as medians (range), since the
lengths were not normally distributed. The statistical significance of
before-after comparisons were analyzed by two-tailed t tests
for paired observations.
RESULTS
A total of 111 manuscripts were analyzed. The median length of
submitted manuscripts was 3057 words (range, 784 to 7844 words), and
the length of original abstracts was 262 words (range, 57 to 525
words). Lengths at publication were 3138 words (927 to 6412 words) and
273 words (135 to 463 words), respectively.
Readability scores are presented in
Table 2.
According to both indexes, abstracts and manuscripts were "very
difficult" to read on receipt at the editorial offices, and although
their readability did improve, they remained in the "very
difficult" range at publication.
Because medical vocabulary includes many long words and because both
the Gunning and Flesch formulas are based on sentence length and word
length, we also analyzed the abstracts and manuscripts for those
factors. Abstract sentence length was 19.30+/-5.16 words initially and
16.95+/-3.35 words at publication, a change of 2.35 words
(P<.00001) The mean number of syllables per word in the
abstracts was 1.91+/-0.15 initially and at publication. Manuscript
sentence length was 22.01+/-2.66 words initially and 21.01+/-2.38 words
at publication, a difference of 1.0 words (P<.00001). The
mean number of syllables per word was 1.85+/-0.09 initially and
1.85+/-0.09 at publication (P=.80).
COMMENT
Readability, as measured by these two well-validated indexes, improved
after abstracts and manuscripts passed through the peer review and
editorial processes. However, at publication, both abstracts and
manuscripts remained in the range of "very difficult" to read.
We could find no previous study of medical journal articles'
readability. Studies of readability in other contexts have shown
correlations between readability, as measured by these indexes, and
readership and comprehension.[3,4] Readability may be an
important factor in physicians' ability to use information in medical
journals to improve their care of patients.
These formulas test only readability. Readership, comprehension, and
retention also depend on other factors: content, format, logical
organization, redundancy, and reader motivation.[11]
Readability has been shown to correlate with the level of readership
and retention of newspaper articles. Readability also may promote
retention of information. In one experiment, a newspaper published two
editions that were identical except for articles' readability scores.
Follow-up surveys of readers showed higher levels of readership and
retention of news in the more readable edition.[12] However,
it is not certain that better readability leads to better readership
when the audience is physicians and the media are journals.
Three caveats should be made regarding our findings. First, we studied
only articles that were published, and they may have represented the
best writing among all submitted papers. Thus, the improvement might be
expected to be small. Second, the editors-in-chief were aware of the
goals of this study, and that knowledge may have caused them to exert
unusual efforts to improve readability. Even so, little effect was
seen. Third, and most important, although both indexes have been
validated in other media, no readability formula has been validated in
medical writing.[3,4] Both the Gunning and Flesch scores are
based on sentence and word length. One might then conclude that these
indexes are not valid for medical writing, since medicine's jargon
includes words from Latin and Greek (eg, "erythema") instead of
Anglo-Saxon (eg, "red skin"), as well as neologisms (eg,
"esophogastroduodenoscopy"). But it is just such characteristics
that editors criticize as detrimental to readability.[13]
Furthermore, the effect of acronyms on the readability of medical
writing may be minimized by these indexes, which treat them as
one-syllable words.
Even though these indexes have not been validated with medical
prose, authors and editors might want to use shorter and more familiar
words. Writing coaches have advised shorter words, and mass media
research demonstrates that the two most important elements of
readability are simplified vocabulary and decreased sentence
complexity: short words and short sentences
(Table 3).[4,13]
In our study, median manuscript length increased slightly during the
peer review and editing processes. Long manuscripts seem to contradict
a well-documented trend toward shorter articles and books, especially
in recent years.[13] Long manuscripts also may partly explain
the popularity of the structured abstract and such publications as
ACP Journal Club and Journal Watch.
In recent years, physicians have been receiving communications from a
rapidly growing number of media: peer-reviewed journals, other
journals, television, audiotapes, videotapes, computers, and the mass
media.[14] The burden of media on the physician
is great, and if peer-reviewed journals are to remain leaders in sustaining
excellent medical practice, they must tailor their editorial elements
to meet the needs and desires of their biggest audience: practicing
physicians. Readability is one element.
From the Editorial Offices,
Annals of Internal Medicine,
Philadelphia, Pa (Drs Roberts, R. Fletcher, and S. Fletcher); and the
Division of General Internal Medicine, Johns Hopkins Bayview Medical
Center, Baltimore, Md (Dr Roberts). Drs R. Fletcher and S. Fletcher are
now with the Division of Ambulatory Care and Prevention, Harvard
Medical School and Harvard Community Health Plan, Boston, Mass.
Presented in part at the Second International Congress on Peer Review
in Biomedical Publication, Chicago, Ill, September 9, 1993.
We gratefully acknowledge the assistance of Andrew Langman for his
technical contributions and Grace Lobb and Tori Ransome for their
efforts in collecting and maintaining the manuscripts on computer
files.
Address correspondence to Division of General Internal Medicine, B-2
North, Johns Hopkins Bayview Medical Center, 4940 Eastern Ave,
Baltimore, MD 21224 (Dr Roberts).
References
1. Fletcher SW, Fletcher RH. Responsibilities of
medical journals to readers. J Intern Med. 1992;232:223-228.
2. Media-Chek/Apex: A Study of Medical Journal
Readership and Advertising Page Exposure--American College of
Physicians. Princeton, NJ: Healthcare Communications; 1993.
3. Klare GR. The Measurement of Readability.
Ames: Iowa State University Press; 1963.
4. Severin WJ, Tankard JW. Communication Theories:
Origins, Methods, and Uses in the Mass Media. 3rd ed. New York,
NY: Longman Group Ltd; 1992.
5. Gunning R. The Technique of Clear Writing. New
York, NY: McGraw-Hill International Book Co; 1952.
6. Flesch RF. A new readability yardstick. J Appl Psychol. 1948;32:221-233.
7. Stone G. Examining Newspapers: What Research
Reveals About American Newspapers. Newbury Park, Calif: Sage
Publishers Inc; 1987.
8. RightWriter 4.0. Carmel, Ind: Que Corp; 1990.
9. Dean AG, Dean JA, Burton AH, Dicker RC. Epi-Info
5.0. Stone Mountain, Ga: USD Inc; 1990.
10. SAS Release 6.03. Cary, NC: SAS Institute
Inc; 1992.
11. Gray WS, Leary BE. What Makes a Book
Readable? Chicago, Ill: University of Chicago Press; 1935.
12. Swanson C. Readability and readership: a controlled
experiment. Journalism Q. 1948;25:339-345.
13. LaRocque P. Fuzziness: don't force readers to clear up
your writing. Quill. September 1993:36.
14. Wyatt J. Use and sources of medical journals.
Lancet. 1991;338:1368-1373.
Table of Contents
