Structural Transformations of the Sciences and the End of
Peer Review
(JAMA. 1994;272:92-94)
Horace Freeland Judson
THREE great structural transformations are now
taking place in the sciences. The structures are the
conditions, the building plan, the institutional relationships that
underlie and shape the way sciences are done today.
Transformations affect the structures radically yet
systematically--in ways that preserve many of the original elements
while creating a set of relationships fundamentally different.
Structural transformations are probably impossible to resist. Their
consequences are hard to predict. The first and most familiar of the
transformations we can think of as internal to research and publishing:
it comprises the declining standards and the growing, built-in tendency
toward corruption of the peer-review and refereeing processes. The
other two transformations we can describe as external to peer review
sensu stricto: they are the transition from exponential growth
of the sciences to a steady state, and the appearance and development
of electronic publishing and electronic collaboration more generally.
The three overlap and interact. The one sure prediction is that they
profoundly affect the nature of peer review in scientific research and
publication: by the first decade of the new millenium, although the
term peer review may persist vestigially, the practices
denoted will be unrecognizably different from what we know today.
The history of peer review and refereeing has been covered by a
number of historians and social
scientists.[1]
[2] [3]
In sum, although peer review and refereeing seem rational, indispensable, and
immutable, the histories demonstrate that they are social constructs of
recent date. They are not laws of nature, nor of epistemology. They
have changed and evolved.
We all distinguish between peer review for grant applications and
peer review of manuscripts for scientific journals. Some scientists
protest this dual use of the term, wanting to keep peer review
for the grant process and to speak of refereeing of journal
submissions. This is more than a quibble, for the histories of the two
forms differ greatly. Yet today certain basic similarities prevail, and
we cannot consider one without the other: they are both intrinsically
part of the system of science we now practice. Both are locked into the
way we distribute recognition, money, hierarchical position, and power
in science in the United States. Crucial to that system, both are
methods evolved to protect the autonomy and self-regulation of the
sciences. Peer review and refereeing reserve judgment of scientists'
work exclusively to other scientists acting in the name of the
scientific community. Peer review shields the choice of research
projects or groups to support, the detailed allocation of funds, from
direct pressures from administrators, politicians, and the public who
are outside the system of science.
Consideration of the first transformation in the sciences begins with
the observation that peer review and refereeing are inherently
threatened by corruption. They are under pressure for several reasons,
of which the most basic is the contradiction that makes peer review
possible at all. The fact is, of course, that the persons most
qualified to judge the worth of a scientist's grant proposal or the
merit of a submitted research paper are precisely those who are that
scientist's closest competitors.[1] [4]
This contradiction is at the root of plagiarism. On a scale of
misconduct, the purloining of intellectual property may appear less
heinous than outright fabrication or falsification of data: after all,
fabricated or falsified data are untrue, are a betrayal of science
itself, while stolen data and ideas have at least a chance to be
correct, or why steal them and betray your colleagues? But what
plagiarism might seem to lack in gravity it more than makes up in
frequency. In several notorious cases in recent years, whole tables and
paragraphs have been lifted. The subtler form of piracy is the theft of
an idea, an insight, a conclusion, leaving your data and your language
untouched. Theft of intellectual property is by far the most common of
all forms of scientific misconduct.[5]
[6] [7]
[8]
The second inherent reason for the vulnerability of refereeing
and reviewing is fatigue: the systems are wearing out with time,
breaking down under pressure. As the number of scientists has increased
so vastly in the last 50 years, as specialties have multiplied and
journals so promiscuously proliferated, the familiar consequence has
been increased competition for funding. On the grants side, even as the
demands on the process have grown, the qualifications of participants,
and above all their dedication and enthusiasm, their morale, have sadly
waned. What was a high and interesting duty has become a wearisome
chore. So, too, with journal publication. Although top journals and
skillful editors can mitigate the consequences, referees are overworked
and increasingly regard reviewing as an endless, onerous chore.
Reviewers have reported that they may be working on as many as five or
six manuscripts from different journals at the same
time[9];
editors report that the quality of reviews they receive tends to vary
inversely with the seniority and standing of the
referees.[10]
Worse, as over the years an ever-smaller proportion of grants gets
funded and as the applications themselves, in the top quartile, are
more difficult to put in any reliable rank
order,[11]
politics become overt in the review process. Rivalries between
scientists, laboratories, and schools of thought emerge as palpable
factors. What began as a means of keeping external pressures at arm's
length has turned, to some extent at least, into a cockpit in which the
internal politics of the sciences are fought out.[4,6]
The most telling example of the effects of competition among
journals and their editors and publishers has been the progress of what
have come to be called the "hot news" publications--for example,
Science, Nature, and Cell. In the 1960s or
1970s, one could pick up a newly published article in one's field,
read it through, and judge with some confidence its credibility and
worth. Such judgment is less and less possible. The reason is in part
the increasing complexity of the subjects--cell biology, say, compared
with molecular biology. More important, though, is the pressure passed
from publishers to editors (in the case of Cell, these are the
same man) and on to would-be contributors to keep articles short and
assertive. Reports are condensed. Discussions and conclusions are
simplified. Qualifications and cautions are abbreviated or penciled
out. The use multiplies of that curious indicator, "(data not
shown)." The general scientific reader is baffled, even to
some degree misled.
However, the first outside readers to be affected by these practices
are, after all, the referees--which leads to the paradox that even as
the general reader depends more than ever on the assurance that some
particular article appears in a refereed journal, the referees
themselves are being deprived of the means to be confident in their
judgments. This problem is hidden but not trivial.
Growth is the root of the second great structural transformation of the
sciences. But the problem that now envelops us, conditioning everything
we do, is not of growth in the simple sense. Rather, it is the
transition of the sciences from exponential growth to the steady state.
In 1963, Price,[12]
who was a historian of science, published
a graph showing that scientific activity had been expanding at an
exponential rate for 300 years--indeed, that the number of scientists
and the output of scientific papers was doubling every 15 years. At
that rate, Price said, within another 100 years "we should have two
scientists for every man, woman, child, and dog in the population."
Like most assertions of malthusian limits, this one was taken seriously
by few. Yet the limits have caught up with us. In the steady state,
scientists and technicians will be recruited and trained at a rate
sufficient to replace those who die, retire, or quit, while research
funds will grow no faster than the inflation of the costs of personnel,
materials, and facilities at constant levels. The transition to a
steady state is already producing enormous systemic strains, some
obvious, some subtle.
For most scientists, the first signs of the transition are shortage of
funds and intensification of competition. Along with this, they feel
ever-increasing pressure from politicians and governmental agencies for
directed or targeted research: the present slogan is "national
needs." Parsimony and political interference: most of us read these
as characteristics of an abnormal situation, antithetical to doing
science in the mode that has proved so successful since the second
world war. One longs for a return to normality.
We shall never see a return to such a nostalgic normality. What we
hear, even from administrations that declare commitment to research and
high technology, is renewed emphasis on the practical, industrial,
technological exploitation of research--while any increases in funding
are at the margin.
The transition to the steady state has other concomitants, which I can
only touch on here. One is the internationalization of
research.[13]
[14]
[15]
[16]
Another comprises the ever greater and more
complex linkages between university and industry, particularly
conspicuous in the biological and biomedical sciences (H.F.J., and
X.Tong, PhD, unpublished data, March
1994).[17]
[18]
[19]
Increasingly, potential profit drives the direction of research. As
importantly, the linkages between academic laboratory and industry are
changing career structures. The young biologist today sees not one but
two career ladders, one academic and the other high-tech industrial;
the talented and ambitious soon realize that as they ascend they will
be able to step from one to the other, either
way.[20]
The aspect of the transition to the steady state that is most
subtle and most powerful, particularly in relation to peer review and
refereeing, is the growing emphasis on evaluation and accountability.
Peer review and refereeing, as noted, make up an institution for
evaluating research. The first characteristic of these evaluations is
that they are made by scientists. Second--and especially in review of
grant applications--evaluation is of inputs. But the emphasis in
advanced management these days is on evaluation of outcomes--the new
management shibboleth. In the federal government, managers of social
programs, of industrial policy, and of technological and scientific
activity of all sorts are learning to call for the evaluation of
outcomes. In industry, as managers become more sophisticated and
competitive in evaluating outcomes, and as relations with university
laboratories grow more intimate, from this side, too, academic research
will increasingly be judged by outcomes. For those in science,
evaluation of outcomes will mean evaluation of their work by
nonscientists, evaluation at end points rather than prospectively or in
mid-process, evaluation according to new criteria over which scientists
will have far less control.
The third great structural transformation of the sciences has only just
begun but is moving at astonishing speed: electronic publishing. This
change is, of course, taking place far more widely than in the
sciences. Its momentum has been dramatized by recent announcements from
one after another of the largest-circulation newspapers in the United
States: by the end of 1995, virtually every major daily will be
available to subscribers in an on-line version (Los Angeles
Times. August 5, 1993; D2, D4).[21]
But in the sciences the transformation holds remarkable promise.
In February 1993, Lederberg[22]
published in The Scientist the text of a talk he had delivered earlier
at an international conference of science editors. Lederberg welcomes
electronic publication. First, it offers the only possible remedy to
the problem of sorting out from the vast scientific literature all the
articles, but only the articles, that are directly relevant to the
individual's work. Second, it will allow one to record, retain, and
keep accessible one's own responses--notes, commentary, linkages,
inspired ideas--to those articles. Many journals, he said, will of
course continue to function best in their present form, on paper, to be
read on airplanes or trains or sitting in the garden. But any journal
should be available for electronic browsing and searching. Electronic
publication together with more sophisticated search algorithms may even
ease the problem of locating highly specific but elusive matters within
the vast literature, what Lederberg calls "the exquisite detail
needed to take the next step."[22]
Electronic publication can overcome two other grave problems that
ink-on-paper journals now present. The first, obviously, is the lag
time from submission to publication, at least insofar as this is a
product of backlog, of less-than-instantaneous communication between
editors, staff, referees, and authors, and of the mechanics of printing
and mailing. One publisher, for example, has announced plans by which,
starting in April 1994, papers in two of its physics journals will be
offered electronically as soon as they are accepted, thus getting them
to subscribers to the service fully 5 months before they appear in
print.[23] The second problem,
less obvious but more
important to maintenance of the fabric of science, is the pressure
already mentioned from editors on authors to condense, to simplify, and
to excerpt the data itself. Electronic publication can eliminate
forever that problematic notation "(data not shown)." (We will
still have "data not read.")
Indeed, we must imagine that remarkable interactive options will
develop for readers of scientific articles published electronically.
Lederberg, prophetically, called for dialogue, "a dialectic,"
between readers, journals, and authors, and he saw this as the greatest
long-term gain from electronic publishing.[22]
The dialectical mode is being built. Today several biomedical
journals publish electronically, including The Online Journal of
Current Clinical Trials--although these have not as yet proved
successful. More important is the rise of electronic preprint clubs.
For decades, groups in physics and in biology have used distribution of
preprints as a primary means of communication. Preprint clubs have two
characteristics: they are not refereed, but they are fast, often months
ahead of the eventual publication. Within the past couple of years, in
at least eight subdisciplines in the physics community the practice has
become to communicate a paper to an electronic bulletin board as soon
as it is submitted to a journal; the bulletin board makes abstracts
available not merely to a select mailing list but to all potential
readers, who can then call up full texts at
will.[24]
[25] Of
the electronic-mail archives, one Yale physicist was recently quoted as
saying that now, "The only thing I use journals for is looking back
for papers that came out before the bulletin boards
existed."[26]
The communities--they are being called
"collaboratories"--that are coming to use Internet in this and
related ways are, in aggregate, potentially numerous, international,
and highly active. Yet the number of scientists in any one such subset
may number a dozen, two score, 200. The members of such a group,
competing and simultaneously collaborating, are each and collectively
one another's peers, doing in effect their own reviewing, not blinded
or anonymously but openly and in a manner that concatenates publication
and responses.[27]
[28] Furthermore, such a group is likely
to be not hierarchical but egalitarian (Wall Street J.
December 9, 1993:1, 9).[20]
Thus, the time is soon coming when the act of publishing a report
electronically will be but a preliminary step. You the reader will be
able to scan not just the compressed references to other articles that
we now see in the notes but, if you wish, the abstracts of those
articles; you will be able to demand the referees' reports, perhaps
the raw data. Publication will invite open criticism, suggestions,
rebuttals; the article so published will be able to go through
revisions on the screen, with comments and old versions retained for
reference; when anyone calls up the article months later it will bring
with it indexes of all these responses and changes that have accrued,
available to anyone who wants to look. Corrections and retractions will
similarly be forever linked to the article itself.
I go into this detail about the potential of electronic
publication because we must recognize how revolutionary it really is.
We are not talking of the substitution of one medium for another, of
the replacement of the printed page by the screen, with everything
else--including editing, refereeing, and readers' correspondence--to
go on as before.
We cannot dismiss the problem of quality control in electronic
publication. Certain journals, at least, must adapt the speed and
potential openness of the network in some fashion to preserve--to
enhance--the values that can be added by skilled editors and thoughtful
expert commentators. These values are what make some journals
authoritative; they will continue to be desirable as one element of the
new dialectic of publication.
Editors of biomedical journals, at least when they publish
clinical research, clearly have the additional responsibility to be
authoritative in matters that will affect the treatment of patients.
They regard this responsibility as primary. At the same time, though,
biomedical journals and their editors are powerfully placed in the
social system of medicine. Both in teaching and in practice, this
social system is steeply hierarchical and in many respects
authoritarian. At many levels of the system, then, the transformation
brought in by electronic publishing will seem threatening, to be
resisted. Yet profound change is inevitable. Research processes
themselves evolve with the emergence of new questions and technologies,
as well as in response to the transformations I have sketched. A new
generation of journal editors will arise who have grown up with
electronic editing and publishing. In 10 years' time, although
procedures will be followed that some journals will still label
"refereeing" or "journal peer review," these procedures will be
startlingly different from those put into place in the years after the
second world war; which, despite their brief history, seem so
monolithic and unchangeable today. To predict the details of the new
procedures is no doubt foolhardy. Yet I venture to say that the
transformation will open up the processes by which scientists judge
each other's work, making them less anonymous, capricious, rigid, and
subject to abuse, and more thorough, responsible, and accountable. It
will oblige the readers of journals--even journals of clinical
research--to take a more active part in the intellectual assessment of
published work. Eventually--but sooner than you can easily imagine--we
will see an evolution toward a form of publication that will be a
continuing open dialogue and collaboration among contributing
scientists, editors, expert commentators, and readers.
What will the editors and readers of those journals, looking back, be
likely to think of the system of peer review and refereeing that they
have replaced?
From the Program in History and Philosophy of Science, Stanford
(Calif) University.
Presented in part at the Second International Congress on Peer Review
in Biomedical Publication, Chicago, Ill, September 9, 1993.
Preparation of this article was supported in part by grants from
the Andrew W. Mellon Foundation and the Lucille P. Markey Charitable
Trust.
I thank Marcia Angell for a crucial demurral to a portion of the
paper as originally presented and two anonymous JAMA referees
for useful criticism.
Address correspondence to Program in History and Philosophy of Science,
Bldg 370, Room 111, Stanford University, Stanford, CA 94305 (Mr Judson).
References
1. Zuckerman H, Merton RK. Patterns of evaluation in
science. Minerva. 1971;9:66-100.
2. Kronick DA. Peer review in 18th-century scientific
journalism. JAMA. 1990;263:1321-1322.
3. Burnham JC. The evolution of editorial peer review.
JAMA. 1990;263:1323-1329.
4. Chubin DE, Hackett EJ. Peerless Science: Peer
Review and U.S. Science Policy. Albany: State University of New
York Press; 1990.
5. McCutchen CW. Peer review: treacherous servant,
disastrous master. Technol Rev. October 1991:28-36, 40.
6. Rennie D, Flanagin A, Glass RM. Conflicts of interest in
the publication of science. JAMA. 1991;266:266-267.
7. Relman AS. Dealing with conflicts of interest.
N Engl J Med. 1984;311:405.
8. Swazey JP, Anderson MS, Lewis KS. Ethical problems in
academic research. Am Sci. 1993;81:542-553.
9. Garfunkel JM, Ulshen MH, Hamrick HJ, Lawson EE. Problems
identified by secondary review of accepted manuscripts. JAMA.
1990;263:1369-1371.
10. Evans AT, McNutt RA, Fletcher SW, Fletcher RH.
Characteristics of peer reviewers who produce good reviews. In: Program
and abstracts of the Second International Congress on Peer Review in
Biomedical Publication; September 9-11, 1993; Chicago, Ill; p 11.
11. Marshall E. Varmus: the view from Bethesda.
Science. 1993;262:1364-1366.
12. Price DJS. Little Science Big Science. New
York, NY: Columbia University Press; 1963:10, 12, 19.
13. Rip A. An exercise in foresight: the research system in
transition--to what? In: Cozzens SE, Healey P, Rip A, Ziman J, eds.
The Research System in Transition. Dordrecht, the
Netherlands: Kluwer Academic Publishers; 1990:387-401.
14. Brooks H. The future: steady state or new challenges?
In: Cozzens SE, Healey P, Rip A, Ziman J, eds. The Research System
in Transition. Dordrecht, the Netherlands: Kluwer Academic
Publishers; 1990:163-177.
15. Krige J. The international organization of scientific
work. In: Cozzens SE, Healey P, Rip A, Ziman J, eds. The Research
System in Transition. Dordrecht, the Netherlands: Kluwer Academic
Publishers; 1990:179-197.
16. US Congress, Office of Technology Assessment.
Biotechnology in a Global Economy. Washington, DC: Office of
Technology Assessment; 1991.
17. Krimsky S. Biotechnics and Society: The Rise of
Industrial Genetics. New York, NY: Praeger Publishers; 1991.
18. Nelsen LL. The lifeblood of biotechnology:
university-industry technology transfer. In: Ono RD, ed. The
Business of Biotechnology: From the Bench to the Street. Boston,
Mass: Butterworth-Heinemann; 1991:39-75.
19. Judson HF. A history of the science and technology
behind gene mapping and gene sequencing. In: Kevles DJ, Hood L, eds.
The Code of Codes: Scientific and Social Issues in the Human
Genome Project. Cambridge, Mass: Harvard University Press;
1992:37-80.
20. Ziman J. Research as a career. In: Cozzens SE, Healey
P, Rip A, Ziman J, eds. The Research System in Transition.
Dordrecht, the Netherlands: Kluwer Academic Publishers; 1990:345-359.
21. Stix G. Extra! extra! Newspaper publishers reinvade
cyberspace. Sci Am. February 1994;270:110-111.
22. Lederberg J. Communication as the root of scientific
progress. Scientist. 1993;7:10-11, 14.
23. Taubes G. Publication by electronic mail takes physics
by storm. Science. 1993;259:1246-1248.
24. Wulf WA. The collaboratory opportunity.
Science. 1993;261:854-855.
25. Krumenaker L. Virtual libraries, complete with
journals, get real. Science. 1993;260:1066-1067.
26. Taubes G. E-mail withdrawal prompts spasm.
Science. 1993;262:173-174.
27. Wulf WA. The national collaboratory--a white paper. In:
Lederberg J, Uncapfer K, eds. Towards a National Collaboratory.
Unpublished report of a workshop presented at Rockefeller
University, New York, NY, March 17-18, 1989.
28. National Research Council. National
Collaboratories: Applying Information Technology for Scientific
Research. Washington, DC: National Academy Press; 1993.
29. Hartman A, Diem JE, Quagliana M. The many faces of multimedia: how new technologies might change the nature of the
academic endeavor. In: Barrett E, ed. Sociomedia: Multimedia,
Hypermedia, and the Social Construction of Knowledge. Cambridge, Mass:
MIT Press; 1992:175-192. [No in-text reference.]
Table of Contents
