JAMA and the BMJ Publishing Group welcome you to Barcelona and the Fourth International Congress on Peer Review in Biomedical Publication. This Congress follows the 3 highly successful International Congresses on Peer Review in Biomedical Publication held in 1989 and 1993 in Chicago and in 1997 in Prague. The program appears below. As before, we will have 3 days for presentations from the podium, including 41 presentations of new research into peer review and all aspects of scientific publication. Each of these presentations will be followed by equal time for discussion and questions from the audience. In addition, there are 65 poster presentations scheduled for Saturday and Sunday. We hope you will take an active part in the program as we depend on you to make this Congress even more successful than the previous 3 Congresses. We wish to thank Ediciones Doyma for assistance with local arrangements as well as the Congress sponsors, exhibitors, and advertisers. Enjoy the Congress and enjoy Barcelona!

Drummond Rennie, Congress Director

Annette Flanagin, Congress Coordinator

Richard Smith, European Director

Jane Smith, European Coordinator

Evarist Feliu, Local Arrangements Coordinator

Special thanks to our

Sponsors

Exhibitors

Congress organizers and planners

Detailed Program: SEPTEMBER 14-16, 2001

Research on Peer Review (Continued)

Moderator: Lisa A Bero (UNITED STATES)

Effect of Statistical Review on Manuscript Quality in Medicina Clínica
Catalina Arnau, Erik Cobo, Francesc Cardellach, Joseph Maria Ribera, Albert Selva, Agustin Urrutia,
Vincente Fonollosa, Celestino Rey-Joly, Miquel Vilardell, J Lluís Segú (SPAIN)

SATURDAY, SEPTEMBER 15

Authorship and Contributorship

Communicating to Readers

Editorial Process

Peer Review of Grants

Quality Issues and Standards

Research on Abstracts

Study of Peer Review

SUNDAY, SEPTEMBER 16

Authorship and Contributorship

Communicating to Readers

Towards Improving the Study of the Usability of Electronic Forms of Scientific Information
Philippa Jane Benson (UNITED STATES)

Importance of Letters to the Editor in the Nederlands Tijdschrift Voor Geneeskunde
(Dutch Journal of Medicine)
Shalindra Mahesh, Martin Kabos, Henk C Walvoort, A John P M Overbeke (THE NETHERLANDS)

What Is Newsworthy? A Comparison of Reporting of Medical Research in
British Tabloid and Broadsheet Newspapers
Christopher Bartlett, Jonathan Sterne, Matthias Egger (UNITED KINGDOM)

Guidelines for Good Publication Practice: The COPE Experience
Richard Horton (UNITED KINGDOM)

Informed Consent in Clinical Trials: Survey of Five Chinese Medical Journals
Wang Mouyou (CHINA)

Effects of Reviewers’ Gender on Assessments of a Gender-Related Standardized Manuscript
Addeane S Caelleigh, Mohammedreza Hojat, Ann Steinecke, Joseph S Gonella, (UNITED STATES)

Influence of Impact Factors on Scientific Publication
Eileen Healy, Michael N Hart (UNITED STATES)

Evidence of Journal Bias Against Complementary and Alternative Medicine
David Moher, Terry Klassen, Margaret Lawson (CANADA)

Effect of Industry Funding on Manuscript Quality
Carin M Olson, Drummond Rennie, Deborah Cook, Mary Mickel, Kay Dickersin, Annette Flanagin,
Qi Zhu, Jennifer Reiling, Brian Pace (UNITED STATES, CANADA)

Relationship Between Manner of Presentation of Illustrations During the Peer Review Process and the
Number of Evaluative Responses
Judith A McKay (UNITED STATES)

What Is the Quality of the Economic Information Provided in Promotional Material for Family Practitioners?
The Case of Proton Pump Inhibitors in the United Kingdom
Vittorio Demicheli, Tom Jefferson (ITALY, UNITED KINGDOM)

Questionable Diagnostic Experiment Design in Laboratory Medicine Publications and
Recommendations for Improvement
Yan Zang, Qin Xiaoguang (CHINA)

Use of Dedicated Methods/Statistical Reviewers for Peer Review:
A Content Analysis of Comments to Authors Made by Methods and Regular Reviewers
Frank C Day, David L Schriger, Christopher Todd, Robert L Wears (UNITED STATES)

Reference Accuracy in Peer-Reviewed Journals: A Systematic Review
Elizabeth Wager, Philippa Middleton, the Peer Review and Technical Editing Systematic Review
(PIRATES) Group (UNITED KINGDOM, AUSTRALIA)

Effects of Published Errata on Subsequent Publications
MaryEllen Sievert, John M Budd, Gabriel Peterson, Kui Chun Su (UNITED STATES)

Medical Journal Publishing: One Culture or Several?
Tim Albert, Alex Williamson (UNITED KINGDOM)

Manuscript Submissions and US Representation at Medical Journals in an Era of Health System Reform:
An Analysis of Trends, 1994-1998
Michael Berkwits, Warren A Bilker, Frank Davidoff (UNITED STATES)

The Influence of War on Publishing in Peer-Reviewed Journals
Rajko Igic (UNITED STATES)

Where Are the High-Quality, Clinically Relevant Studies Published?
K Ann McKibbon, Nancy L Wilczynski, R Brian Haynes (CANADA)

The Coverage of Women’s Health in General Medical vs Women’s Specialty Journals:
Not Just "Navel-to-Knees"
Jocalyn P Clark, Julie M Dergal, Penelope de Nobrega, Anjali Misra,
Georgina D Feldberg, Paula A Rochon (CANADA)

Prodding Tardy Reviewers: Randomized Comparison of Phone, Fax, and e-Mail
Leon F Burmeister, Roy M Pitkin (UNITED STATES)

Effect of Formal Training Session on Subsequent Performance of Journal Peer Reviewers
Michael L Callaham, David L Schriger (UNITED STATES)

Structured Training Resources for Scientific Peer Reviewers
Richelle J Cooper, Michael L Callaham, David L Schriger (UNITED STATES)

The Quality of Reviewers of the Chinese Journal of Internal Medicine
Ding Yunqiu, Qian Shouchu (CHINA)

Frequency and Consistency of Reviewers’ Comments on a Methods Section
Erica Frank, Lucia McLendon, Donna Brogan, Dorothy Fitzmaurice (UNITED STATES)

Reviewers and Reviews Under Scrutiny: An Assessment of Strengths and Weaknesses
Sheila Mary McNab (THE NETHERLANDS)

Peer Reviewers’ Recommendations to Publish and Ratings of Manuscript Quality for
Accepted and Rejected Manuscripts
Robert McNutt, Richard M Glass (UNITED STATES)

Attitudes Toward Open Peer Review and Electronic Transmission of Papers for Their Review
Remedios Melero, F López-Santoveña (SPAIN)

The Care of Peer Review: Reviewers vs Editors and Future Citation of Manuscripts
Tobias Opthof, Ruben Coronel, Jaques M T de Bakker, Jan W T Fiolet, Marcel M Levi,
Martin Pfaffendorf, Marieke W Veldkamp, Arthur A M Wilde, Michiel J Janse (THE NETHERLANDS)

Peer Reviewers’ Age and Peer Review Quality
Qian Yue, Cui Xiaolan, Li Wenhhui, Bao Yalin, Huang Yubin, Cai Lifeng, Zhao Hongmei, Wang Guizhen (CHINA)

Referees’ Opinions About Editorial Policies and Practices of an Academic Medical Journal in Brazil
Julio Cesar Voltarelli,Valderez Aparecida Coelho Falaschi, Maria De Lourdes Veronese Rodrigues (BRAZIL)

Back to Top of Program

PLENARY SESSION ABSTRACTS

FRIDAY, SEPTEMBER 14

New Research on Authorship and Contributorship

Design: Annals of Internal Medicine and Radiology developed a taxonomy of author contributions and collected data for a sample of consecutive research reports in each journal (n=106 Annals of Internal Medicine, n=100 Radiology). Contributions by byline position were examined. At the time of data collection, the ICMJE stated that authors should contribute to study conception and design, analysis and interpretation, and manuscript writing.

Results: The 206 publications included 1389 authors (median n=6, range 1-19). Compared to second or third authors, more first authors reported contributing to study conception/design (97% first vs 77% second and 55% third authors), data analysis/interpretation (90% vs 68% and 48%), and drafting (95% vs 51% and 27%), revising (92% vs 83% and 62%), and approving the manuscript (93% vs 84% and 70%). Authors in other positions least often reported contributions in all categories, but frequently (53%) provided study materials or patients. Last authors made fewer contributions than first authors, but generally reported contributions in each area more often than authors in positions other than first through third or last on the byline. Most first authors satisfied ICMJE criteria (93%) compared to only 72% of second authors, 51% of third authors, 54% of last authors, and 33% of authors in other positions. Somewhat higher percentages of authors satisfied a more relaxed ICMJE definition of authorship criteria (98% of first authors, 86% of second authors, 62% of third authors, and 66% of last authors).

Conclusions: It is feasible for journals to combine authorship data. Contributions vary by byline position. Most first authors satisfy ICMJE criteria, but substantial proportions of authors who are not first on the byline do not.

¹Annals of Internal Medicine, 190 N Independence Mall West, Philadelphia, PA 19106, USA, e-mail: claine@mall.acponline.org; ²Radiology, Richmond, VA, USA

Lydia E Vos and A John P M Overbeke

Objective. To study the definition of the contributions of authors to articles in the Nederlands Tijdschrift van Geneeskunde (NTvG; Dutch Journal of Medicine) within different headings. The contributions are compared with the authorship criteria of the International Committee of Medical Journal Editors (ICMJE).

Design: From January 1 to May 1, 1999, all authors of accepted articles were asked to define their contribution to the study design, the performance, and the reporting. Before publication of the article, the written contributions attended by the signature of the author had to be present at the editorial office. The first study investigator scored the contributions of the authors on 23 keylist words and obscure contributions were scored by both study investigators. The contributions were compared with the ICMJE criteria. Conception, design of the study, collecting data, and (statistical) analysis belonged to criterion 1. Writing the first version, rewriting, and critical revision of the paper belonged to criterion 2; and approving the last version to criterion 3.

Results: Of 91 accepted articles, 83 (91.2%) could be analyzed. The 310 authors wrote 690 contributions down, mostly by the first author with an average of 2.6. Thirty-one authors (10.0%) fulfilled criterion 1, 151 authors (48.7%) fulfilled criterion 2, and 111 authors (35.8%) fulfilled both 1 and 2. Only 6 authors (1.9%) reported approving the last version (criterion 3). Sixteen (5.2%) of authors fell outside the criteria. The contribution patient care does not belong to the ICMJE criteria, but was nevertheless written down by 42 authors (13.5%).

Conclusion: In 35.8% of the self-defined contributions of authors the contributions are congruent with the ICMJE criteria 1 and 2. Although most of the authors read the definitive version, it is reported rarely. Patient care is often mentioned. The order of the authors show that the first authors write the first version of the paper and collect data and that the other authors critically revise.

Nederlands Tijdschrift voor Geneeskunde (Dutch Journal of Medicine), PO Box 75971 1070 AZ Amsterdam, the Netherlands,
e-mail: overbeke@ntvg.nl

Susan van Rooyen,¹ Sandra Goldbeck-Wood,¹ and Fiona Godlee²

Context: Authorship of research articles is a central currency in biomedical science. But the rules that are supposed to govern who can claim authorship (the so-called Vancouver Group guidelines from the International Committee of Medical Journal Editors [ICMJE]) are considered by many to be unrealistic and unnecessarily restrictive. Studies have found that these guidelines are unknown to many researchers, are often disregarded, and do not reflect the actual contributions as declared by authors.

Objective: To produce a taxonomy of authors’ declared contributions to their research, to assess whether these contributions

conform to the ICMJE guidelines, and to recommend changes to the current requirements for authorship.

Design: Observational study of consecutive research papers published in the BMJ from July to December 1998 and the extent to which declared contributions conform to the ICMJE guidelines and to our view of an ideal paper.

Results: One hundred twenty-nine research articles were included in the analysis, with 588 authors. The number of contributions per author ranged from 1 to 9 (mean 3.8). First authors declared most contributions (mean 5.0). The most commonly declared contribution was writing/editing the paper. Since we had no data on whether authors had approved the final version of the paper, we used 4 proxy measures for this criterion. The proportion of authors conforming to the ICMJE guidelines ranged from 24% to 71%, depending on which of these 4 definitions was used. We defined an ideal paper as one in which key contributions were declared by at least 1 author. Only 17% of papers conformed to this definition. The contributions and the proportion of papers for which the contribution was not declared were as follows: initiation (26%), design (21%), data collection or data entry (26%), analysis (6%), interpretation of results (60%), writing/editing (6%), and being a guarantor (18%).

Conclusions: This study confirms the view that the ICMJE guidelines do not reflect current practice among authors. It suggests that we should move away from a prescriptive approach to authorship to a more descriptive approach, using the declared contributions of authors as a basis.

¹BMJ, BMA House, Tavistock Sq, London WC1H 9JR, UK, e-mail: svanrooyen@kscable.com; ²BioMed Central, London, UK

Graham Mowatt and Jeremy M Grimshaw, for the Survey of Contributorship in Cochrane Reviews Project Team

The Survey of Contributorship in Cochrane Reviews Project Team includes Graham Mowatt, Liz Shirran, Jeremy Grimshaw, Graeme MacLennan, Phil Alderson, Lisa Bero, Iain Chalmers, Annette Flanagin, Peter Gøtzsche, Adrian Grant, Melissa Ober, Drummond Rennie, and Veronica Yank

Objective: A previous survey by Flanagin et al found that 26% of review articles in peer-reviewed medical journals had evidence of honorary authorship and 10% had evidence of ghost authorship (Flanagin et al. JAMA. 1998;280:222-224). Jadad and colleagues observed that Cochrane reviews had fewer authors than reviews published in paper-based journals; this may suggest that honorary authorship is less of a problem in Cochrane reviews (Jadad et al. JAMA. 1998;280:278-280). The objective of this survey was to determine the prevalence of honorary and ghost authorship in Cochrane reviews.

Design: In March 2000, primary contacts for 577 Cochrane reviews published in issues 1-2 1999 of the Cochrane Library were invited to complete, on behalf of their co-reviewers, a 29-question, Web-based questionnaire. Data were analyzed using SPSS.

Results: Primary contacts for 362 reviews (63% response rate) completed the questionnaire. Thirty-nine percent of reviews (141) had honorary authors (based on the International Committee of Medical Journal Editors’ authorship criteria, March 2000); in 31 reviews (9%) primary contacts reported that not all authors would feel comfortable explaining the major conclusions of the review in an oral presentation. Nine percent (32) had ghost authors (defined as individuals not listed as reviewers who made contributions that merited authorship or assisted in drafting the report of the review). The group of authors (31%) or lead author (25%) decided authorship in the majority of reviews. There was no formal mechanism for deciding authorship in 28% of reviews. Authorship order was assigned according to contribution in the majority (75%) of reviews.

Conclusions: The prevalence of ghost authorship in Cochrane reviews was broadly similar to that reported for review articles in peer-reviewed medical journals; the prevalence of honorary authorship was greater in Cochrane reviews (this may be partly explained by the way in which honorary authorship was defined). Honorary authorship was more prevalent than ghost authorship. The Cochrane Collaboration and Cochrane Review Groups need to develop mechanisms to address these issues.

Health Services Research Unit, University of Aberdeen, Polwarth Building, Foresterhill, Aberdeen AB25 2ZD Scotland, UK,
e-mail: g.mowatt@abdn.ac.uk

Kay Dickersin,¹ Roberta Scherer,² Michelle Gil-Montero,³ and Eunike Suci³

Objective: To examine the ways in which reports of controlled trials with corporate authorship (ie, authors listed by research group name) are indexed and citations counted in bibliographic databases. We were interested in whether reports with corporate authors are difficult to identify and whether this may lead to problems with citation. Corporate authorship has been increasingly used, especially for multicenter clinical trials and other types of studies (eg, genomics) with many contributors.

Design: Cross-sectional, descriptive study. All 47 controlled trials funded by the National Eye Institute and their associated reports were identified. MEDLINE and Science Citation Index (SCI) were searched and citing practices were recorded.

Results: A total of 285 published reports were identified; 44% had corporate authorship, 36% had modified corporate authorship (individual names plus name of research group), and 20% had named authors only. MEDLINE listed no corporate authors in the author field; in SCI, corporately authored reports generally were incorrectly listed with named authors (first name on investigator list [35%], first name on writing committee [25%], contact name [17%], other [23%]). Using the SCI "general search," citations to 17% of corporately authored reports were identified, compared to 97% with modified corporate authorship, and 94% with named authors. Other search methods revealed this was a large undercount of actual citations to corporately authored reports (ie, >_ 73% of corporately authored reports actually had been cited). Furthermore, corporately authored reports were cited fewer times than reports with other types of authors. SCI listed most citations to corporately authored reports using an abbreviation of the corporate name.

Conclusions: Although corporate authorship allows investigators to share credit, indexing systems are not currently adapted to this approach. This may result in user confusion and fewer citations. Modification of indexing systems to list corporate authorship in the author field, as is currently under way for MEDLINE, may help to correct the problem.

¹Department of Community Health, Brown University School of Medicine, 169 Angell St, Box G-S2, Providence, RI 02912, USA,
e-mail: kay_dickersin@brown.edu; ²University of Maryland Medical School, College Park, MD, USA; ³Brown University, Providence, RI, USA

Richard Horton

Objective: Contributorship has revealed the range of intellectual and practical input into published medical research. Are the views expressed in a research paper accurate representations of contributors’ opinions?

Design: For this analysis, 10 papers published in the Lancet during 2000 were selected. Articles selected included varying numbers of contributors (range 2-11), subject areas, and methods. Contributors were asked 6 questions about their study: the key findings, strengths, weaknesses, interpretation within the totality of evidence, implications, and future research. The contributors’ answers were compared, individual views were contrasted with those in the discussion sections of the published paper, and private and public opinions were matched with stated contributions.

Results: Complete data for 5 groups of contributors were available. Discussion sections were haphazardly organized and did not deal systematically with important questions about the study. Weaknesses were often admitted on direct questioning but were not included in the published paper. Contributors frequently disagreed about key findings, weaknesses, implications, and future research. This diversity of opinion was commonly excluded from the published report. Contributors who were less involved in writing the paper were not only more critical of the study’s design but also did not have their criticism included in the published article.

Conclusions: A research paper rarely represents the opinions of scientists whose work it reports. Weaknesses in study design, acknowledged by co-contributors, may be hidden by the contributor who will receive most credit from publication. These data provide empirical support for both structured discussions and for means to recover the plurality of contributors’ opinions.

Lancet, 42 Bedford Sq, London WC1B 3SL, UK, e-mail: r.horton@elsevier.co.uk

Quality Issues and Standards for Published Material

Karin Huwiler,¹ Peter Jüni,^2,3 Matthias Egger,³ and Christoph A Junker¹

Objective: The quality of reporting is often used as a proxy measure for the methodological quality of randomized controlled trials (RCTs). In this study, the relationship between reporting and methodological quality was examined using the example of analysis by intention to treat (ITT).

Design: Sixty RCT reports were randomly selected from a database of 254 reports. Reporting quality was assessed by 2 reviewers using a 30-item scale based on the 1996 CONSORT statement. Each item earned 1 point, for a maximum score of 30 points. Trials with adequate concealment of treatment allocation were considered to be of high methodological quality. Reporting and methodological quality was compared between trials explicitly reporting an ITT analysis, trials reporting exclusions after randomization, and trials with unclear reporting.

Results: The overall quality of reporting was identical for trials with explicit reporting, independently of whether the analysis was by ITT (n=33, median score 18, range 10-22) or not (n=13, median score 18, range 9-27), but lower for trials with unclear reporting (n=14, median score 11.5, range 4-16) (P=.001 by Kruskal-Wallis test). Conversely, there was a decreasing trend in methodological quality from trials with ITT analyses (methodological quality high in 33.3% of trials) to trials without ITT analyses (methodological quality high in 23.1% of trials) to trials with unclear reporting (methodological quality high in 14.3% of trials) (P=.17 by chi-square test for trend).

Conclusions: Similar quality of reporting may hide differences in methodological quality and a distinction between the 2 is therefore required. When assessing methodological quality a distinction should be made between the presence or absence of a quality criterion and unclear reporting. Indeed, in the landmark study by Schulz et al (JAMA 1995) the larger effect estimates in trials without explicit exclusions may be due to confounding by inadequate reporting.

¹Departments of Social and Preventive Medicine and ²Rheumatology and Clinical Immunology/Allergology, University of Berne, Switzerland; ³MRC Health Services Research Collaboration, University of Bristol, Whiteladies Rd, Canyge Hall, Bristol BS8 2PR, UK,
e-mail: juni@bristol.ac.uk

Phil Alderson, Mike Clarke, and Iain Chalmers

Objective: Anyone wishing to interpret a trial needs to know how its results compare to those of similar studies. This was recognized by the original CONSORT statement, which recommended that the report of a randomized trial should discuss its findings in light of the "totality of relevant evidence." In a previous study, Annals of Internal Medicine, BMJ, JAMA, Lancet, and the New England Journal of Medicine published 26 reports of randomized trials in May 1997. Reports of apparently similar trials were found for 25 of these. In only 2 were the trial’s results placed in the context of a systematic review of other relevant studies. Thus, almost all trials did not provide sufficient information to interpret their results reliably.

Design: The study was repeated in May 2001.

Results: Thirty-three reports of randomized trials were identified in the 19 issues of these 5 journals published in May 2001. None of these reports contained a discussion of the trial’s results in the context of an updated systematic review of earlier trials. Reference was made to relevant systematic reviews in 3, but no attempt was made to integrate the results of the new trial in updated versions of these reviews. Four reports claim to have been the first published trial to address the question studied. Six reports did not claim to be the first trial but did not cite any related randomized trials. The remaining 20 reports cited other trials, but there was no evidence that any systematic attempt had been made to set the new trial’s results in context.

Conclusions: If anything, the situation has worsened during the last 4 years. There is evidence of a Balkanization in the reporting of new trials, with relevant, related research appearing to be ignored. Trials reported in these major medical journals still do not provide sufficient information to interpret their results reliably.

UK Cochrane Centre, NHS R&D Programme, Summertown Pavilion, Middle Way, Oxford, OX2 7LG, UK, e-mail: mclarke@cochrane.co.uk

Denise Cheng, Joy Melnikow, and Jim Nuovo

Objective: To examine the frequency of reporting the number needed to treat (NNT) and the number needed to harm (NNH) in randomized controlled trials (RCTs).

Design: Five journals were selected for investigation: Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine. For each journal, 4 specific years were evaluated: 1989, 1992, 1995, and 1998. A MEDLINE/HEALTHSTAR search was conducted of each journal for the years identified. The search terms used were "randomized controlled trials" and "controlled trials." The search was supplemented by a manual review of each journal for the specific year of interest. Eligible articles included those in which an RCT was conducted on the use of a medication for any treatment effect. The study needed to demonstrate a significant positive effect of the treatment drug. The following was abstracted from each eligible article: condition investigated, event being treated/prevented, intervention, study results, and reporting methods (relative risk reduction, absolute risk reduction, odds ratio, NNT, NNH, or other).

Results: Of 358 eligible articles, NNT was reported in 7 articles; 6 of these articles were published in 1998. Number needed to harm was not reported.

Conclusions: Despite the usefulness of this reporting method, few authors express their findings in terms of NNT and none reported NNH. Consideration should be given to including these values in reports of RCTs.

University of California, Department of Family & Community Medicine, 4860 Y St, Sacramento, CA 95817, USA, e-mail: jim.nuovo@ucdmc.ucdavis.edu

Yen-Hong Kuo

Objective: An estimated association between 2 variables is valid only within the range of the data. Extrapolation is risky and should be handled with caution. This study was undertaken to assess how the extrapolation issue was managed in 4 weekly general medical journals.

Design: All of the articles published from January through June 2000 in the BMJ, JAMA, the Lancet, and the New England Journal of Medicine were reviewed manually. Articles containing a scatterplot with raw data and a corresponding fitted regression line were included for analyses.

Results: A total of 178 articles presenting at least 1 scatterplot were identified. Among them, 37 articles (21%) containing a fitted line were included: 5 from BMJ, 7 from JAMA, 23 from the Lancet, and 2 from the New England Journal of Medicine. Of them, 31 articles (84%) used a simple linear regression line to illustrate the association between 2 variables. Nine articles (24%) provided the confidence interval (CI) for the fitted line. Twenty-two articles (59%; 95% CI, 42%-75%) from all of the 4 journals involved extrapolation, that is, the fitted line beyond the observed data. None of them changed the line type to indicate extrapolation. Four articles (11%) contained a plot with the fitted line reaching the unreasonable or meaningless values. Three articles (8%) stated explicit conclusions about values outside the range of the observed data.

Conclusions: A high proportion of the reviewed articles contained a fitted line beyond the data without indication. This extrapolation problem should be eliminated by presenting the fitted line within the range of the observed data or by changing the line type outside the data range.

Department of Research, Jersey Shore Medical Center, Meridian Health System,1945 State Route 33, Neptune, NJ 07753, USA, e-mail: yhkuo@monmouth.com

Objective: To investigate the nature and frequency of statistician involvement in medical research and its relation to the final editorial decision.

Design: In a prospective cohort study, authors of original research articles consecutively submitted to the BMJ and Annals of Internal Medicine have been sent a short questionnaire at the time of manuscript submission. This questionnaire explores whether they received any assistance from a person with statistical expertise, the nature of the statistical contribution, the availability of statistical expertise at their institution, and the qualifications of the person providing that contribution. In cases where there was no such input, they are asked for reasons why. The study is expected to include more than 500 papers submitted during the first half of 2001.

Results: Answers to the questionnaire will be summarized, both overall and in relation to the outcomes of the editorial decision. Analysis will be stratified by design of studies.

Conclusions: Statistical input to medical research is widely recommended but inconsistently obtained. This study will provide information about the timing and nature of statistical input into medical research submitted for publication and will indicate how often those providing such expertise go unrecognized by either authorship or acknowledgment. The study will enhance understanding of the impact of statistical input as well as help to clarify why so much medical research is conducted without such assistance.

Vittorio Demicheli and Tom Jefferson

Objective: To identify and assess systematic reviews of economic evaluations in health care and to evaluate quality of review methods and evidence of changes in conduct and reporting of economic evaluations.

Design: Descriptive Cochrane review of published and unpublished original reports of exploratory or analytical reviews of published and unpublished economic evaluations with clear aims, methods, inclusion criteria, list of excluded and included studies, results and discussion sections commenting on the quality of methods, and/or reporting of economic evaluations. Once inclusion criteria had been applied, quality assessment of included reviews was carried out based on 8 criteria, each with a 1 to 4 score.

Results: Thirty-seven reports of reviews were identified. Eighteen failed to meet inclusion criteria. The remaining 19 reports were included and grouped by broad topic. The quality of included reviews was high with 18 reports having a mean score of at least 3.25. Correlation of score between the 2 reviewers for quality criteria was good (Spearman’s test range 0.989-1.000). Consistent evidence of serious methodologic flaws was found in a significant number of economic evaluations of health care interventions, regardless of publication status, period of preparation or publication, topic, and study design. There is evidence of modest and slow improvements in quality in the last decade.

Conclusions: Quality of systematic reviews of economic evaluations is good, but given the importance of economic evaluations in resource allocation decisions, the quality of published and unpublished economic evaluations must be improved. Editorial teams, peer reviewers, and regulatory bodies should implement quality assurance based on a single widely accepted and validated standard instrument, such as the BMJ checklist.

Cochrane Health Economics Group and Health Reviews Ltd, 35 Minehurst Rd, Mytchett, Surrey GU16 6JP, UK, e-mail: toj1@aol.com

Richelle J Cooper,¹ David L Schriger,¹ Roger C Wallace,¹ Vladislav J Mikulich,¹ and Michael S Wilkes²

Objective: Are pharmaceutical advertisements sophisticated medical communications akin to scientific publications, or hollow slogans akin to popular advertising? If the former, graphs within advertisements should be similar to graphs in scientific manuscripts. This study characterized the quantity and quality of graphs in pharmaceutical advertisements.

Design: All pharmaceutical advertisements in 10 leading US medical journals published in 1999 were reviewed and each data graph was evaluated to characterize its features. Pharmaceutical advertisement graphs were contrasted with graphs in articles of JAMA and Annals of Emergency Medicine.

Results: There were 836 glossy and 455 small-print pages in 484 unique advertisements (of 3190 total advertisements). Forty-nine percent of glossy page area was nonscientific figures/images, 0.4% tables, and 1.6% scientific graphs (74 graphs in 64 advertisements). The remaining 49% was text or blank page. Eight percent of graphs had errors, 5% had visual obfuscation, and 12% used nonstandard graphing conventions. Only 36% of graphs were self-explanatory. No graphs contained advanced features (pairing, symbolic dimensionality, or small multiples). Fifty-eight percent presented data on an outcome relevant to the drug’s indication. When comparing the pharmaceutical advertisement graphs (n=74) with scientific graphs from JAMA (n=64) and Annals of Emergency Medicine (n=128), more simple univariate graphs (96%) were noted in advertisements than in articles published in JAMA (63%) or Annals of Emergency Medicine (53%). Pharmaceutical advertisement graphs had more visual noise (66% vs JAMA 0% or Annals of Emergency Medicine 10%), more numeric distortion (36% vs JAMA 6% or Annals of Emergency Medicine 5%), and more redundancies within the graphs (46% vs JAMA 14% or Annals of Emergency Medicine 16%). The efficiency of data presentation quantified by the data depiction index (area of graph that contains information) was less in the pharmaceutical advertisement graphs; median and interquartile ranges were for pharmaceutical advertisements 0.22 (0.11, 0.43), JAMA 1.1 (0.52, 3.36), and Annals of Emergency Medicine 0.94 (0.54, 1.7).

Conclusions: Graphs in pharmaceutical advertisements were rare and when present were of lower quality than those in journal articles. The pharmaceutical advertisement graphs’ designs frequently resulted in numeric distortion, which is specifically prohibited by US Food and Drug Administration regulations.

¹UCLA Emergency Medicine Center, UCLA School of Medicine, 924 Westwood Blvd, #300, Los Angeles, CA 90024, USA, e-mail: richelle@ucla.edu; and ²Department of Medicine, UCLA School of Medicine

Michael L Callaham,¹ Robert K Knopp,² and E John Gallagher³

Objective: Whether simple written feedback by editors to peer reviewers improves the quality of subsequent reviews

Design: All reviews were routinely rated for quality on a 1 to 5 scale by editors blinded to the study. Study 1: Peer reviewers with review volume of 3 or less per year, and median quality scores of 3 or less, were randomized to an intervention group (receiving the editor’s score of their review, plus unscored copies of other reviews of the same manuscript, plus a very brief summary of journal objectives for review content) or to a control group (receiving other unscored reviews). The study was designed to have a power of 0.95 to detect a difference in score of 1, using 52 subjects. Study 2: Same design as study 1 except different reviewers with median score of 4 or less and 3 or more reviews, and the intervention included: receiving the editor’s score of each manuscript review, scored copies of other reviewers’ reviews of the same manuscript, a brief summary of journal objectives for review quality, and a sample outstanding review of another manuscript (power of 0.95 to detect difference of 0.5).

Results: Study 1: 52 active reviewers were eligible and were randomized; 43 had sufficient data (a total of 153 rated reviews) for analysis. The mean rating change during the study was 0.36 (95% confidence in interval [CI] -0.18 to 0.9) for controls and
-0.31 (-0.78 to 0.15) for intervention. Study 2: Based on 245 rated reviews to date, 54 control group reviewers had a mean rating change of -0.01 (95% CI -0.29 to 0.28) and 51 intervention group reviewers had a mean rating change of 0.03 (-0.21 to 0.27).

Conclusions: In study 1, minimal feedback from editors on review quality had no effect on subsequent performance of low-volume, poor-quality reviewers. This feedback may actually have had a significant negative effect, perhaps due to the discouragement of poor review scores with minimal guidance as to remediation. In study 2, feedback to average reviewers was more extensive and supportive but to date produced no improvement in reviewer performance.

¹University of California, San Francisco, Box 0208, San Francisco, CA 94143-0208, USA, e-mail: mlc@itsa.ucsf.edu; ²University of Minnesota and St Paul-Ramsey Medical Center, Rochester, MN, USA; and ³Albert Einstein College of Medicine, New York, NY, USA

Michael Berkwits¹ and Frank Davidoff²

Objective: Peer reviewers are currently the only journal contributors whose identities and contributions are unknown to readers. We assessed reviewers’ opinions of a proposal to acknowledge their contributions to published manuscripts by publishing reviewer names with those manuscripts.

Design: In a pilot survey, reviewers for Annals of Internal Medicine were asked what they thought generally about published manuscript-specific acknowledgment with consent, if they would want it personally, if it would be recognized at their institutions, and if they would want it if it were institutionally recognized. Respondents answered on a 5-point scale with 1 representing unfavorable and 5 favorable responses. We hypothesized that junior and clinical more than senior and research reviewers would desire published manuscript-specific acknowledgment and compared these groups using the t test and rank sum test.

Results: Seventy-eight reviewers responded (80%). Reviewers were an average of 22.5 years out of training and spent equivalent mean times of 30% in clinical and research activities. Sixty-six percent (55,76) were indifferent to or against manuscript-specific acknowledgment (mean score and standard deviation was 2.7±1.3); 68% (57,79) said they would not want it personally (2.6±1.4), 84% (73,91) thought it would not be recognized by their institutions (2.2±1.3), but 51% (39,63) said they would want it if it were recognized (3.3±1.3). There were no differences in responses based on years since training or clinical or research focus, but the difference in wishes for acknowledgment without and with institutional recognition was significant (P<.001).

Conclusions: A majority of peer reviewers did not favor publishing manuscript-specific acknowledgment, in part because of perceptions of a lack of institutional recognition for the effort. Half favored the idea if it were linked to recognition, however. Though surveys assessing attitudes are often poorly predictive of actual behaviors, these data suggest that at least some support exists among reviewers for broadening acknowledgment and recognition of peer review activities.

¹University of Pennsylvania, Division of General Internal Medicine, 423 Guardian Drive, 1229 Blockley Hall, Philadelphia, PA 19104, USA, e-mail: berkwits@mail.med.upenn.edu; and ²Annals of Internal Medicine, Philadelphia, PA, USA

Ana Marusic,¹ Ivan Kresimir Lukic,¹ Matko Marusic,¹ David McNamee,² David Sharp,² and Richard Horton²

Objective: To compare peer reviewers’ recommendations in the Lancet, a high-impact journal, and the Croatian Medical Journal (CMJ), a small general medical journal.

Design: Comparison of peer review forms for all research manuscripts submitted to CMJ in 1999 and part of 2000 (n=140 manuscripts; 292 review forms) and a sample from the Lancet (n=141; 321 review forms), chosen systematically to cover the same period. Statistical reviews were excluded from the analysis.

Conclusion: Reviewers’ recommendations have similar influence on editorial decision in big and small journals, although reviewers’ assessment of the quality of manuscripts may differ. Agreement between reviewers in general medical journals is poor, regardless of their size and importance.

¹Croatian Medical Journal, Salata 3, 1000 Zagreb, Croatia, e-mail: marusica@mef.hr; ²Lancet, London, UK

Catalina Arnau, ¹ Erik Cobo,² Francesc Cardellach,² Joseph María Ribera,² Albert Selva,² Agustín Urrutia,² Vicente Fonollosa,² Celestino Rey-Joly,² Miquel Vilardell,² and Josep Lluís Segú²

Objective: To estimate the influence on manuscript quality of adding a statistical and methodological reviewer to the clinical peer review team.

Design: After random allocation, manuscripts were reviewed either by 1 statistical and 2 clinical experts or only by 2 clinical reviewers. Two blinded statistical evaluators measured each paper 2 times, before and after the journal peer review process, using a slightly modified version of the Manuscript Quality Assessment Instrument. The primary endpoint was the mean, of the 2 evaluators, of the sum of improvements on all study report modifications. Our postulated effect was 80% of the within group standard deviation.

Results: Twenty-six manuscripts received clinical review without statistical review and 17 manuscripts received clinical and statistical review. The improvements favored the inclusion of a statistical reviewer although the difference did not reach statistical significance. Mean quality scores and 95% confidence intervals (CIs) were 3.06 (2.03-4.09) for clinical review only and 4.12 (2.34-5.90) for clinical and statistical reviews. The 95% CIs mean difference of improvements were -0.8, 2.9. Point estimate of the statistical effect was 1.1, being 36% of the pooled standard deviation.

Conclusions: Statistical review improved the quality of manuscripts, although it did not reach statistical significance. A qualitative study is needed to investigate if the statistical reviewers fail to give accurate advice, if the authors fail to follow their recommendations, or if the design of this study was not precise enough to show the postulated effect.

¹Statistics and Operational Research, Universitat Politecnica de Catalunya, C/ Pau Gargallo 5, 08028 Barcelona, Spain, e-mail: erik.cobo@upc.es; and ²Medicina Clinica, Barcelona, Spain

Phil A Alderson,¹ Frank Davidoff,² Tom Jefferson,¹ and Elizabeth Wager³

Objective: To estimate the effect of editorial peer review on importance, relevance, usefulness, methodological soundness, ethical soundness, completeness, and accuracy of submissions to journals.

Design: Descriptive systematic review of prospective or retrospective studies with 2 or more comparison groups generated by random or other appropriate methods. Methods consisted of extensive searches, independent application of inclusion criteria, and descriptive quality assessment of included studies.

Results: The well-researched practice of reviewer and/or author concealment, while laborious and expensive, appears to have little effect on the outcome of the quality assessment process (9 studies). Checklists and other standardization media have little reliable evidence to support their use (2 studies). There is no evidence that referees’ training has any effect on the quality of the outcome (2 studies). Communication media do not appear to have an effect on quality (2 studies). On the basis of 1 study little can be said about the ability of the peer review process to detect bias against unconventional drugs. Validity of the process was tested by only 1 small study in a specialist area. Editorial peer review appears to make papers more readable and improve the general quality of reporting (2 studies), but the evidence for this has limited generalizability.

Conclusions: There is only a small and scattered amount of empirical evidence supporting the use of editorial peer review as a mechanism to ensure quality of biomedical research. Higher-sensitivity and lower-sensitivity inclusion criteria will be applied and the validity of these conclusions will be tested by carrying out a sensitivity analysis.

¹UK Cochrane Centre, Summertown Pavilion, Middle Way, Oxford OX2 7LG, UK, toj1@aol.com; ²Annals of Internal Medicine, Philadelphia, PA, USA; ³GlaxoSmithKline, Middlesex, UK

Pål Gulbrandsen,¹ Torben Veith Schroeder,² Josef Milerad,³ and Magne Nylenna¹

Objective: To evaluate whether the presentation (paper or computer screen, mother tongue or English) influences Scandinavian general practitioners’ ability to retain information from a scientific paper.

Design: Controlled trial of 111 general practitioners, ranging in age from 28 to 76 years (mean 47.6 years, SD 8.5), including 45 females (41%), in Denmark, Norway, and Sweden. The participants were randomized to read a review article for 10 minutes in either English or Danish/Norwegian/Swedish, and either on paper or on a computer screen. Immediately after reading, they completed a questionnaire related to the article, with a possible score range from 0 (no correct answers) to 13 (all questions answered correctly). A significance level of 5% and a power of 80% were selected and the necessary number of participants was calculated to be 96 before conducting the study. Mean scores of the different groups were compared using independent sample t tests.

Results: The mean score was 3.9 (95% confidence interval [CI] 3.5-4.4, range 0-11). For those reading English the score was 3.4 (95% CI 2.8-4.0) compared to 4.4 (95% CI 3.8-5.0) for those reading the article in their mother tongue (P=.02). There was no difference between those reading the article on paper (score 4.0, 95% CI 3.3-4.7) and those reading on screen (score 3.9, 95% CI 3.3-4.4)). Physicians younger than 40 years scored significantly higher (5.7, 95% CI 4.4-7.0) than those 40 years or older (3.6, 95% CI 3.1-4.0) (P<.01), and male physicians scored higher (4.3, 95% CI 3.7-4.8) than female physicians (3.4, 95% CI 2.7-4.1) (P=.05).

Conclusion: Medical scientific information is acquired more easily by Scandinavian general practitioners if presented in their mother tongue compared to English. Whether the information is presented on paper or on computer screen does not seem to have any impact.

¹Tidsskrift for Den norske lægeforening, PO Box 1152 Sentrum, NO-0107 Oslo, Norway, e-mail: pal.gulbrandsen@legeforeningen.no; ²Ugeskrift for læger, København, Denmark; ³Läkartidningen, Stockholm, Sweden

Paula A Rochon,^1,2,3 Lisa A Bero,⁴ Ari M Bay,¹ Jennifer L Gold,¹ Julie M Dergal,¹ Malcolm A Binns,¹ David L Streiner,^1,2 and Jerry H Gurwitz⁵

Objective: Throwaway journals are not peer reviewed and not cited in the literature. The quality, presentation, readability, and clinical relevance of review articles published in peer-reviewed journals and throwaway journals were compared to determine why throwaway journals are so popular.

Design: All review articles focusing on the diagnosis or treatment of a medical condition published in the Annals of Internal Medicine, BMJ, JAMA, Lancet, and the New England Journal of Medicine or a high-circulation throwaway journal (ie, Consultant, Hospital Practice, Patient Care, and Postgraduate Medicine) in 1998 were eligible. Our sample included 394 articles. Two reviewers independently assessed the methodologic and reporting quality of each article. A reviewer evaluated the article’s presentation and assigned readability scores using standard instruments. Clinical relevance was evaluated by 6 independent clinically oriented physicians who read a randomly generated list of 394 article titles.

Results: Of the 394 articles, 16 (4.1%) were peer-reviewed systematic reviews, 135 (34.3%) were peer-reviewed nonsystematic reviews, and 243 (61.7%) were nonsystematic reviews published in throwaway journals. The mean (SD) quality scores were highest in peer-reviewed articles. Quality scores were .94 (.09) for systematic reviews, .30 (.19) for nonsystematic reviews), and .23 (.03) for throwaway articles. Throwaway journal articles were significantly more likely to use tables (P=.023), figures (P=.011), photographs (P<.001), color (P<.001), and larger font size (P<.001) than peer-reviewed articles. Readability scores were more often in the college or higher range for peer-reviewed articles compare to the throwaway articles (104 [77.0%] vs 156 [64.2%], P=.01). Peer-reviewed article titles were judged to be significantly less relevant to clinical practice than throwaway article titles.

Conclusions: Although lower in methodologic quality, review articles published in throwaway journals appear to be better at communicating their messages than review articles in peer-reviewed journals.

¹Baycrest Centre for Geriatric Care, 3560 Bathurst St, Toronto, Ontario M6A 2E1, Canada, e-mail: paula.rochon@utoronto.ca; ²University of Toronto, Ontario, Canada; ³Institute for Clinical Evaluative Sciences, Ontario, Canada; ⁴University of California, San Francisco, San Francisco, CA, USA; and ⁵Meyers Primary Care Institute, University of Massachusetts, Worcester, MA, USA

Lisa M Schwartz,^1,2 Steven Woloshin,^1,2 and Linda Baczek¹

Objective: Although they are preliminary and have undergone limited peer review, research presentations at scientific meetings may receive prominent attention in the news media. This coverage is described, the quality of the research is assessed, and whether the research was subsequently published in the medical literature is reported.

Design: Lexis-Nexis was searched to identify major US news media coverage in the 2 months following 5 scientific meetings held in 1998 (American Heart Association, 12th World AIDS Conference, American Society of Clinical Oncology, Society for Neuroscience, and the Radiological Society of North America). Abstracts that generated the new stories were located and MEDLINE was searched to identify subsequent publication in the next 2 years.

Results: A total of 255 news stories reporting on 149 research presentations were found, an average of 51 news stories per meeting. With the exception of the Wall Street Journal, 10 or more stories appeared in each of the top 5 US newspapers (USA Today, New York Times, Los Angeles Times, Washington Post). Of the 149 research presentations receiving news media coverage, 76% were nonrandomized, 25% were small (ie, less than 30 human subjects), and 15% did not involve patients (eg, animal studies). Nearly half (48%) of the covered presentations were not subsequently published in journals indexed by MEDLINE. Notably, some of the most provocative headlines came from these unpublished studies: "Experimental Drug Reverses Fatal Outlook for Cancer"; "Laser Treatment Kills Tumor Without Incisions"; and "Cancer Breakthrough Attacks Genetic Flaws." Even among the 39 presentations covered on page 1 of these newspapers, 17 (44%) were not subsequently published.

Conclusions: Scientific meeting presentations receive substantial attention in high profile by US news media. This coverage is concerning since many of the studies are small, have weak designs, and are not subsequently published.

¹VA Outcomes Group, 111B, White River Junction, VT 05009, USA, e-mail: lisa.schwartz@dartmouth.edu; and ²Center for the Evaluative Clinical Sciences, Dartmouth Medical School, Hanover, NH, USA

Research on e-Journals and Online Information

Ann C Weller

Objective: To determine any different editorial peer review practices or variations in qualitative and quantitative measures among different types of indexed health sciences electronic journals (e-journals).

Design: Comparison of 3 types of e-journals indexed during 2000 in MEDLINE: T1: fully electronic with no print counterpart (identifier: location - Internet only); T2: print (T2P) and electronic (T2E) versions not scientifically coequal (identifier: separate ISSNs for the print and electronic versions); T3: print and electronic versions scientifically coequal (identifier: link to an electronic version and a single ISSN). Data were collected from MEDLINE and the journals’ home pages on 11 T1, 16 T2, and 13 T3 journals randomly selected from Abridged Index Medicus.

Results: Most journals in each category (84.6%-93.7%) have a stated editorial peer review policy. T2E and T2P journals always share a homepage. Significant differences (P<.05, analysis of variance) existed among the e-journals for the presence of editorials (0%-75.0%), letters to the editor (8.3%-87.5%), and case reports (16.7%-93.7%), as well as the average number of items indexed in MEDLINE (22.5-544.5); and the average number of editorials (0.5-18.0), letters (1.5-48.4), and case reports (0.1-47.8). No statistically significant differences existed for the requirement of either structured abstracts (36.6%-68.7%) or original research (81.8%-87.5%), or size of editorial boards (43.7-83.5).

Conclusions: As health sciences e-journals evolve, T1 and T2E journals will probably become more numerous, but now neither has the complexity of traditional print journals. While editors’ statements on editorial peer review are similar, there are differences in number and type of material included in the e-journals. Many T2E publications serve a specialized function, publishing short reports, letters, case reports, or rapid communication and do not have the depth or breadth of print journals. The transformation to electronic publication is progressing rapidly with expectations of its increased impact on the indexed medical literature.

Library of the Health Sciences, 1750 W Polk St, University of Illinois at Chicago, Chicago, IL 60612, USA, e-mail: acw@uic.edu

Christopher Sigouin¹ and Alejandro R Jadad^1,2

Objective: To examine and compare the level of awareness of sources of research evidence on the Internet, available to patients, clinical oncologists, and oncology nurses involved in the management of cancer.

Deswign: Cross-sectional survey of all physicians and nurses and 1998 patients with cancer affiliated with the Hamilton Regional Cancer Centre, in Ontario, Canada, conducted between July 1998 and January 2000. Descriptive statistics and relevant comparisons were made between and within groups using chi-square.

Results: The response rate was 72%, 97%, and 84% for patients, oncologists, and nurses, respectively. More physicians than nurses or patients reported that they look for health information on the Internet (71%, 50%, 15%, respectively; P<.01); that they have heard about the Cochrane Collaboration (75%, 8%, <1%; P<.01) and MEDLINE (100%, 89%, 7%; P<.01). More physicians than nurses reported having heard about PubMed (64% and 28%; P<.01) and CancerLit (100% vs 83%; P=.03), but there were no differences in the proportion that heard about CancerNet (93% vs 89%; P=.69) or OncoLink (82% vs 69%; P=.38). Overall, 17% of patients said they had not heard of the Internet or the World Wide Web.

Conclusions: Some patients have not heard about the Internet and few use it to find health-related information. Some of the most rigorously developed sources of information are still unknown, even to health care professionals. The awareness of information on the Internet varies between patients and health care professionals and within health care professionals.

¹Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada; and ²Program in eHealth Innovation, University Health Network, Eaton Wing EN 6-240, 200 Elizabeth St, Toronto, Ontario M5G 2C4, Canada, e-mail: ajadad@uhnres.utoronto.ca

Anthony George Crocco,¹ Miguel Villasis-Keever,² and Alejandro Jadad³

Objective: Much has been written about the potential harms associated with poor-quality health information on the Internet. The objective of this study was to determine the number and nature of reported cases of harm resulting from alleged misuse of heath information obtained on the Internet.

Design: A systematic review of the literature, including MEDLINE, EMBASE, PSYINFO, CINAHL, and HEALTHSTAR was performed in April 2001 using a refined search strategy. Two authors separately reviewed the abstracts. The goal was to identify articles that describe at least 1 case of harm attributed to the use of health information found on the Internet, published in peer-reviewed and non–peer-reviewed journals. Papers of any format and in any language deemed possibly relevant were obtained. Each full article selected was screened independently and decisions on inclusion were made by consensus. They were divided into those that described harm related to inappropriate use of accurate information, and harm related to use of inaccurate information.

Results. The search yielded 1512 citations and 15 were included. Eight articles reported cases of self-injury associated with accurate Internet information, 4 articles reported harm that resulted in adverse effects after illegal substance abuse, and 1 reported emotional harm resulting from misinterpretation of information. Only 2 articles met the criteria of harm associated with inaccurate information: 1 described poisoning in dogs related to misinformation obtained on the Internet, and the other described a lung cancer patient who self-medicated with an unproved drug and died as a result.

Conclusions: Despite the important number of publications claiming or suggesting potential harm associated with misuse of health information on the Internet, this search failed to find supporting evidence. It remains to be determined whether this is due to publication bias, to absence of studies, or to lack of harm associated with misuse of information found on the Internet.

¹Health Information Research Unit, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; ²Clinical Epidemiology Unit at Pediatric Hospital, Instituto Mexicano del Seguro Social, Mexico; and ³Program in eHealth Innovation, University Health Network, University of Toronto, Eaton Wing EN 6-240, 200 Elizabeth St,Toronto, Ontario M5G 2C4, Canada, e-mail: ajadad@uhnres.utoronto.ca

SUNDAY, SEPTEMBER 16

Publication Bias

Chris A Silagy,¹ Philippa Middleton,² and Sally Hopewell²

Objective: To assess the nature and extent of changes between published systematic reviews and their previously published protocols and to assess any potential impact these changes may have had in introducing bias to the review.

Design: We identified previously published protocols for all new completed new systematic reviews appearing on the Cochrane Library (issue 3, 2000). The text of the published protocol and the completed review were compared electronically. Two raters independently identified changes to the different sections of the protocol. These were classified as none, minor, or major and their potential impact on the review assessed.

Results: Of 66 completed new reviews, we identified a previously published protocol for 47 reviews. Of these, 43 reviews had at least 1 section that had undergone a major change when compared to the most recently published protocol. The greatest variation between the protocol and review was in the methods section where 68.1% of reviews had undergone a major change. Changes that may have resulted in the introduction of bias include narrowing the objectives, adding comparisons, subgroup analyses or new outcome measures, broadening study design criteria, and narrowing the types of participants included.

Conclusions: Although research protocols are likely to remain at least to some extent iterative documents, it is concerning that some of the changes being made to Cochrane reviews could be prone to influence by prior knowledge of the results. Even if many of the changes improve the overall review, the reasons for making these changes should be clearly documented.

¹Monash Institute of Public Health, Monash Medical Centre, Clayton, Victoria 3168, Australia, e-mail: chris.silagy@med.monash.edu.au;
²UK Cochrane Centre, Summertown Pavilion, Middle Way, Oxford OX2 7LG, UK

Carin M Olson,^1,2 Drummond Rennie,^1,3 Deborah Cook,^1,4 Kay Dickersin,⁵ Annette Flanagin,¹ Joseph W Hogan,⁵ Qi Zhu,⁵ Jennifer Reiling,¹ and Brian Pace⁶

Objective: To assess whether submitted manuscripts reporting results of controlled trials are more likely to be published if they report positive results than if they report negative results.

Design: Prospective cohort study of manuscripts submitted to JAMA from February 1996 through August 1999. For inclusion, manuscripts had to report results of a prospective study in which participants were assigned to a treatment or comparison group and that used statistical tests to compare differences between groups. Manuscripts were followed up until the publication decision. We classified results as positive if there was a statistically significant difference (P<.05) reported for the primary outcome. Manuscripts were further classified according to indicators for quality and other study characteristics.

Results: Among 745 manuscripts meeting inclusion criteria, 133 (17.9%) were published: 78 (20%) of 383 with positive results, 51 (15%) of 341 with negative results, and 4 (19%) of 21 with unclear direction of results. The crude relative risk for studies with positive results being published compared with studies with negative results was 1.36 (95% confidence interval [CI] 0.99-1.88). After adjusting simultaneously for quality indicators and other study characteristics, the odds ratio for studies with positive results being published was 1.30 (95% CI 0.87-1.96).

Conclusions: Publication bias may occur even after manuscripts reporting controlled trials have been submitted to a medical journal. However, since associations were not statistically significant, the possibility that the effect observed was due to chance cannot be ruled out.

¹JAMA, Chicago, IL, USA; ²University of Washington Medical Center, Box 356123, Seattle, WA 98195, USA, e-mail: colson@u.washington.edu; ³University of California, San Francisco, San Francisco, CA, USA; ⁴McMaster University, Hamilton, Ontario, Canada; ⁵Brown University, Providence, RI, USA; ⁶News and Information, American Medical Association, Chicago, IL, USA

Carin M Olson,^1,2 Drummond Rennie,^1,3 Deborah Cook,^1,4 Kay Dickersin,⁵ Annette Flanagin,¹ Qi Zhu,⁵ Jennifer Reiling,¹ and Brian Pace⁶

Objective: Follow-up of protocols submitted to an ethics committee, funded studies, and completed clinical trials show that studies with positive results are published more rapidly than studies with negative results. There is no evidence that such bias toward speed of publication occurs once manuscripts have been submitted to a medical journal. This study determined whether submitted manuscripts reporting positive results are published more rapidly than those reporting negative results.

Design: Prospective cohort study of 133 manuscripts reporting controlled trials accrued over 40 months (February 1996-August 1999) and accepted for publication at JAMA. Direction of study results was determined by 2 investigators using objective criteria before the publication decision. Intervals from manuscript to publication were derived from JAMA’s database.

Results: Of the 133 manuscripts, 78 reported positive results (median sample size = 242), 51 reported negative results (median sample size = 361), and 4 had unclear results (median sample size = 322 ). The interval (mean [SD]) from submission to publication was 270 (116) days for manuscripts reporting positive results and 248 (86) days for manuscripts reporting negative results. This difference was not statistically significant (mean difference, 22 days; 95% confidence interval for difference 16 to 59 days; P=.26).

Conclusion: Among manuscripts reporting controlled trials submitted to JAMA, there was no evidence that those reporting positive results are published more rapidly than those reporting negative results.

¹JAMA, Chicago, IL, USA; ²University of Washington Medical Center, Box 356123 Seattle, WA 98195, USA, e-mail: colson@u.washington.edu; ³University of California, San Francisco, San Francisco, CA, USA; ⁴McMaster University, Hamilton, Ontario, Canada; ⁵Brown University, Providence, RI, USA; ⁶News and Information, American Medical Association, Chicago, IL, USA

Grant Review and Ethical Issues

Mark Taylor

Objective: To determine whether reductive grant applications (eg, molecular biological studies) are rated more highly on a consistent basis than integrative applications (eg, behavioral sciences) in the peer review process of the Heart and Stroke Foundation of Canada (HSFC).

Design: The 7 grant peer review committees of the HSFC are based primarily on research approach, ranging from molecular biology to clinical research to behavioral science. The average and median ratings assigned by each of these committees were examined from 1995-1996 through 2000-2001 (n = 2646 grant applications). The funding outcome for each committee was also determined on both a relative and absolute basis.

Results: Those committees that reviewed molecular and cellular applications had the highest average ratings, with those reviewing clinical and behavioral science applications having the lowest ratings. This was seen for all 6 years of the study period with the difference in average rating from the lowest to highest committee ranging from 0.4 to 0.6 (on a 0 to 4.9 scale). This had a major impact on the percentage of grants funded per committee in any given year.

Conclusions: Reductive vs integrative applications do not fare equally well in the HSFC peer review process. While the reasons for this are likely multifactoral, these results suggest that different review criteria, committee structures, and/or other measures may be required to ensure the appropriate evaluation of all applications.

Research Programs, Heart and Stroke Foundation of Canada, 222 Queen St, Suite 1402, Ottawa, Ontario K1P 5V9, Canada, e-mail: mtaylor@hsf.ca

Veronica Yank¹ and Drummond Rennie^1,2

Objective: A 1997 study found that major medical journals had poor rates of reporting on informed consent and ethics committee approval. The current study assesses whether journals have improved their reporting.

Design: Pre-post comparison of clinical trials before and after 1997 in the Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine. Three hundred articles per time period, 60 per journal, were randomly selected. Primary outcome measures were rates of reporting on informed consent and on ethics committee approval.

Results: Informed consent was not mentioned in 26.3% of articles published before 1997 vs 17.7% of articles published after 1997 (P=.011). Similarly, ethics committee approval was not mentioned in 31% of articles published before 1997 vs 18.8% of articles published after 1997 (P<.001). There was no mention of either in 16.0% of pre-1997 vs 9.3% of post-1997 articles (P=.014). Vulnerable populations (eg, children) were the study participants in 29.2% of pre-1997 and 17.9% of post-1997 articles that reported neither protections. In subgroup analyses, journals with the worst initial rates of reporting generally improved the most. BMJ did not describe informed consent in 41.7% of pre-1997 vs 25.0% of post-1997 articles (P=.053). JAMA did not describe ethics committee approval in 41.7% of pre-1997 vs 21.7% of post-1997 articles (P=.019). BMJ, JAMA, and Annals of Internal Medicine had the lowest initial rates of reporting both protections, with 41.7%, 53.3%, and 56.7%, respectively, but improved markedly to 63.3%, 71.7%, and 75.0% (P=.018, .038, .053). The Lancet and New England Journal of Medicine had the best initial rates and showed a nonsignificant trend toward improvement.

Conclusions: Major medical journals have improved their reporting on informed consent and ethics committee approval. But 1 in 10 studies still do not report on either of these protections.

¹Institute for Health Policy Studies and School of Medicine, University of California, San Francisco, c/o 1221 Shrader St, San Francisco, CA, USA, e-mail: vyank@itsa.ucsf.edu; ² JAMA, Chicago, IL, USA

Tammy J Clifford, David Moher, and Nicholas Barrowman

Objectives: To examine the relationships between funding source, trial outcome, and trial quality.

Conclusions: The observed nonsignificant associations between funding source, trial outcome, and trial quality may reflect inadequate power or a true absence of an association. This finding may also reflect limitations inherent in the study’s reliance on voluntary disclosure of financial conflicts of interests. For example, it is not known whether the absence of disclosure in a published report reflects the absence of competing interests among authors or less-than-optimal compliance with journal policies. The need for standardized reporting is suggested by discrepancies between journals, with respect to the manner in which financial conflicts of interest are reported, along with a preponderance of trials with low Jadad scores. The persistence of an excess of published reports with statistically positive results poses a continued challenge.

Thomas C Chalmers Centre for Systematic Reviews, Children’s Hospital of Eastern Ontario Research Institute, 401 Smyth, Ottawa, Ontario K1H 8L1, Canada, e-mail: TClifford@cheo.on.ca

Anu R Gupta, Cary P Gross, and Harlan M Krumholz

Objective: To analyze the nature of disclosures in published randomized controlled trials (RCTs) and adherence of these disclosures to the Uniform Requirements for Manuscripts Submitted to Biomedical Journals, which recommend disclosures of financial conflict of interest including a specific description of "the type and degree of involvement of the supporting agency."

Design: Research outcomes and source(s) of support for the study and authors were abstracted from all RCTs published in 5 journals (Annals of Internal Medicine, BMJ, JAMA, the Lancet, and the New England Journal of Medicine) between April 1, 1999, and March 31, 2000. Chi-square and Wilcoxon rank-sum tests were used to compare groups.

Results: Of the 268 RCTs reviewed, 54% were financially supported solely by nonindustry sources, 24% were supported by industry sources alone, 13% by both, and 9% did not identify source of study support. In the 100 industry-supported trials, 31% did not provide any information about authors’ relations with industry, 48% had an author disclosure printed, and 21% had a presumed conflict based on an author’s address affiliation with industry. Only the Annals of Internal Medicine disclosed the type and degree of involvement of the funding source as specified in the Uniform Requirements. Of the 9 industry-supported studies in this journal, 5 disclosed additional industry involvement in the study through trial design and data analysis. Compared to non–industry-supported trials, industry supported trials were significantly more likely to have more enrollees (378 vs 199, P=.001); to be multicenter (82% vs 58%, P<.0001); and to conclude that experimental therapy was better than control therapy in the abstract (84% vs 71%; P=.02).

Conclusion: While sources of study support and author specific conflicts of interest are being disclosed, description of the type and degree of involvement of the supporting agency as specified by the Uniform Requirements is not routinely published.

Department of Medicine, Yale University School of Medicine, PO Box 208025, IE-61 SHM, New Haven, CT 06520-8025, USA, e-mail: anu.gupta@yale.edu

Ronald M Davis^1,2 and Marcus Müllner^2,3

Objective: To assess the degree of editorial independence at a sample of medical journals and the relationship between the journals and their owners.

Design: Survey of the editors of 33 medical journals owned by not-for-profit organizations (associations), including 10 journals represented on the International Committee of Medical Journal Editors (9 of which are general medical journals) and a random sample of 23 specialist journals with high impact factors that are indexed by the Institute for Scientific Information.

Results: Of the 33 editors, 23 (70%) reported having complete editorial freedom, and the remainder reported a high level of freedom (a score of >8, with 10 indicating complete editorial freedom and 1 indicating no editorial freedom). Nevertheless, a substantial minority of editors reported having received at least some pressure in recent years over editorial content from the association’s leadership (42%), senior staff (30%), or rank-and-file members (39%). The association’s board of directors has the authority to hire (48%) or fire (55%) the editor for about half of the journals, and the editor reports to the board for 10 journals (30%). Twenty-three editors (70%) are appointed for a specific term (median term=5 years). Three fifths of the journals have no control over their profit. The majority of journals use the association’s legal counsel and/or media relations staff.

Conclusions: Most editors report having complete or nearly complete editorial freedom, although many receive modest pressure from their owners over editorial content. Stronger safeguards are needed to give editors protection against this pressure, including written guarantees of editorial freedom and governance structures that support those guarantees. Strong safeguards are also needed because editors may have less freedom than they believe (especially if they have not yet tested their freedom in an area of controversy).

¹Center for Health Promotion and Disease Prevention, Henry Ford Health System, One Ford Place, 5C, Detroit, MI 48202-3450, USA,
e-mail: rdavis1@hfhs.org; ²BMJ; London, UK; ³University of Vienna Medical School, Vienna, Austria

Debra Parrish^1,2 and David Bruns³

A laboratory test led to false diagnoses of cancer, and patients suffered harm from resulting surgical procedures. Two peer-reviewed journals, A and B, published reports. A third journal C had rejected the paper published in journal B. A test manufacturer, sued by patients, sought access to records of manuscripts from journals A and C. A related manuscript was in press at journal A.

Questions considered included: should third parties be provided access to publication-related records (1) from journal A that published the initial report (a peer-reviewed letter to the editor); (2) from journal C that rejected 1 paper; and (3) from the author of the unpublished third paper?

Journal A indicated to the court that, by journal policy, the review process was confidential, and that authors submitted their work trusting in that policy. The journal’s motion to quash the subpoena was not contested. Journal C asked permission from the author of the rejected paper to supply the subpoenaed records. The author consented.

In addition, the court ruled that the unpublished paper was confidential: "[The author’s] interest cannot be protected unless he is able to retain control of the article until it is complete; and, where the purpose of the article is publication, complete must be defined as published. Only in this way is [the author] able to preserve the integrity of his work, protect his intellectual property and safeguard his reputation and credibility. ‘Academicians engaged in pre-publication research should be accorded protection commensurate to that which the law provides for journalists.’"

These cases reflect the reactions of journals to third-party subpoenas and the confidentiality of peer review accorded by US courts.

¹Debra M Parrish, PC, 615 Washington Rd, Suite 200, Pittsburgh, PA 15228, USA, e-mail: debbie@parrishlawoffices.com; ²University of Pittsburgh Program Teaching Survival Skills and Ethics for Scientists, Pittsburgh, PA, USA; ³Clinical Chemistry, Charlottesville, VA, USA

Mike Farthing,¹ Richard Horton,² Richard Smith,³ and Alex Williamson³

The Committee on Publication Ethics (COPE) is an informal group founded in 1997 as a response to growing anxiety about the integrity of authors submitting studies to medical journals. Founded by British medical editors, including those of the BMJ, Gut, and the Lancet, the committee had 5 aims:

1. To advise on cases brought by editors. Cases are presented anonymously, and full responsibility for action remains with the reporting editor. The committee has so far considered 103 cases. In 80 cases there was evidence of misconduct. Several cases have been referred to employers and to regulatory bodies like Britain’s General Medical Council. The commonest problems were undeclared redundant publication or submission (29 cases), disputes over authorship (18), falsification (15), failure to obtain informed consent (11), performing unethical research (11), failure to gain approval from an ethics committee (10), and fabrication of data.

2. Publish an annual report describing the cases it considers. The committee has published 3 annual reports and established a Web site (www.publicationethics.org.uk).

3. Draft guidance on these issues. The committee drafted guidelines and after extensive consultation published them in 1999 (available on the Web site). They have been adopted by many journals.

4. Promote research into publication ethics. Little has been achieved so far.

5. Consider offering teaching and training. The committee has run 2 seminars, and individual members of the committee have lectured and taught on research misconduct.

COPE has also been co