PEER-REVIEWER BIAS AGAINST UNCONVENTIONAL MEDICINE?
Edzard Ernst1 and Karl-Ludwig Resch2
1Department of Complementary Medicine, Postgraduate Medical School, University of Exeter, 25 Victoria Park Rd, Exeter EX2 4NT, UK;
2Forschungsinstitut für Balneologie und Kurortwissenschaft, Bad Elster, Germany
Objective: To test the hypothesis that there is reviewer bias against testing of an unconventional drug.
Design: Randomized, controlled, double-blind study of peer review. From a convenience sample of 291 medical doctors from a wide variety of specialties (drawn from a list of participants of an interdisciplinary, international conference), study participants were randomly assigned to receive 1 of 2 versions of a manuscript. Version M dealt with an in-vitro experiment on a mainstream drug (metoprolol), while the otherwise identical version V used a highly unconventional yet commercially available drug (beef spleen cell extract). All participants were asked to complete a standardized evaluation sheet to provide ratings from "poor" to "excellent" on a visual analog scale for a set of predefined quality criteria. Participants were debriefed after completion of the study by means of a letter explaining the experiment and asking for permission to include each participant's data in the analysis.
Results: Overall response rate was 63% (n=183); 127 responses were deemed suitable for evaluation. This sample size provided an 80% chance to detect a 10% difference in ratings between groups at the level of 5% (2-sided). No differences in ratings between the 2 versions of the manuscript were observed (P values for all variables >0.2). Ratings covered the entire range of the visual analogue scales.
Conclusions: In this study, there was no reviewer bias against testing an unconventional drug. The low interrater reliability, however,
suggested inadequate validity of peer review.
Return to Session Information
