A SOFTWARE PROGRAM TO AID IN PEER REVIEW
Alvar Loria,1 and Gladys Faba,2 for the ARTEMISA Selection Committee
1Instituto Nacional de Nutrición Salvador Zubiran, 14000 Mexico City, Mexico; 2Centro Nacional de Informacion y Documentacion en Salud, Mexico City, Mexico
Objective: To characterize a personal computer-based software program developed as an aid to peer review of medical papers.
Design: The software is a Windows-based application that records automatically a numeric score to a series of questions related to 8 sections of
scientific papers (introduction, methods, results, and discussion, plus 4 other sections). The questions and sections vary according to type of
paper (original reports, case reports, or reviews), and the final output is a score with a maximum of 100 for a "perfect" paper. The software was
tested using a single reviewer to judge 289 papers (169 original reports, 50 case reports, and 70 reviews) from 44 Mexican medical journals.
All statistical analysis of scores were done with nonparametric tests.
Results: The paper scores ranged from 29 to 97 with slightly higher median and less dispersion of scores for reviews as compared with original
reports and case reports, but these differences did not reach significance. Two observations suggest that the software operated reasonably well: a)
there were some differences in the section scores by type of paper that agreed well with differences in their complexity; b) the journal scores showed
an association with their number of original papers and their percentage of original papers (Kruskal-Wallis test, P=.06 and 0.07, respectively).
Conclusions: The software operated reasonably well when used to compare the relative quality of 289 papers. The validity of the program is restricted
in this study to the experience of 1 reviewer. An analysis of the raw scores helped in detecting some ambiguous and redundant questions that have been modified
in an improved version. The program has a potential as a training tool for inexperienced reviewers or as a scorekeeper for experienced peer
reviewers.
Return to Session Information
