by Keith L. Black, MD

How effective is the peer-review process in producing rigorous, valid articles and advancing science?

Most biomedical journal editors are practicing clinicians and self-taught, part-time editors. They usually send papers out to two reviewers but use their own judgment to determine whether to accept a paper. The goal of reviews is to select the best papers for publication but also to have high-quality reviews to improve papers through the review process.

Even though the review process is central to research and the advancement of science, few studies have analyzed how to select good reviewers. What few studies have been done have found that young reviewers at top institutions or reviewers known to the editor produce the best reviews. Research training and postgraduate work does not appear to be related to review quality, although those trained in epidemiology or statistics were more likely to produce good reviews. Also, the longer reviewers spent on a review, especially if it was more than three hours, the better the review.

Even so, the British Journal of Medicine started making reviewers' names known to authors in 1998, while BioMed Central, an online publishing house, posts the reviews and authors' responses for its 40 medical journals on the Web. No one has studied the change, if any, in quality of articles published under these systems.

The review process does have biases. All reviewers, whether from the United States or from outside the United States, evaluate papers submitted by U.S. authors more favorably, with U.S. reviewers having a significant preference for U.S. papers. This is not solely a language bias, since U.S. reviewers show a bias for U.S. publications even over British publications, both of which, of course, are in English.

Other biases abound. There is bias toward publishing positive outcomes over negative outcomes, while review articles tend to show a bias toward the discipline in which the author practices.

Once articles are published, although those that go through a systematic peer-review process have higher methodologic and reporting quality, non-peer reviewed journals and those employing a non-systematic review process often have more tables, figures, photographs and are easier to read due to their larger font size. These factors may make them more appealing to physicians and may increase the impact of findings published in non-peer reviewed articles over drier appearing, peer-reviewed articles.

Even studying the peer-review process is difficult, requiring the cooperation of many parts of the scientific community. The National Institutes of Health, Medical Research Council and the European Union are reluctant to fund research into the scientific process, focusing instead on studying diseases.

For all of its problems, however, peer review is likely to remain the cornerstone of science because, to paraphrase Winston Churchill's comment about democracy, it is the worst system for selecting papers to publish, save for all the others.