|
Judges do have the final word on scientific evidence, by virtue of 1993's landmark Daubert v Merrell Dow Pharmaceuticals, Inc. which affirmed that the trial judge must act as the "gatekeeper" for accepting or rejecting scientific testimony. Under the Federal Rules of Evidence (FRE), a federal trial judge "must ensure that any and all scientific testimony or evidence admitted be not only relevant to the case but also scientifically reliable." A large task indeed.
J. Michael Veron, writing in a recent issue of the Louisiana Law Review, reduced Daubert to a four-step test for admissibility of expert testimony in federal court (see Table I).
Obviously, all of these elements are important; however, in this article, we will focus on critiquing scientific literature to ensure that the evidence you present will be accepted by the court, specifically examining the peer review and publication process.
Two general questions sould be asked regarding the testimony in question as to the peer review of the theory or technique in question: How "peer" is it and how "reviewed" is it? Peer review is typically defined as "the evaluation by experts of the quality and pertinence of research or research proposals of other experts in the same field." The idea posits that if a study is subjected to such evaluation and is found to be valid that the science is therefore sound and should be allowed as evidence. Scientific journals enjoy the prestige of being referred to as a peer review journal for the status in the scientific community it provides. However, such is not always the case. First, both those who author articles and review them are not necessarily representative of the mainstream scientific community and may, in fact, be proponents of scientific theories that have not been proven or in many cases are not considered to be provable. Additionally, many of the publications indexed are published not by mainstream organizations but by fringe groups added to the present scientific "canon" on the basis more of current awareness than true scientific inquiry. Electronic publishing on the Internet is also adding a whole new dimension to the "peer" issue, ensuring that any interest group can have "peer reviewed" publications.
| 100% |  |
| 80% | |
| 60% | |
| 40% | |
| 20% | |
| 0 | |
| (3,211 Journals equals 100%) | |
 | Approximate Number of Journals Indexed by NLM for Index Medicus |
 | Approximate Number of Peer Reviewed Journals (60.5% mean) |
 | Approximate Number of Journals with Peer Review Statements (30.25% mean) |
Apart from the problem of the credentials of those reviewing scientific work is the entire concept of peer review. It is, in fact, being steadily replaced in many journals by editorial review, with standards often unrelated to true scientific inquiry. For example, in most studies accepted for publication under editorial review, the number one critieria is that of timeliness. As could be expected, good science and timeliness rarely go hand-in-hand. Of course, there are attempts to maintain high scientific standards. According to Sheldon Kotzin, chief of the Bibliographic Services Division at the National Library of Medicine, an independent committee makes decisions on the articles the Library will index. While timeliness is a consideration, he says the true rationale of selection remains focused on the quality of the article. Still, of the thousands of articles indexed by the National Library of Medicine's Index Medicus, only 56% to 65% are from peer reviewed journals and over half of those that are do not publish clear statements on their peer review process (see Table II). A study of peer review at one prestigious journal found that, the "review" took, on average, less than two hours per article. While relevance to other studies may be ascertained in such a short review, it is difficult to see how the reliability of scientific methodologies and the quality of their findings can be confidently verified with such slight study.
Beyond the exclusion of testimony such review is likely to prompt, the effect of this type of peer review on science itself can be staggering. A survey of recent work in the biomedical and health care literature reveals a large body of work that feature enough methodological and statistical flaws to make the conclusions drawn invalid. For instance, thirty years ago in a celebrated review, Schor and Karden revealed that, out of 149 papers submitted by 10 leading medical journals, a paltry 28% were deemed acceptable, 67% were judged deficient but improvable, and 5% were completely unsalvageable. Between 44% and 46% of the articles were found to contain statistical mistakes. In the intervening years, the problem has only gotten worse. In an evaluation of original articles submitted to the American Journal of Diseases in Children for 1983-84, statistical irregularities were discovered in 54.2% and 50.5% of the articles, respectively. Thus the quality of the "review" not only damages individual cases, but can incur a loss of confidence in forthcoming cases in which scientific testimony promises to be an important part of a litigant's evidence.
Perhaps the most glaring example of the problem of peer review as it is often practiced now is the ramifications of a flawed study featuring researchers at Tulane University studying the synergistic effect of pesticides on the endocrine system. The research was published in a peer reviewed journal (Science) in June of 1996 before any replication efforts of the research had been undertaken. The "timeliness" of the article coincided with the much-publicized Food Quality Protection Act of 1996, one provision of which mandated that a screening program for pesticides with "estrogenic properties" be implemented. In the year since the study was published, however, the researchers at Tulane--as well as scientists at Duke University, Texas A&M, the Chemical Industry Institute of Technology, and the National Institute of Health--have been unable to replicate the findings. Researchers in the United Kingdom were also unsuccessful at replicating the results. After severe criticism, the Tulane researchers formally recanted their findings (Science 1997; 277: 459-463), admitting there was "a fundamental flaw" in the study's methodology.
Although peer review and publication is only one small part of the admissibility test, it has often taken on a much larger role, particularly by the courts, and as our examination has shown may not always be a reliable barometer.
|
Selected Bibliography Following
A
federal district judge ruled July 15 that expert testimony on Multiple Chemical
Sensitivity (MCS) is too speculative to be considered "scientific knowledge,"
and therefore excluded it from a case in New York. In Frank v New York,
(DC NNY, No. 95-CV-399) in which state workers claimed ill health effects from
being exposed to pesticides, U.S. District Court Judge Thomas J. McAvoy ruled
MCS to be "untested, speculative, and far from generally accepted in the medical
or toxicological community," and that it failed to meet the standard of evidentiary
reliability as stated in Daubert
Peer Review Reference Tip: The Serials Directory:, published by EBSCO, contains a separate international listing of active serials that purport to be peer reviewed or refereed.
Copyright © 1996-2005 Medical &
Toxicological Information (MTI) A Division of NICOM, Inc. All rights
reserved
703 684-4636 703 MTI-INFO
Home
About MTI
Contact MTI
News/Pub
Services
