The Safety Institute defended its October study on the injury and fatality risk of different energy absorbing guardrail end treatment designs based on two states’ data, in the wake of a highly critical review released by the Federal Highway Administration late Friday afternoon.
The Safety Institute’s case-control study, In-Service Evaluation of FHWA-Accepted Guardrail Terminals, conducted by Kevin Schrum of the University of Alabama at Birmingham (UAB) School of Engineering, examined eight years of severe injury and death data for crashes that occurred in Missouri and Ohio involving five different guardrail terminal designs. The study found that the ET-Plus design, manufactured by Trinity Industries, Inc. was a statistical outlier in injury and fatality crashes involving guardrails — 1.36 to 1.95 times more likely to produce an injury and 2.86 to 3.95 times more likely to produce a fatality than the ET-2000 end terminal design.
The FHWA’s Peer Review of Report: “Relative Comparison of NCHRP 350 Accepted Guardrail Terminals” attacked the study’s methodology and repeatedly called its results “unreliable.”
“We absolutely stand behind the results of our study,” said Sean Kane, The Safety Institute’s founder. “We undertook it because there was a lack of information about the field performance of the ET-Plus, and because the FHWA had been content to brush those questions aside. It is disappointing that the agency expended its resources to dismiss the study, rather than use it to build on the body of knowledge. The history of guardrail design is the search for a device that helps vehicle occupants survive an impact. The energy absorbing end-terminal has done just that; however, questions about the field performance of ET-Plus continue which is why it’s important to study the data to better understand what is happening.”
The FHWA engaged The John A. Volpe Transportation Center, which sent the UAB study to four outside reviewers – academics from institutions ranging from the University of Toronto to the University of Florence, Italy. The review concluded:
All four reviewers raised concerns about limitations or flaws in the study’s methodology, which led all the reviewers to question the validity of the study’s findings and conclusions. The reviewers cited reasons to doubt assumptions made in the report, expressed concern about potential biases in the approach to selecting the crashes included or excluded from the analysis, and questioned the decision to consider only fatal and severe injury crashes (and not also less severe injury and property damage only crashes). All four reviewers observed that the findings and conclusions are either questionable or invalid.
Although the TSI study had undergone an independent review last fall, in light of the FHWA’s criticisms, The Safety Institute sought an examination of the reviewers’ comments from statistician Randy Whitfield of Quality Control Systems. Whitfield expressed surprise that the reviewers focused on the soundness of the methodology. Case-control studies have long been an accepted tool in transportation research studies. He also questioned the FHWA’s decision identify the reviewers, but not the authorship of specific critical remarks, likening the practice to an anonymous letter to the editor. (The entirety of Whitfield’s observations are found in the section of this article entitled “The Value of a Case-Control Study.)
Another Skirmish over the ET-Plus
This controversy is just the latest in a series over the performance of ET-Plus guardrails and the federal government’s response to questions raised by state highway officials, the media, victims’ families, safety advocates, other guardrail designers and a competitor, Joshua Harman.
In 2012 Harman, the president of SPIG Industries in Bristol, Virginia alleged that that sometime between 2002 and 2005, Trinity modified the design of its original guardrail end terminal design, the ET-Plus, causing it to fail in crashes and injure and kill occupants in striking vehicles. The newer versions of the ET-Plus, manufactured in 2005, bear a dimensional change to the width of the guide channel, or “feeder chute,” through which the rail is extruded. Harman alleged that the rail jams in the narrower channel causes the rail to jam, ending away, the rail jams in the chute, causing it to fold in half, forming a spear that can penetrate the striking vehicle. Manufacturers must seek approval from the FHWA when making substantive changes in highway equipment designs in order to be considered a qualified vendor, eligible for federal government reimbursement when states purchase approved products.
Harman took his concerns to the FHWA, the American Association of State Highway Transportation Officials and the media. The latter publicized his claims and state Department of Transportation officials undertook their own informal surveys, in which a few members reported poor performance issues in the ET-Plus. Instead of conducting an independent investigation, the FHWA accepted seven-year-old tests from Trinity purporting to show that the re-designed ET-Plus end terminal passed the most severe test.
Meanwhile, Harman sued Trinity, a global manufacturer of highway safety equipment, under qui tam provisions of the False Claims Act, in which a private individual can sue federal contractors on behalf of the government, alleging fraud. In his suit, Harman alleged that Trinity changed the design without notifying the FHWA, as required, until seven years later, when he brought his concerns to the agency.
Harman’s first whistleblower suit ended on July 18 in a mistrial, over a dispute regarding the deposition of Dr. Dean Sicking, the University of Alabama professor who designed the ET-2000. But on October 21, a jury in Marshall County, Texas found that Trinity had knowingly made a false claim to the government, and awarded the Federal Highway Administration and Harman trebled damages of $175 million. The trial revealed that the dimensional changes saved the company $50,000 annually, and that company officials’ decision not to inform the FHWA was deliberate. The Texas Transportation Institute, an arm of Texas A & M University, invented the ET-2000 and the ET-Plus, and was responsible for conducting the testing that would be submitted to the federal government in support of its application for approval. TTI agreed to the change without telling the government, and drafted a test report that failed to reference the change to the guide channel and other changes to the original design, first approved in 1999. Trinity President Gregg Mitchell testified that Trinity had no meetings to discuss the dimension changes and performance issues with the ET-Plus. In fact, the company did no evaluations of the product.
After the trial, the trickle of states that had dropped the ET-Plus from their lists of approved products turned into a flood. More than 40 states have removed the ET-Plus from their qualified products lists. The FHWA ordered Trinity to conduct a new round of tests. But the protocol it chose was immediately criticized, Sicking and Harman’s attorneys wrote to the FHWA urging the agency ensure that the end terminal is tested in a configuration that mimics the way it has failed in the field.
The FHWA declined to test the ET-Plus at a low-impact angle, the mode in which the guardrail terminal system has alleged to have failed in the field. The FHWA also allowed Trinity to select the end terminals and determined that the testing would be done by the Southwest Research Institute, which has ties to Trinity.
The Value of a Case-Control Study
Guardrail energy absorbing end terminals are designed to lessen the severity of a crash, by allowing the striking vehicle to ride down the crash forces safely, without deflecting the vehicle back onto the roadway. Since the 1960s, guardrail end designs have evolved from blunt-end terminals, which often acted like spears, penetrating the vehicle compartment, to twist turn-down designs that served as a ramp, causing vehicle rollovers. Today, the W-Beam guardrail with an energy absorbing end is the most commonly used energy absorption barrier system. In this design, the end terminal rail is deformed away from the striking vehicle, either by flattening, cutting or kinking the rail.
TSI’s case-control study, sponsored by The Safety Institute and the Missouri Highways and Transportation Commission, analyzed single-vehicle ran-off-road crashes that occurred in Ohio, between 2005-2013, and Missouri, between 2005-2014, in which contact with the guardrail end was identified as the Most Harmful Event. The crash performance of four designs was assessed against that of the ET-2000, the first crashworthy energy-absorbing end terminal, also manufactured by Trinity.
Sample sizes in this two state study were insufficient to reliably assess the comparative safety performance of three of the designs, but there was a sufficient sample size to compare the performance of the ET-Plus against that of its predecessor. The study found that the ET-Plus was 1.36 to 1.95 times more likely to produce an injury and 2.86 to 3.95 times more likely to produce a fatality than the ET-2000 end terminal design.
The researchers cautioned that further study was needed, and that “the results and conclusions pertinent to one state are not necessarily pertinent to any other state.”
The initial release of the study in September entitled, “In-Service Evaluation of FHWA-Accepted Guardrail Terminals” was revised after Trinity Industries’ expert released an unsigned review of the study’s methodologies and results. Author Kevin Schrum, considered Trinity’s analysis of the initial report, sought input from credentialed reviewers, and revised the analysis to reflect their comments.
In this instance, The Safety Institute asked statistician Randy Whitfield of Quality Control Systems to evaluate the FHWA peer report. Whitfield offered the following observations:
“This post-publication review commissioned by the Federal Highway Administration (FHWA) of “Relative Comparison of NCHRP350 Accepted Guardrail Terminals” by Kevin Schrum is remarkable in many ways. One of the most striking is that the reviews are unsigned, although they are collectively known to have been written by a panel of four eminent experts. The FHWA states that granting anonymity to the reviewers protects “the independence of the reviews.”
Of course it is true that anonymity is traditionally granted to reviewers in the prepublication editorial process for scientific research in order to elicit candid criticism. But in this instance, the shielding of the review authors’ identities is much more like the publication of an anonymous letter to the editor. This practice allows the FHWA to present to the public a collection of very negative reviews, while allowing any individual reviewer to plausibly assert that specific expressed opinions are not their own.
It is highly unusual for scientific journals to publish unattributed, public comments or letters to the editor after the publication of the research. Why then would it be necessary for the government to keep secret the identity of reviewers it relies upon? Because of this secrecy, we can’t know for certain if the reviewer who said: “The conclusions [of Dr. Schrum’s study] are basically invalid because the method is fundamentally flawed” is, in fact, one of the reviewers who has in the past used the same method in their own published research.
For example, it would be reasonable to ask if the reviewer who criticized the case-control methodology of Dr. Schrum’s study could be the same scientist who once wrote: “The case-control design is well established in epidemiology where it is used to relate risk factors within a study population to a particular outcome or disease.” If the reviews were signed, the question could be properly addressed and the apparent contradiction explained.
It is even more relevant to note that, as recently as 2010, the FHWA itself published important guidance to public agencies interested in the effectiveness of crash modification countermeasures. The FHWA’s report noted that: “Case-control studies assess whether exposure to a potential treatment is disproportionately distributed between the cases and controls, thereby indicating the likelihood of an actual benefit from the treatment.” (A Guide to Developing Quality Crash Modification Factors, FHWA, 2010, page 30).
In this context, it is very surprising that the FHWA would widely disseminate reviewers’ comments such as “The findings are not sound and defensible because the methodology is not correct…” or “Therefore the report provides no basis for forming a judgment about the relative safety of various ET types,” or even “The validity of the conclusions is questionable because the analytical method used to arrive at these conclusions is inappropriate.” (pages 1 and 2). If the case-control method is basically unsound, why didn’t the FHWA say so five years ago?
It is very surprising to find the need in 2015 to defend the use of the case-control method employed by Dr. Schrum. A search of Google Scholar of the combined terms, “case control” and “injury epidemiology” returns “about 1,330 results” in 0.09 seconds. If the FHWA wanted to determine the basic validity of Dr. Schrum’s methods, why not simply search Google Scholar and plug in the names of their panel of experts?
Like any research, “Relative Comparison of NCHRP350 Accepted Guardrail Terminals” is very far from perfect. The study might well be refuted, qualified, or even strengthened by the use of additional data and better methods. Given the current, pressing importance of this topic, public health would be better served by the swift commencement of additional, competent research, rather than by a collection of unsigned critiques pronouncing as invalid a basic methodology in use in the highway safety field for more than fifty years (see Haddon, W. Jr., Valien, P., McCarroll, J.R., Umberger, C.J., “A Controlled Investigation of the Characteristics of Adult Pedestrians Fatally Injured by Motor Vehicles in Manhattan,” Journal of Chronic Diseases, 1961, 14:6:655-678.)
Kane added: “We also fully understand the limitation of a case-control study. But like any other piece of research, it provides additional data and support in our understanding of the performance of guardrail end terminals. Unfortunately, the agency is going out of its way to ignore important data about a reasonable hypothesis – that the physical change to this mechanical device would have had an effect on its safe performance.”