Office of Inspector General Audit Report INADEQUATE DATA AND ANALYSIS UNDERMINE NHTSA’S EFFORTS TO IDENTIFY AND INVESTIGATE VEHICLE SAFETY CONCERNS National Highway Traffic Safety Administration Report Number: ST-2015-063 Date Issued: June 18, 2015 Memorandum U.S. Department of Transportation Office of the Secretary of Transportation Office of Inspector General Subject: From: To: ACTION: Inadequate Data and Analysis Undermine NHTSA’s Efforts To Identify and Investigate Vehicle Safety Concerns National Highway Traffic Safety Administration Report No. ST-2015-063 Calvin L. Scovel III Inspector General Date: Reply to Attn. of: June 18, 2015 JA-30 Secretary of Transportation National Highway Traffic Safety Administrator Since February 2014, the General Motors Corporation (GM) has recalled 8.7 million vehicles in the United States 1 due to an ignition switch that can unexpectedly move from the “run” or “on” position to the “accessory” or “off” position, shutting down the engine and disabling power steering, power brakes, and air bags. More than 110 fatalities and 220 injuries have been linked to the vehicles’ defective ignition switches. 2 The National Highway Traffic Safety Administration’s (NHTSA) Office of Defects Investigation (ODI)—which is responsible for identifying and investigating potential vehicle safety issues and requiring recalls when warranted—looked at GM air bag non-deployments as a potential safety issue starting in 2007. However, it ultimately decided not to investigate the problem and never identified the ignition switch defect as the root cause. In an April 2014 hearing before the Senate Commerce, Science, and Transportation Committee’s Consumer Protection, Product Safety, and Insurance Subcommittee, NHTSA’s Acting Administrator 3 testified that the Agency and the Department’s Office of General Counsel were assessing ODI’s process in light of the GM recalls. 4 The Secretary of Transportation also requested that we assess 1 Recalled vehicles include Chevrolet Cobalts and HHRs, Saturn Ions and Skys, and Pontiac G5s and Solstices that were manufactured between 2003 and 2011. 2 GM’s ignition switch compensation fund had approved 114 death and 229 injury claims as of June 12, 2015. 3 At the time, NHTSA’s Deputy Administrator served as the Agency’s Acting Administrator. 4 The Department issued a report summarizing its review results and planned actions on June 5, 2015. We did not assess the Department’s review as part of this audit. 2 NHTSA’s vehicle safety procedures and determine whether NHTSA had information on GM’s ignition switch issues. Accordingly, we assessed ODI’s procedures for (1) collecting vehicle safety data, (2) analyzing the data and identifying potential safety issues, and (3) determining which of these issues warrant further investigation. In doing this, we considered how those procedures affected ODI’s handling of concerns related to the GM ignition switch. We examined information available to NHTSA prior to GM’s 2014 recall announcement; however, our audit did not assess whether GM fully disclosed information on the ignition switch issue to NHTSA. In addition to the Secretary’s request, we are also conducting a separate audit to assess NHTSA’s actions to implement OIG’s 2011 5 recommendations aimed at strengthening ODI’s process for identifying and addressing safety defects. We plan to report our findings on this topic later this year. We conducted our work in accordance with generally accepted Government auditing standards. See exhibit A for our full scope and methodology. RESULTS IN BRIEF ODI’s processes for collecting vehicle safety data are insufficient to ensure complete and accurate data. Deficiencies in ODI’s vehicle safety data are due in part to the Agency’s lack of detailed guidance on what information manufacturers and consumers should report. For example, ODI regulations specify 24 broad codes for categorizing early warning reporting data for vehicles. However, according to ODI, an average vehicle may have over 15,000 components. Without detailed guidance, decisions regarding key aspects of early warning reporting data are left to the manufacturers’ discretion—resulting in inconsistent reporting and data that ODI investigative chiefs and vehicle safety advocates consider to be of little use. Further, ODI’s processes for verifying that manufacturers submit complete and accurate early warning reporting data are insufficient. For example, in May 2014, ODI officials told us that one vehicle manufacturer reported less early warning reporting data than comparable manufacturers. However, ODI took no enforcement action until the manufacturer self-reported the omission of 1,700 death and injury claims in October 2014, even though ODI contacted the manufacturer about inconsistencies in its reporting in late 2011 or early 2012. Consumer complaints—ODI’s primary source for identifying safety concerns— often lack detail, including information to correctly identify the vehicle systems involved. In the GM case, ODI received data on the ignition switch defect as early as 2003. Some of these data specifically described the ignition switch problems; however, other information lacked sufficient detail or was inconsistently categorized. 5 Process Improvements Are Needed for Identifying and Addressing Vehicle Safety Defects, (OIG Report Number MH-2012-001), Oct. 6, 2011. OIG reports are available on our Web site at: www.oig.dot.gov. 3 Weaknesses in ODI’s processes for analyzing vehicle safety data further undermine ODI’s efforts to identify safety defects. Specifically, ODI does not follow standard statistical practices when analyzing early warning reporting data, such as establishing a base case for what statistical test results would look like in the absence of safety defects. Consequently, ODI cannot differentiate trends and outliers that represent random variation from those that are statistically significant. In addition, ODI does not thoroughly screen consumer complaints. For example, ODI’s initial screening of the roughly 330 complaints received daily is not thorough, and about 90 percent of complaints are set aside. While screeners are encouraged to query all complaints for similar issues in their area of concentration, half of them told us that they do not consistently do this. Finally, ODI does not adequately train or supervise its staff. For example, NHTSA has a training plan for ODI staff, but it has not implemented this plan. As a result, ODI’s preinvestigative staff told us they have received little or no training in their areas of concentration, some of which are technologically complex. Collectively, these weaknesses have resulted in significant safety concerns being overlooked. For example, in June 2007, GM provided ODI with a State trooper’s report that identified the 2005 Chevrolet Cobalt’s ignition switch as a possible cause of air bag non-deployment during a fatal accident. However, two ODI staff who reviewed the report in 2007 did not note this potential link when documenting their reviews. Additionally, ODI officials told us that at the time, they were uncertain under what conditions the air bags were supposed to deploy. ODI’s process for determining when to investigate potential safety defects is also insufficient to prompt needed recalls and other corrective actions. While ODI has identified factors for deciding whether an investigation is warranted, it has not developed sufficient guidance or reached consensus on how these factors should be applied. ODI emphasizes investigating issues that are most likely to result in recalls, which has led to considerable investigative duties being performed during the pre-investigative phase, often by screeners who are not trained to carry out these responsibilities. In addition to these shortcomings, ODI’s investigation decisions lack transparency and accountability. Specifically, ODI does not always document the justifications for its decisions not to investigate potential safety issues and does not always make timely decisions on opening investigations. In the GM case, ODI considered a proposal to investigate air bag non-deployments in the Chevrolet Cobalt and Saturn Ion in November 2007 but did not document why it decided not to investigate. Further, NHTSA’s Associate Administrator for Enforcement directed ODI to gather more information on the issue after reports of fatal accidents associated with the air bag non-deployments. However, the ODI screener responsible for monitoring the issue left NHTSA in 2008, and the Defects Assessment Division Chief did not reassign that responsibility. ODI also missed other opportunities to investigate the ignition switch when new evidence came to light in subsequent years. 4 We are making recommendations aimed at improving ODI’s processes for collecting and analyzing vehicle safety data and for determining which potential safety issues warrant investigation. BACKGROUND NHTSA, established by the Highway Safety Act of 1970, 6 administers highway safety and consumer programs intended to reduce deaths, injuries, and economic losses resulting from motor vehicle crashes. NHTSA’s ODI is responsible for reviewing vehicle safety data, identifying and investigating potential vehicle safety issues, and requiring and overseeing manufacturers’ vehicle and equipment recalls (see table 1). NHTSA reports that it has influenced, on average, the recall of nearly 9 million vehicles every year since 2000. Table 1. ODI’s Vehicle Safety Oversight Process Phase Number of Staff Description Pre-Investigation 13 ODI collects and analyzes vehicle safety data to identify and select potential safety issues for further investigation. Investigation 20 ODI investigates the potential safety issue to determine whether a recall is warranted. Recall management 8 ODI ensures that manufacturer recalls comply with statutory requirements. Source: OIG analysis ODI’s pre-investigative phase includes four key elements: • Collection and analysis of early warning reporting data. The Transportation Recall Enhancement, Accountability, and Documentation (TREAD) Act 7 of 2000 authorized NHTSA to require manufacturers to report on a variety of early warning data. These data include property damage claims, consumer complaints, warranty claims, and field reports from incidents involving certain vehicle components and conditions defined in NHTSA regulations. 8 In addition, manufacturers are required to report all death and injury claims and notices. ODI’s Early Warning Division staff 9 are responsible for verifying that manufacturers submit these data, prioritizing the data using statistical tests, and identifying and referring potential safety trends to the Defects Assessment Division for further analysis. 6 Pub. L. 91-605. Pub. L. 106-414. 8 Title 49, Code of Federal Regulations (CFR), Part 579. 9 The Early Warning Division currently has four staff including two safety defects analysts, one statistician, and one safety defects engineer. 7 5 • Collection and analysis of consumer complaints. ODI receives consumer complaints through a variety of sources including letters, vehicle safety hotline calls, and submissions through NHTSA’s safercar.gov Web site. ODI’s Defects Assessment Division screens all complaints and forwards ones with potential safety significance for additional review. 10 • Identification of potential safety issues. If a potential safety issue is identified, the Defects Assessment Division researches and analyzes available safety data and prepares an investigation proposal for ODI’s investigative division chiefs to review. 11 • Selection of potential safety issues to investigate. ODI’s investigative division chiefs review investigation proposals and recommend to the Director of ODI whether to open an investigation, decline an investigation, or refer the proposal to the Defects Assessment Panel for further review. Most recently in October 2011, we reported on NHTSA’s oversight of vehicle safety. 12 We noted that NHTSA followed its established procedures in investigating unintended acceleration issues for Toyota and other manufacturers; however, process improvements were needed for identifying and addressing vehicle safety defects. We also reported that ODI’s limited information sharing and coordination with foreign countries reduced opportunities to identify safety defects or recalls. NHTSA fully or partially concurred with all 10 of our recommendations. As of May 29, 2013, ODI had taken action to address 9 of our 10 recommendations but had not yet completed a recommended workforce assessment. At the end of April 2015, we received NHTSA’s workforce assessment and closed the remaining recommendation. We are conducting a separate audit to assess NHTSA’s actions to implement our 2011 recommendations and plan to report our findings on this topic later this year. ODI LACKS EFFECTIVE PROCESSES FOR COLLECTING COMPLETE AND ACCURATE VEHICLE SAFETY DATA ODI’s processes for collecting vehicle safety data are insufficient to ensure complete and accurate data. Deficiencies in ODI’s vehicle safety data are due in part to the Agency’s lack of detailed guidance on what information manufacturers and consumers should report. Further, ODI does not verify the completeness and accuracy of manufacturers’ early warning reporting data, or take timely action to 10 The Defect Assessment Division currently has nine staff including eight screeners and a Division Chief. ODI has three investigative divisions: the Vehicle Control Division, Vehicle Integrity Division, and the Medium and Heavy Duty Vehicle Division. 12 Process Improvements Are Needed for Identifying and Addressing Vehicle Safety Defects, (OIG Report Number MH-2012-001), Oct. 6, 2011. 11 6 correct identified inaccuracies and omissions. In the GM case, ODI received data on the ignition switch defect as early as 2003. Some of these data specifically described the ignition switch problems; however, other information lacked sufficient detail or was inconsistently categorized. ODI Lacks Detailed Guidance and Verification Processes To Obtain Complete and Accurate Early Warning Reporting Data The TREAD Act and related regulations require vehicle and equipment manufacturers to report quarterly to NHTSA on a variety of early warning reporting data that could indicate a potential safety defect (see table 2). Table 2. Early Warning Reporting Data Requirements Aggregate data • Aggregate data are quarterly counts of manufacturers’ documentation of potential safety issues. Examples of aggregate data include quarterly counts of dealer field reports, consumer complaints, warranty claims, and property damage claims. • ODI cannot trace aggregate data to a specific vehicle or incident without requesting additional information from a manufacturer. • Manufactures are not required to report incidents that do not involve a component or condition specified in the regulations. Disaggregate data • Disaggregate data are documents describing specific incidents that may be safety-related. Disaggregate data that manufacturers must submit include: o non-dealer field reports that involve components or conditions specified in the regulations and o all death or injury claims and notices, including those that do not involve components or conditions specified in the regulations. • ODI can trace disaggregate data to a specific vehicle or incident. Other • Manufacturers are required to notify NHTSA of technical service bulletins, 13 consumer advisories, and warranty communications • Manufacturers are also required to report foreign recalls of vehicles substantially similar to ones sold in the United States. Source: OIG analysis ODI’s assessment of early warning reporting data is greatly influenced by the codes manufacturers assign to incidents. While regulations specify 24 broad vehicle codes (see exhibit D for a complete list of codes), ODI notes that an average vehicle may have over 15,000 components, and categorizing them can be open to interpretation. For example, ODI staff told us that a manufacturer could categorize a malfunction of an air bag component located in a seat using three 13 Technical service bulletins are documents provided by the manufacturer containing information on safety recalls, defective product components, service campaigns, and customer satisfaction campaigns. 7 different vehicle codes: air bags, seats, or electrical system. Additionally, the regulations allow manufacturers to decide if an incident not included in the 24 defined codes should be reported. However, this does not apply to death and injury claims, all of which must be reported. Despite this complexity, ODI does not provide detailed guidance to help ensure manufacturers interpret and apply the appropriate codes. According to ODI staff, additional rulemaking would be required in order to provide more guidance to manufacturers. ODI analysts told us that when a manufacturer asks for specific guidance on assigning codes, their practice is not to provide guidance and instead allow each manufacturer to make its own decisions. However, ODI investigative chiefs and vehicle safety advocates told us that ODI’s early warning aggregate data are ultimately of little use due to the inconsistencies in manufacturers’ categorizations of safety incidents. According to ODI staff and a January 2008 report issued by the Volpe National Transportation Systems Center (Volpe), 14 non-dealer field reports 15 are the most important source of early warning reporting data because they can provide a specific, technical basis for launching investigations. However, lacking guidance on the reporting format or what information to report, manufacturers submit reports of varying usefulness. For example, one manufacturer’s non-dealer field reports contain a few lines of text briefly describing the consumer’s complaint. Another manufacturer’s reports provide considerably more information, including the technician’s analysis of the condition, root cause analysis, corrective actions taken, and whether the action resolved the condition. ODI staff check that manufacturers submit early warning reporting data on time and may request that manufacturers provide underlying documentation for aggregate data and death and injury data. Additionally, ODI staff told us that they request additional documentation for aggregate data if they identify an anomaly in the data. However, ODI staff noted that their requests for such documentation have declined, from an average of 23 annually between 2006 and 2009 to an average of 4 annually between 2010 and 2014, as a result of their increased workload. Moreover, ODI does not verify that manufacturers’ early warning reporting data are complete and accurate. Although ODI has the authority to inspect manufacturers’ records for compliance with early warning reporting 14 In 2006, ODI initiated an evaluation of its early warning reporting system, with support from Volpe. Non-dealer field reports are communications between consumers, authorized service facilities, and manufacturers regarding the failure, malfunction, lack of durability, or other performance problem related to a vehicle or vehicle part. 15 8 requirements, 16 NHTSA officials told us the Agency has never used this authority. In addition, the Agency has no processes in place for systematically assessing the quality of early warning reporting data or internal guidance on using oversight tools to enforce data reporting requirements. The Agency also has not established best practices for providing early warning reporting data and does not periodically review manufacturers’ early warning reporting procedures. Instead, the Director of ODI told us ODI relies on the “honor system.” However, according to ODI staff, manufacturers routinely miscategorize safety incidents. For example, staff told us that some manufacturers avoid using the word “fire” in non-dealer field reports and instead use phrases such as “strange odor” to avoid categorizing an incident as fire-related. Miscategorizations such as these compromise ODI’s efforts to quickly identify potential safety defect trends. Yet even in cases where ODI suspects noncompliance, it has not taken prompt enforcement action. For example, ODI officials told us they were aware that a vehicle manufacturer was “conservative” in reporting early warning reporting data. According to a November 2014 audit prepared for the manufacturer, two ODI employees called the manufacturer’s officials in late 2011 or early 2012 to ask about inconsistencies between previously reported early warning reporting data and reported death and injury incidents pertaining to an air bag recall. 17 However, ODI took no enforcement action to address this issue until the manufacturer self-reported the omission of about 1,700 death and injury claims in October 2014. NHTSA subsequently required the manufacturer to describe its procedures for complying with early warning reporting requirements and provide the Agency with supporting documentation for all third-party audits of its reporting. In another case, ODI knew a major recreational vehicle manufacturer was noncompliant but did not take action for nearly a decade. In November 2004, ODI discovered that the manufacturer did not report required death and injury data and other early warning reporting data. The lack of reporting continued without ODI action until September 2014, when the office opened an investigation into the manufacturer’s reporting following a suspected recall noncompliance issue. During the investigation, the manufacturer stated that it failed to report the early warning reporting data for over 10 years because of internal miscommunications and a failure in the manufacturer’s software. 16 Title 49 United States Code (U.S.C.) Section 30166 establishes NHTSA’s subpoena power and its authority to inspect manufacturers’ records and require recordkeeping to assess compliance with early warning reporting requirements. 17 The manufacturer officials did not follow up with ODI to provide a full explanation of the inconsistencies. 9 ODI Does Not Provide Sufficient Guidance to Consumers on the Type of Information To Include When Submitting Complaints ODI relies primarily on consumer complaints to identify potential safety concerns. However, consumer complaints often do not provide enough detail to determine the existence of safety concerns or do not correctly identify the vehicle systems involved. The majority of consumer complaints are submitted through NHTSA’s safercar.gov Web site, which prompts consumers to provide details about the vehicles and incidents in question. The online complaint submission form requires consumers to select up to 3 affected parts from a drop-down list of 18 options, such as air bags and electronic stability control (see figure 1). Additionally, the Web site provides a text field for consumers to describe the incidents underlying their complaints. Figure 1. ODI’s Online Submission Form for Consumer Complaints Source: http://www.safercar.gov According to ODI’s initial screener, roughly 50 to 75 percent of complaints incorrectly identify the affected parts, and roughly 25 percent do not provide adequate information to determine the existence of safety concerns. These data quality issues occur in part because ODI does not provide consumers with detailed guidance on submitting complaints. For example, safercar.gov does not define the 18 affected parts categories—some of which may be unfamiliar to consumers, such as “adaptive equipment.” Furthermore, safercar.gov does not allow consumers to submit, or encourage them to retain, supporting documentation (such 10 as photographs or police reports), which ODI’s screeners and management have indicated are valuable in identifying potential safety concerns. In contrast, the U.S. Consumer Product Safety Commission’s complaint Web site (saferproducts.gov) allows consumers to upload as many as 25 documents or photos related to their complaints. 18 ODI Received Early Warning and Consumer Complaint Data Related to the GM Ignition Switch Defect ODI received early warning reporting data and consumer complaints related to the GM ignition switch defect 19 for more than a decade before GM notified ODI of the recall on February 7, 2014. (Exhibit E provides a timeline of select data NHTSA received on the GM ignition switch defect.) However, some of this information lacked sufficient detail or was inconsistently categorized. From 2003 through 2013, GM submitted about 15,600 non-dealer field reports and about 2,000 death and injury reports on vehicles subject to the ignition switch recall— especially related to the 2005 to 2010 Chevrolet Cobalt (see table 3). In a 2011 ODI early warning reporting analysis of 22 vehicles with potential air bag issues, the 2005 to 2010 Chevrolet Cobalt ranked fourth for fatal incidents and second for injury incidents involving air bags. 20 Table 3. Early Warning Reporting Data Related to Vehicles Subject to GM Ignition Switch Defect Non-dealer field reports GM submitted about 15,600 non-dealer field reports. The Cobalt represented 36 percent of these non-dealer field reports. Death and injury claims and notices GM submitted about 2,000 death and The Cobalt represented 63 percent of injury reports. About 90 indicated at the death and injury reports—and 74 percent of these reports indicated least 1 fatality. at least one fatality. Source: OIG analysis GM inconsistently categorized some of the early warning reporting data it submitted to ODI. For example, GM assigned different codes for similar nondealer field reports related to the ignition switch defect. • In March 2005, GM submitted a non-dealer field report in which a GM employee described the ignition switch defect in a 2005 Chevrolet Cobalt. The 18 The U.S. Consumer Product Safety Commission is charged with protecting the public from unreasonable risks of injury or death associated with the thousands of types of consumer products under the Agency’s jurisdiction. 19 According to GM recall documents and the evaluation of how GM handled the ignition switch defect conducted by Anton Valukas, the defect is unintended movement of the ignition switch from the “run” or “on” position to the “accessory” or “off” positions. Additionally, the defect can cause unintended engine stalling while driving and air bag non-deployment during crashes in which the air bags should have deployed. 20 In addition to the Cobalt, ODI analyzed consumer complaints and death and injury data categorized as air bag-related for 21 other passenger vehicles from GM and other manufacturers. 11 employee wrote that the vehicle stalled on a highway when the employee’s knee “hit the GM brown leather key holder.” The employee concluded that minor impact to the ignition key could easily cause the engine to shut off. GM categorized this report using the “Engine and Engine Cooling” code. • In May 2007, GM submitted another non-dealer field report in which a GM employee describes the ignition switch defect in a 2006 Pontiac Solstice. The employee wrote that the vehicle ignition system turned off several times while driving when his knee hit the accessories attached to the key ring. GM categorized this report using the “Electrical” code. In addition, GM’s categorization of a death and injury report pertaining to a fatal accident involving a 2005 Chevrolet Cobalt was inconsistent with supporting documentation. NHTSA regulations state that manufacturers must identify each vehicle system or component that allegedly contributed to the incident when reporting death and injury claims and notices. 21 GM categorized the accident as not involving any of the systems, components, or conditions defined in regulations. However, underlying documentation for the report included a Wisconsin State trooper’s report indicating that the ignition switch and air bags were both involved in the accident: The ignition switch on the…vehicle appears to have been in the accessory position when it impacted the trees preventing the air bags from deploying. A search of the [NHTSA] web site indicates five complaints of 2005 Chevrolet Cobalt ignition switches turning off while the vehicle was being driven. Three of the complaints talk about the knee or leg touching the ignition or key chain causing the engine to turn off...It appears likely that the vehicles’ key turned to accessory as a result of the low key cylinder torque/effort. 22 In February 2007, a GM technical service bulletin uploaded to Artemis 23—ODI’s primary database for storing data used to identify and address potential safety defects—described inadvertent turning of the key cylinder and loss of electrical systems. The bulletin applied to vehicle models and model years that would eventually be subject to the February 2014 recall. Although the bulletin does not describe the potential for the vehicle to stall as a result of inadvertent turning of the ignition switch, it does state that the problem was more likely to occur when the vehicle was turning. GM categorized this bulletin using the “Steering” code. From January 1, 2003, through February 7, 2014, ODI received 9,266 complaints involving the vehicles subject to the GM ignition switch recall—including 21 49 C.F.R. §§ 579.21(b)(1)-(2). Collision Analysis and Reconstruction Report, prepared by the Technical Reconstruction Unit of the Wisconsin State Patrol Academy, Feb. 14, 2007. 23 The Advanced Retrieval of Tire, Equipment, and Motor Vehicle Information System. 22 12 72 complaints indicating at least 1 injury and 3 complaints indicating at least 1 fatality. The majority of these complaints involved the 2005 to 2010 Chevrolet Cobalt and the 2003 to 2007 Saturn Ion. Some consumer complaints were miscategorized or lacked sufficient detail to link them to the ignition switch defect. For example, a June 2005 complaint stated only that an accident had destroyed a 2005 Chevrolet Cobalt and injured one person and that the air bags did not deploy. The complaint did not specify whether this accident occurred on or off the road, or whether the impact was to the front, side, or back of the vehicle—details that were essential to ODI’s analysis of air bag non-deployment in these vehicles. However, some consumer complaints described the ignition switch defect in detail. For example, in June 2005, a consumer sent NHTSA a copy of a letter that she sent to the GM customer service department describing how her 2005 Chevrolet Cobalt had turned off on three occasions while driving. The letter stated that the service manager tested the vehicle and was able to turn the ignition switch when his knee hit the bottom of the “opener gadget” on the keychain. The letter goes on: This is a safety/recall issue if there ever was one. Forget the bulletin. I have found the cause of the problem. Not suggested causes as listed in bulletin. The problem is the ignition turn switch is poorly installed. Even with the slightest touch, the car will shut off while in motion. I don’t have to list to you the safety problems that may happen, besides an accident or death… 24 Furthermore, ODI contractors miscategorized some consumer complaints related to ignition switch defects. For example, in September 2003, a driver of a 2003 Saturn Ion reported experiencing engine shutoff on three occasions when the driver’s knee accidently hit the car keys. According to the complaint, two of these events occurred when the car was traveling at 65 miles per hour on a freeway. When entering this complaint into Artemis, ODI contractors miscategorized this complaint using the codes “Unknown or Other” and “Exterior Lighting: Headlights: Switch” rather than the correct code “Electrical Systems: Ignition: Switch.” WEAK DATA ANALYSES AND REVIEWS UNDERMINE ODI’S EFFORTS TO IDENTIFY VEHICLE DEFECTS ODI does not follow standard statistical practices when analyzing early warning reporting data, conduct thorough reviews of consumer complaints, or provide 24 Consumer letter to General Motors Corp. regarding a 2005 Chevrolet Cobalt, dated June 29, 2005 (attachment to ODI Vehicle Owner Questionnaire Number 10129121). 13 adequate supervision or training for staff responsible for reviewing these data and complaints. As a result, it cannot reliably identify the most statistically significant safety issues to pursue. ODI’s complaints process is not thorough and in the case of GM, ODI missed multiple opportunities to link the GM ignition switch defect to air bag non-deployments because ODI staff lacked technical expertise and did not consider all available information. ODI Does Not Follow Standard Statistical Practice When Analyzing Early Warning Reporting Data ODI uses four statistical tests to analyze aggregate early warning reporting data (such as consumer complaints, warranty claims, and property damage claims)—as well as a fifth test to analyze non-dealer field reports (see table 4). Table 4. ODI’s Statistical Tests for Analyzing Early Warning Reporting Data Statistical test Description Crow-AMSAA Trend analysis used to analyze aggregate data Mahalanobis distance Test used to analyze aggregate data Probability measure Test used to analyze aggregate data Logistic regression Regression test used to analyze death and injury aggregate data CRM-114 Filter used to analyze non-dealer field reports Source: OIG analysis While the statistical experts we consulted 25 note that conducting multiple tests provides a sound basis for analysis, ODI does not follow standard statistical practices when implementing the tests of the aggregate data. Specifically, ODI does not consistently identify a model (a set of assumptions) for the aggregate data to establish a base case—that is, what the test results would be in the absence of safety defects. According to the statistical experts, identifying assumptions and models—and checking to see whether they fit the data—are essential for establishing a base case. Without a base case, ODI cannot differentiate trends and outliers that represent random variation from those that are statistically significant—that is, scores that indicate a safety issue should be pursued. ODI has missed opportunities to update and improve its statistical methods for analyzing early warning reporting data. For example: • ODI does not regularly assess the performance of its aggregate data tests. According to the statistical experts, out-of-sample testing—a standard 25 The statistical experts we consulted with are from academia and research institutes. See exhibit B for a complete list of the experts and their affiliations. 14 statistical assessment practice—would allow ODI to determine whether potential safety issues identified in one portion of its aggregate data turn up in the remaining portion. However, ODI performed out-of-sample testing on only one aggregate data test and only when the test was first implemented. ODI also conducted out-of-sample tests on non-dealer field reports, but it has not done so since 2009. • Despite recent developments in data analytics, ODI has not updated its statistical tests from initial implementation in 2006 through 2009, so it has not taken advantage of recent methodological advances. Although ODI has periodically recalibrated some of its tests (such as logistic regression) using current data, it has not updated the analytical methodologies it uses. • Volpe conducted the only external review of ODI’s aggregate data tests since their implementation. According to its January 2008 report, Volpe reported that the review’s scope was limited because of concerns about the informational burden on ODI and manufacturers. As a result, Volpe was unable to reach any conclusions about the tests’ effectiveness. ODI has not requested any other external reviews of its statistical tests. ODI similarly lacks procedures to promote timely screening of early warning reporting data. For example, ODI’s Early Warning Division staff review nondealer field reports based on the results they receive from a statistical test; however, there is no process for ensuring that all non-dealer field reports are included in the universe from which the sample is drawn. ODI has overlooked non-dealer field reports for months or even years if, for example, manufacturers submit the reports in formats that ODI’s statistical test cannot process. In addition, advanced screeners, who are responsible for proposing safety defect investigations, told us that they are less likely to rely on early warning reporting data because of the data’s lack of timeliness. The information in early warning reporting data can be delayed by months because manufacturers submit the reports quarterly. ODI Does Not Thoroughly Screen Consumer Complaints In October 2010, ODI established a two-tiered process for screening consumer complaints, its primary source for identifying potential vehicle safety concerns. Currently, one employee reviews all submitted consumer complaints, determines which complaints have potential safety implications, and forwards those complaints to eight advanced screeners who perform more in-depth reviews (see figure 2). In 2011, we recommended that ODI conduct a workforce assessment to determine the number of staff required for ODI to meet its objectives and determine the most effective mix of skill sets. ODI has recently completed its 15 workforce assessment. We are conducting a separate audit to assess NHTSA’s actions to implement our 2011 recommendations—including the workforce assessment—and plan to report our findings on this topic later this year. Figure 2. ODI’s Consumer Complaint Review Process Source: OIG analysis Since 2010, ODI has received at least 40,000 complaints a year. In 2014, it received nearly 78,000 complaints (see figure 3). In other words, the initial screener’s workload is roughly 330 complaints each day. Determinations of whether complaints warrant further review are made within a matter of seconds— in part because the initial screener spends roughly half of the day carrying out other work responsibilities. 80,000 350 329 60,000 279 250 209 40,000 300 200 214 178 150 100 20,000 50 0 0 2010 2011 2012 Annual Complaint Volume 2013 2014 Average Daily Complaint Volume Note: Calculation of average daily complaint volume assumes 236 working days per year. Source: OIG analysis of complaint data in Artemis Average Number of Daily Complaints Number of Annual Complaints Figure 3. ODI’s Annual and Average Daily Complaint Volume 16 According to the initial screener and our independent verification, about 10 percent of complaints are forwarded to advanced screeners for in-depth reviews, 26 leaving no assurance that the remaining 90 percent of complaints receive additional review. In making determinations, the initial screener relies on his professional experience and judgment, as well as informal guidance and precedent. While he noted that ODI informally established certain complaint categories that automatically warrant further analysis—including most air bag non-deployments and seatbelt issues— ODI lacks formal guidance for initial screening. The initial screener further noted that he prioritizes incidents that occur suddenly, with little warning for the consumer, but assigns lower priority to engine, transmission, and vehicle body issues and generally does not forward certain incidents that most likely do not lead to investigations, such as sharp door edges. The initial screener also does not forward complaints he believes are covered by existing recalls. ODI’s process for initially screening consumer complaints leaves the office vulnerable to a single point of failure and the risk that complaints with potential safety significance may not be selected for further review. Like the initial screener, ODI’s eight advanced screeners have access to a variety of data sources—such as technical service bulletins and special crash investigation reports—and have the authority to reach out to consumers and perform field inspections to augment their research. However, three advanced screeners told us that they rely mainly on consumer complaints to identify safety concerns, and four advanced screeners said they only occasionally use other sources of data. While screeners are encouraged to query all complaints for issues in their areas of concentration, four screeners told us they do not consistently do this—in some cases because it takes too much time. Advanced screeners also have access to early warning reporting data; however, four advanced screeners told us that they are less likely to rely on these data because they are untimely. Two screeners were also concerned about the early warning reporting data’s lack of usefulness because they felt the data provided no significant additional detail. In 2013, ODI began requiring advanced screeners to annotate the complaints they review by documenting the condition that led to the incident and their reasons for deciding not to pursue potential issues. According to the Defects Assessment Division Chief, the annotations are intended to identify and correct inconsistencies and inaccuracies in complaints—and thereby enable ODI to properly link them to relevant safety concerns—and provide a record of review. However, an ODI internal audit found that roughly half the complaints were incorrectly annotated or 26 We independently verified that, in 1 week of review, the initial screener forwarded about 10 percent of complaints to advanced screeners. 17 lacked critical information. Additionally, we analyzed annotations for complaints received in the fourth quarter of 2013 and found that about 57 percent of the complaints that screeners determined did not warrant further review lacked justifications. Advanced screeners told us that annotating complaints is time consuming. ODI’s Pre-Investigation Staff Lack the Training and Supervision To Effectively Analyze Vehicle Safety Data While NHTSA has a training plan, it has not been implemented effectively. As a result, ODI staff who review early warning reporting data and consumer complaints lack adequate training to carry out their responsibilities. For example: • ODI staff charged with interpreting statistical test results for early warning reporting data told us they have no training or background in statistics. Three screeners assigned to analyze air bag incidents lacked training in air bags. One screener who was originally hired to review child seat restraint issues was assigned in 2008 to review air bag issues—without any air bag training and without an engineering or automotive background. • Screeners told us that training to maintain professional certifications—such as the Automotive Service Excellence certification for automotive mechanics— must be completed on their own time and generally at their own expense. • Screeners also noted that ODI lacked the funding to allow them to attend training to stay abreast of the latest developments in vehicle technology. Further, ODI has not established an adequate supervisory review process to evaluate the quality of screeners’ work in identifying potential safety issues. Except for reviews of final investigation proposals, we found no documentation of supervisory review. In addition, ODI staff told us that their data analysis and screening efforts were generally not reviewed and that they received little feedback on the quality of their work. For example, the Defects Assessment Division Chief characterized his oversight of the initial complaint screener’s work as “minimal” and acknowledged that he does not provide much guidance to the initial screener. Instead, ODI relies on the screener’s years of experience and professional judgment to identify complaints warranting further analysis. Advanced screeners also told us that supervisory review is often informal and that the Defects Assessment Division Chief does not regularly review their complaint annotations. Inadequate training and supervisory review have led to deficient analyses of early warning reporting and complaint data. For example, the developer of one 18 statistical test that ODI uses to analyze early warning reporting data stated that the test should produce the same results every time for the same data input in the same order. However, ODI staff told us that different test runs produce different results, and management has not considered this to be a problem. ODI Staff Overlooked Documentation Pointing to the GM Ignition Switch Defect NHTSA staff and contractors reviewed non-dealer field reports that described the GM ignition switch defect, and reviewed death and injury and special crash investigation reports that explicitly linked the ignition switch defect and air bag non-deployments. However, ODI staff missed opportunities to connect the ignition switch defect to air bag non-deployments because they did not consider all available information. For example, in 2007, two ODI employees reviewed the underlying documentation for a death and injury report on a fatal accident involving a 2005 Chevrolet Cobalt, which contained evidence that linked the ignition switch defect to the vehicle’s air bag non-deployment. However, neither employee—an early warning reporting analyst and an ODI air bag investigator—made this connection during their analyses of the documentation. The death and injury report documentation specifically included: • A Wisconsin State Trooper’s report that identified the ignition switch defect as a possible cause of air bag non-deployment during the accident. However, the two ODI staff who reviewed the report did not note this finding when documenting their reviews of the report. • Event data recorder data 27 showed the vehicle’s power mode status had been in the “accessory” position during the accident—a key indicator of the ignition switch defect. However, the ODI analyst reviewing this report did not include this information in his annotation. The air bag investigator noted this information in his review but ultimately concluded that the air bag nondeployment was caused by the long delay between the first and final impacts. A NHTSA special crash investigation report on the same fatal accident also suggested a link between the ignition switch defect and air bag non-deployments. Specifically, the report concluded that the vehicle’s air bags failed to deploy possibly due to “power loss due to movement of the ignition switch just prior to the impact,” among other potential reasons. NHTSA’s special crash investigation 27 An event data recorder is a device installed in a vehicle to record technical vehicle and occupant information for a brief period of time (seconds, not minutes) before, during, and after a crash. 19 staff told us that they submitted their report to ODI for review in April 2007. However, ODI told us its staff did not review the report. 28 Between the second quarter of 2012 and the fourth quarter of 2013, ODI received 13 non-dealer field reports on the 2005 to 2010 Chevrolet Cobalts that GM categorized as air bag-related and that we determined may be related to the ignition switch defect. 29 However, ODI staff reviewed only one of these nondealer field reports before the February 2014 recall. According to ODI staff, they did not review the majority of these reports because in the second quarter of 2012, GM began using a new file format for most of their document submissions (.docx), which could not be read by the statistical test ODI uses to analyze these reports. 30 ODI staff acknowledged that they did not notice the reports were not analyzed until after the recall. In addition to the non-dealer reports, ODI received 9,266 consumer complaints between January 1, 2003, and February 7, 2014, that involved GM vehicles subject to the ignition switch recall. Because ODI’s screeners were not required to annotate their reviews of complaints until 2013, ODI cannot establish a full picture of why it did not investigate complaints related to the GM ignition switch and air bag non-deployment issues prior to 2013. From the time that the annotations were required to the date of the recall, ODI received 926 consumer complaints involving the recalled vehicles. ODI’s initial screener advanced 27—or 3 percent—of these complaints for further review, compared to the average of 10 percent that are typically forwarded. ODI’s advanced screeners noted in their annotations that 11 of the 27 complaints included allegations of front air bag nondeployment, but they did not advance these complaints for further consideration because they concluded there was either “no actionable trend indicated” or “minimal hazard.” ODI staff did not thoroughly understand when air bags were supposed to deploy in these vehicles, which prevented them from linking the ignition switch defect to the air bag non-deployment. This may be explained by ODI staff’s acknowledged lack of training on air bags. ODI prepared three investigation proposals for the Chevrolet Cobalt and Saturn Ion about loss of electric power steering and air bag non-deployment. Each proposal was supported by early warning reporting referrals identifying these 28 Artemis records for the GM air bag non-deployment issue contain a preliminary version of the special crash investigation report (IN-06-033) completed in December 2006. According to the preliminary report, evidence showed that the ignition switch was in the “accessory” position at the time of the crash and that the contractor was “continuing its investigation into this aspect of the crash.” 29 To determine which non-dealer field reports were related to the ignition switch recall, we limited this analysis to vehicle models, model years, facts, and circumstances that would make an incident eligible for compensation through the GM ignition switch compensation fund. 30 ODI’s written instructions to vehicle manufacturers for submitting early warning reporting documents specify seven acceptable electronic file formats (including .doc and .html), but .docx is not one of those specified. Docx is the Microsoft Office extensible markup language file format. 20 potential safety concerns. However, ODI staff did not establish the ignition switch defect as a potential root cause for these issues. For example, in September 2007, an ODI screener submitted an investigation proposal on front air bag nondeployment in the 2005 and 2006 Chevrolet Cobalt and the 2003 through 2005 Saturn Ion. The proposal attributed 4 fatalities, 11 injuries, and 29 crashes to the potential safety defect, and it explained that “driver and passenger side frontal air bags fail to deploy during crash events where [data] suggest the air bags should have deployed.” However, the proposal did not link the air bag non-deployment to the ignition switch defect, even though proposal documentation included an interview with a vehicle owner who mentioned the special crash investigation report that identified the position of the ignition switch as a possible cause of the air bag non-deployments. ODI officials told us that they did not understand the safety consequences of the ignition switch defect before the GM recall. ODI INITIATES INVESTIGATIONS WITHOUT ASSURANCE THE MOST SIGNIFICANT SAFETY DEFECTS ARE TARGETED ODI lacks the procedures needed to effectively identify safety defects that warrant an investigation. Specifically, ODI has not developed guidance for applying the factors it established for opening an investigation. In addition, the factors that influence ODI’s decisions on whether to open an investigation are not transparent, and it is unclear who is accountable for these decisions. This was the case with ODI’s decision not to investigate the GM air bag non-deployment defect. ODI Lacks Consensus and Detailed Guidance on the Amount and Type of Information Needed To Open Investigations According to ODI’s Defects Assessment Division Chief, ODI considers three factors when proposing a vehicle safety defect investigation: (1) rate of consumer complaints, 31 (2) severity of the potential safety issue, and (3) identification of a potentially defective vehicle component or root cause. However, ODI has not developed specific guidance on how screeners should apply these factors, and there is a lack of consensus among ODI leadership on the factors necessary to open an investigation—leaving screeners uncertain about how much support is needed to propose an investigation. Attorneys in NHTSA’s Office of Chief Counsel state that while NHTSA must establish severity for all cases, it can establish either frequency or root cause to force a manufacturer to initiate a recall. However, according to ODI’s Defects Assessment Division Chief, all three factors should be met before proposing an investigation. Specifically, the Defects Assessment Division Chief expects 31 The rate of complaints is the number of relevant complaints received by NHTSA divided by the number of vehicles in production. 21 advanced screeners to find the root cause in order to build a compelling proposal for an investigation, but the Director of ODI does not think the root cause is necessary and prefers screeners to focus on establishing the safety consequences of a potential defect. ODI’s two investigative chiefs agree that establishing a pattern of safety concerns is more important than identifying root cause. The Director of ODI can also unilaterally decide not to open an investigation after discussion with Defects Assessment Panel participants. For example, the Director of ODI decided not to pursue the following investigative proposals after concluding that they presented minimal hazards: • In June 2014, an advanced screener proposed an investigation of 2007 to 2011 vehicles that suddenly lost steering power assist. The proposal established the rate of complaints (over 1,000 as of November 2014), severity of the issue (increased crash risk), and the defective component (steering pump and/or steering gear). The vehicle manufacturer had issued an extended warranty to cover the defect, but ODI did not verify whether the warranty adequately addressed the issue. • In July 2014, an advanced screener proposed an investigation of 2012 model vehicles that experienced intermittent loss of electrical power. In the proposal phase, ODI established the rate of complaints (over 46 complaints), severity of the issue, and the defective component. Without specific guidance on the amount and type of information needed to launch an investigation, screeners largely rely on precedent and professional judgment to determine which issues merit investigation. One screener told us he uses his “gut feeling” when reviewing complaints to gauge the “appetite” of the office for specific issues. Another screener told us he only proposes investigations that have the greatest chance of being selected to avoid the extra work of proposing investigations that are ultimately denied. Three screeners said they are hesitant to propose investigations if similar proposals have been rejected in the past. For example: • In October 2012, a screener proposed an investigation of a 2002 and 2003 model vehicle for subframe rust. The severe corrosion allegedly caused the wheels to collapse, potentially resulting in loss of driver control. According to the initial screener, ODI officials were opposed to investigating these vehicles because of their age. As a result, the initial screener told us that he now hesitates to advance similar issues in older vehicles because of the skepticism this proposal met. 22 • A screener told us he did not propose an investigation of a safety defect that caused a vehicle’s hood to fly open while driving because previous proposals on hood latch issues did not lead to investigations. In general, ODI officials prefer to open investigations that are most likely to result in a manufacturer recall—an assertion echoed by four of the eight screeners we spoke with. In 2011 and 2012—the most recent years for which ODI has actionable data—about 70 percent of the investigations eventually resulted in recalls. According to an ODI investigative division chief, repeatedly opening investigations that do not result in a recall could cause ODI to lose credibility with manufacturers. However, ODI’s focus on issues most likely to result in recalls creates the potential for missed opportunities to investigate issues that have serious safety implications. For example: • In June 2013, an advanced screener proposed an investigation of headlamp outages on 2003 to 2005 vehicles. Although the proposal cited about 400 complaints, ODI declined to open an investigation due in part to concerns that it would not lead to a recall. • In October 2014, ODI decided not to open an investigation of a vehicle’s faulty brake lights due to an investigative division chief’s concerns that the issue may be covered by an existing recall—even though an advanced screener had noted prior to the meeting that they did not believe this issue was covered by a recall. Targeting potential safety defects that most likely lead to recalls blurs the line between pre-investigative and investigative duties. According to the Defects Assessment Division Chief, ODI relies heavily on the pre-investigative phase because of the resources needed to conduct investigations. However, it is unclear whether screeners have access to the data needed to prompt an investigation, such as manufacturer data. While NHTSA’s Office of Chief Counsel stated that ODI may compel information from manufacturers during the pre-investigative stage, the Defects Assessment Division Chief told us they generally do not compel this information without first launching an investigation. Regardless, three screeners were unaware that their division has the authority to compel information from manufacturers without launching an investigation. Additionally, considerable investigative duties—such as research and engineering analysis work—are being performed in the pre-investigative phase, often by screeners who are not adequately trained and may not have access to complete information. For example: • One screener told us he attempted to identify the cause of a potential safety defect by freezing a brake component to assess the impact of ice buildup on the component. 23 • Another screener told us he could not detect any exhaust odor in a vehicle when subsequent work by investigative staff found that the carbon monoxide level reached Consumer Product Safety Commission thresholds for noticeable headache, fatigue, and nausea, and exceeded Occupational Safety and Health Administration standards if exposure exceeded 8 hours. 32 • Three advanced screeners assigned to analyze air bag incidents lacked training in air bags. ODI relies on screeners, who are not provided adequate training, to conduct technical research and testing before opening investigations. This may result in potential safety defects being overlooked. In addition, these added duties take time away from the advanced screeners’ primary duty of screening safety data, resulting in backlogs of those data. ODI’s Investigation Decision Process Lacks Transparency and Accountability In ODI’s investigation decision process, the Defects Assessment Chief provides a list of proposals to ODI’s investigative division chiefs—along with supporting documentation, such as consumer complaints and warranty claims. The division chiefs review the proposals and decide whether to open an investigation, decline to investigate, or send the proposal to ODI’s Defects Assessment Panel for further review. 33 According to ODI’s written policy, division chiefs have 2 weeks to complete their review. However, the investigative division chiefs consider the 2-week requirement to be a suggested timeframe that should be balanced against other competing priorities. Untimely proceedings by the Defects Assessment Panel have delayed investigation decisions. During a 9-month period we reviewed, the panel conducted five meetings. The panel often reschedules meetings and according to some screeners, the meetings tend to be pro forma. For example, one screener stated the meetings focus on the reasons for not opening an investigation rather than reasons for opening one; another called the meetings “dog and pony shows.” The panel also repeatedly delays decisions on proposals to obtain additional information. For example: • In August 2014, the panel reviewed a proposal to investigate a side air bag non-deployment that resulted in a fatality. At that meeting, the Director of ODI, who sits on the panel, requested additional information. By October, the manufacturer had responded to ODI’s questions, but an investigative division 32 The investigation proposal for this potential safety defect is currently pending further review. The Defects Assessment Panel is a body chaired by the Director of ODI that is intended to meet monthly to review investigation proposals and decide whether to open an investigation. 33 24 chief requested that an investigation not be opened until his team had completed an on-site inspection of the vehicle involved in the accident. As of the most recent panel meeting in February 2015—5 months after the panel first reviewed the potential defect—a decision to investigate this issue remains pending. • In January 2014, the panel discussed a proposal on a vehicle’s steering failure. However, the panel has delayed the decision whether to investigate this issue for over a year—despite a recommendation from the investigative division to open an investigation. In addition to delays, ODI’s decisions are not transparent. Of the 56 investigation proposals for light vehicle safety defects in 2013, 32 were not investigated—18 of which lacked documented justifications for not investigating. While the panel may provide a reason for declining an investigation, such as “minimal hazard,” it does not document the evidence that supports its decision. In addition, a proposal may be rejected by investigation divisions, which do not always document reasons for declining to investigate. Lack of transparency exacerbates the problems created by reliance on precedent because screeners do not learn what management deems worthy of investigation. Transparency and accountability are especially critical since ODI generally does not revisit proposals once they are declined for investigation. Screeners told us that there is a need for ever increasing numbers of incidents to consider reopening previously rejected investigative proposals. While ODI lists declined proposals in Artemis as being “monitored,” it does not track who monitors these issues. Four of the eight advanced screeners noted that they consider monitored proposals to be essentially denied and rarely resubmit proposals unless there is a new angle or “smoking gun.” One screener said resubmitting a proposal is like “beating a dead horse.” ODI Did Not Investigate or Adequately Monitor the GM Air Bag Non-Deployment or Ignition Switch Issues At a November 2007 Defects Assessment Panel meeting, ODI management and staff discussed a proposal to investigate frontal air bag non-deployments related to the Chevrolet Cobalt and Saturn Ion. ODI ultimately declined the proposal but did not document the justification for doing so. According to ODI staff, the decision not to investigate was based on a flawed understanding of air bag technology. Specifically, the Defects Assessment Panel believed the air bags did not deploy because the drivers were not wearing their seatbelts and because the vehicles left 25 the road during the accidents. 34 At the same panel meeting, an ODI air bag investigator advocated against opening an investigation because he had concluded, based on his analysis of complaints, that the rate of air bag non-deployment complaints for the Cobalt and Ion was similar to that of peer vehicles. According to ODI staff who attended the 2007 panel meeting, the Defects Assessment Panel had requested that the potential safety defect be monitored to identify future air bag non-deployments occurring on the road, where air bag deployment would be expected. In addition, NHTSA’s Associate Administrator for Enforcement, who did not attend the panel discussion, told the Director of ODI and the Defects Assessment Division Chief that “given the reports of fatal crashes, this [investigation proposal] looks like one we want to jump on and learn as much as we can quickly.” The ODI screener who prepared the investigation proposal was initially assigned to monitor the issue. However, the Defects Assessment Division Chief did not reassign that responsibility after the screener responsible for monitoring the issue left NHTSA in 2008. ODI missed other opportunities to investigate the air bag non-deployment issue. For example, in April 2009, the Defects Assessment Division Chief requested a special crash investigation of a collision involving air bag non-deployment in a 2005 Chevrolet Cobalt. However, ODI did not follow up on the investigation’s results, and the Defects Assessment Division Chief had no explanation for why ODI did not pursue the issue. Two ODI staff members reviewed the findings of the special crash investigation in February 2010, but neither reported the results of their reviews. The first, an investigator, told us he did not report the results because he was not responsible for screening safety issues. The second, an advanced screener, told us that while he does not recall reviewing the report, he would only have noted issues in his area of concentration: engine, power train, and speed control. According to ODI officials, in 2010, an ODI screener suggested revisiting the 2007 investigation proposal on air bag non-deployments in the Chevrolet Cobalt because of new consumer complaints. However, after the air bag investigator updated his analysis of consumer complaints and identified a downward rate of complaints for the vehicles, the screener decided that the issue did not present enough of a safety trend to warrant renewing the investigation proposal. While ODI identified air bag non-deployments as a potential safety issue, it did not identify or propose an investigation of the GM ignition switch issue. 34 According to GM, frontal air bag deployment takes into consideration factors such as speed of the vehicle, severity and location of the impact, and rate of deceleration. Air bags are programmed not to deploy in non-accident circumstances, such as driving over potholes or rough terrain. 26 According to ODI staff, there were no discussions of the ignition switch defect prior to the February 2014 recall. CONCLUSION NHTSA administers highway safety and consumer programs intended to save lives, prevent injuries, and reduce economic losses resulting from motor vehicle crashes. NHTSA’s ODI is charged with requiring manufacturers to recall vehicles with safety-related defects. However, weaknesses in ODI’s training and supervision of pre-investigation staff and its processes for identifying potential safety concerns and initiating investigations, as evidenced by NHTSA’s handling of the GM ignition switch defect, deter NHTSA from successfully meeting its mandate to help prevent crashes and their attendant costs, both human and financial. RECOMMENDATIONS To improve ODI’s collection of vehicle safety data, we recommend the National Highway Traffic Safety Administrator take the following actions: 1. Develop and implement a method for assessing and improving the quality of early warning reporting data. 2. Issue guidance or best practices on the format and information that should be included in non-dealer field reports to improve consistency and usefulness. 3. Require manufacturers to develop and adhere to procedures for complying with early warning reporting requirements; and require ODI to review these procedures periodically. 4. Expand current data verification processes to assess manufacturers’ compliance with regulations to submit complete and accurate early warning reporting data. At minimum, this process should assess how manufacturers assign vehicle codes to specific incidents and how they determine which incidents are reportable. 5. Develop and implement internal guidance that identifies when and how to use oversight tools to enforce manufacturers’ compliance with early warning reporting data requirements. 6. Provide detailed and specific guidance to consumers on the information they should include in their complaints, as well as the records they should retain (such as police reports and photographs) in the event that ODI contacts them for more information. 27 To improve ODI’s processes for screening and analyzing vehicle safety data, we recommend the National Highway Traffic Safety Administrator take the following actions: 7. Develop an approach that will determine which early warning reporting test scores provide statistically significant indications of potential safety defects. 8. Periodically assess the performance of the early warning reporting data tests using out-of-sample testing. 9. Institute periodic external expert reviews of the statistical tests used to analyze early warning reporting data to ensure that these methods are up-to-date and in keeping with best practices. 10. Implement a supervisory review process to ensure that all early warning reporting data are analyzed according to ODI policies and procedures. 11. Develop and implement a quality control process to help ensure complaints are reviewed thoroughly and within a specified timeframe. 12. Update standardized procedures for identifying, researching, and documenting safety defect trends that consider additional sources of information beyond consumer complaints, such as special crash investigation reports and early warning data. 13. Document supervisory review throughout the pre-investigative process including data screening. 14. Evaluate the training needed by pre-investigative staff to identify safety defect trends; and develop and implement a plan for meeting identified needs. To promote a streamlined process for opening investigations of potential safety concerns, we recommend the National Highway Traffic Safety Administrator take the following actions: 15. Develop and implement guidance on the amount and type of information needed to determine whether a potential safety defect warrants an investigation proposal and investigation. 16. Develop a process for prioritizing, assigning responsibility, and establishing periodic reviews of potential safety defects that ODI determines should be monitored. 28 17. Document and establish procedures for enforcing timeframes for deciding whether to open investigations; and establish a process for documenting justifications for these decisions. AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL RESPONSE We provided NHTSA a copy of our report on April 30, 2015, and received its response—included in full in the appendix—on June 16, 2015. NHTSA concurred with our 17 recommendations, agreed to implement them as written, and provided appropriate target completion dates. Accordingly, we consider all recommendations resolved but open pending completion of the planned actions. We appreciate the courtesies and cooperation of National Highway Traffic Safety Administration representatives during this audit. If you have any questions concerning this report, please call me at (202) 366-1959 or Mitchell Behm, Assistant Inspector General for Surface Transportation Audits, at (202) 366-1995. # cc: DOT Audit Liaison, M-1 NHTSA Audit Liaison, NPO-330 29 EXHIBIT A. SCOPE AND METHODOLOGY We conducted our work from May 2014 through April 2015 in accordance with generally accepted Government auditing standards. Those standards require that we plan and perform an audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence we obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Secretary of Transportation requested that we assess NHTSA’s procedures for collecting, analyzing, and managing vehicle safety data and determine whether information on ignition switch issues or non-deploying air bags was available to NHTSA but not used in the GM defect analysis. Our work examined information available to NHTSA prior to GM’s 2014 recall announcement and did not assess whether GM fully disclosed information on the ignition switch issue to NHTSA. Specifically, we assessed ODI’s procedures for (1) collecting vehicle safety data, (2) analyzing the data and identifying potential safety issues, and (3) determining which issues warrant further investigation. We also assessed how those procedures affected ODI’s handling of issues related to the GM ignition switch recalls. To assess ODI’s procedures to collect and analyze safety-related vehicle defect data and determine which issues warrant further investigation, we reviewed relevant legislations and regulations, and ODI’s early warning reporting and defect assessment procedures. To assess ODI’s analyses of early warning reporting data, we reviewed the documents ODI provided and interviewed statistical experts, ODI staff responsible for conducting the statistical analyses, and ODI staff responsible for using the statistical test results to refer potential safety issues. We reviewed data in Artemis (ODI’s primary data repository), annotations for consumer complaints received during the last quarter of 2013, investigation proposals forwarded for review in 2013, special crash investigation reports, and documentation NHTSA provided for a congressional investigation of the GM recall. Our review was focused primarily on ODI’s treatment of passenger cars and trucks and not on other vehicles, such as commercial trucks, busses, and motorcycles. We interviewed the Director of ODI, Defects Assessment Division Chief, three investigative division chiefs, and a former Early Warning Division Chief. We interviewed all four Early Warning Division screeners and eight Defects Assessment Division screeners. We also observed the initial screener’s screening of incoming consumer complaints. We interviewed special crash investigation personnel, Volpe staff, and external stakeholders—including Joan Claybrook, former NHTSA Administrator, and Clarence Ditlow, Executive Director of the Exhibit A. Scope and Methodology 30 Center for Auto Safety, for their input on the pre-investigative process. Finally, we observed Defects Assessment Panels held from August 2014 through February 2015 to witness discussions of investigation proposals. Exhibit A. Scope and Methodology 31 EXHIBIT B. ORGANIZATIONS AND EXPERTS CONTACTED ODI’s Contractors: • BLF Technologies Inc. • Volpe National Transportation Systems Center Safety Advocates: • Clarence Ditlow, Executive Director, the Center for Auto Safety • Joan Claybrook, President Emeritus, Public Citizen Experts in Statistical Analysis: • Steve MacEachern, Ph.D., Professor, Ohio State University, Department of Statistics • Peter Craigmile, Ph.D., Associate Professor, Ohio State University, Department of Statistics • Lisa Goldberg, Ph.D., Adjunct Professor and Director of Research at the Center for Risk Management Research, University of California, Berkeley, Department of Statistics • James Scott, Ph.D., Assistant Professor, University of Texas, Department of Statistics • Bill Yerazunis, Ph.D., Mitsubishi Electric Research Laboratories (developer of CRM114, the algorithm underlying ODI’s filter used to analyze non-dealer field reports) Other Organizations: • U.S. Consumer Product Safety Commission Exhibit B. Organization and Experts Contacted 32 EXHIBIT C. FLOWCHART OF ODI’S PRE-INVESTIGATIVE PROCESS Source: OIG’s analysis of ODI’s pre-investigative processes Exhibit C. Flowchart of ODI’s Pre-Investigative Process 33 EXHIBIT D. EARLY WARNING REPORTING CODES NHTSA regulations specify 33 codes that vehicle and equipment manufacturers must assign to early warning reporting data based on the vehicle systems, components, and conditions alleged to have contributed to the incident. Codes for vehicles Codes for child restraints 1 Steering-01 25 Buckle-51 2 Suspension-02 26 SeatShell-52 3 ServiceBrake-03 27 Handle-53 4 ServiceBrakeAir-04 28 Base-54 5 ParkingBrake-05 6 EngAndEngCooling-06 7 FuelSys-07 8 FuelSysDiesel-08 Codes for tires 9 FuelSysOther-09 29 Tread-71 10 PowerTrain-10 30 SideWall-72 11 Electrical-11 31 Bead-73 12 ExtLighting-12 13 Visibility-13 14 Air bags-14 15 SeatBelts-15 Codes for death and injury reports 16 Structure-16 32 Other-98 17 Latch-17 33 None-99 18 SpeedControl-18 19 TiresRelated-19 20 Wheels-20 21 TrailerHitch-21 22 Seats-22 23 FireRelated-23 24 Rollover-24 Source: ODI Exhibit D. Early Warning Reporting Codes 34 EXHIBIT E. TIMELINE OF SELECT DATA NHTSA RECEIVED ON THE GM IGNITION SWITCH DEFECT AND RELATED ACTIONS Data NHTSA Received NHTSA Actions 2003 MAY Consumer complaint that a 2003 Saturn Ion stalled on a highway, causing the steering column to lock, and that the vehicle had to be turned off and restarted to regain control. SEPTEMBER Consumer complaint that a 2003 Ion engine shut off on three occasions—twice when the vehicle was traveling at 65 miles per hour—when the driver’s knee hit the car keys. 2004 JANUARY Consumer complaint that a 2003 Ion’s air bags failed to deploy in a frontal, on-road collision while traveling 60 miles per hour, injuring the driver. MARCH Consumer complaint that a 2003 Ion’s air bags did not deploy when it was travelling 45 miles per hour and rear-ended another vehicle. NOVEMBER Consumer complaint that a 2003 Ion’s air bags did not deploy during a frontal collision that resulted in injury and destroyed the vehicle. DECEMBER Consumer complaint that a 2004 Ion’s engine shut off when going over road bumps and that the dealer was not able to diagnose the problem. 2005 MARCH GM non-dealer field report that a 2005 Chevrolet Cobalt stalled on a highway when the driver’s knee hit the key holder. JUNE Copy of a consumer letter to GM’s Customer Service Department describing a 2005 Cobalt that stalled on three occasions, stating that a Chevrolet dealer gave her a GM bulletin on the problem and that the dealer’s service manager turned the ignition switch when he hit the keychain with his knee. SEPTEMBER Consumer complaint that a 2005 Cobalt’s “ignition shuts off while driving.” NOVEMBER Consumer complaint that a 2005 Cobalt “has a defective ignition switch” and that the dealer removed a key from the key chain and advised the owner to “watch his knee position while driving.” DECEMBER Consumer complaint on a 2005 Cobalt that “on three occasions, simply brushing the key chain...was enough to turn the car off” and that the vehicle dealer was not able to address this concern. AUGUST NHTSA launches a special crash investigation of a July 2005 fatal crash involving a 2005 Cobalt. The air bags did not deploy, and the ignition switch was in the “accessory” position. According to ODI staff, they were invited to participate in the on-site inspection. Exhibit E. Timeline of Select Data NHTSA Received on the GM Ignition Switch Defect and Related Actions 35 Data NHTSA Received NHTSA Actions 2006 APRIL GM documentation on a July 2005 fatal accident involving a 2005 Cobalt showing that the air bags did not deploy and the ignition switch was in the “accessory” position, and including a letter from an attorney representing the deceased’s family, stating that experts who examined the vehicle and data concluded that the air bags should have deployed. NOVEMBER Consumer complaint that a 2005 Ion’s air bags did not deploy during a frontal accident, injuring the driver, who wore a seatbelt. NOVEMBER Consumer complaint that a 2005 Cobalt’s air bags did not deploy during a frontal accident, killing two and injuring one. 2007 FEBRUARY October 2006 GM technical service bulletin is uploaded to Artemis. The bulletin describes inadvertent turning of the key cylinder and loss of electrical systems; it applied to vehicles that would be subject to GM’s ignition switch recall. MAY GM non-dealer field report describing ignition switch defect in a 2006 Pontiac Solstice: The “vehicle ignition system turns off when my knee hits the accessories attached to the key ring...I have turned the car off several times while driving.” SEPTEMBER Consumer complaint that a 2005 Cobalt shut off while driving, along with Chevrolet dealership service records showing that the dealership diagnosed the ignition switch as the cause of the problem. MARCH An analyst submits an early warning referral on an air bag nondeployment trend in Cobalts. According to ODI staff, they met with GM officials to discuss the issue but did not request GM to follow up. APRIL A special crash investigation report concludes it is possible that a 2005 Cobalt’s air bags did not deploy in a fatal accident due to the ignition switch position. NOVEMBER A Defects Assessment Panel decides not to investigate front air bag nondeployments in Cobalts and Ions; the panel’s reasoning is not documented. NOVEMBER Reports of fatalities prompt NHTSA’s Associate Administrator for Enforcement to request a follow up on Cobalt and Ion air bag non-deployments. NOVEMBER An air bag investigator reviews GM documentation for a fatal incident involving a 2005 Cobalt. The documentation links the ignition switch defect and air bag non-deployment, but the investigator concludes that delay between the first and final impacts caused the nondeployment. NOVEMBER Consumer complaint that a 2006 Chevrolet HHR “turns itself off” when going over road bumps. DECEMBER Consumer complaint that a 2006 Cobalt’s air bags did not deploy during an on-road frontal accident, injuring the driver. DECEMBER Consumer complaint that a 2006 Cobalt’s air bags did not deploy in an accident that destroyed the vehicle, asserting that a collision mechanic said the air bags should have deployed and that GM could not explain why the air bags failed. (Second complaint on this incident.) 2008 JANUARY Consumer complaint that a 2004 Ion’s air bags and seatbelt pre-tensioner did not deploy during a frontal crash, injuring the driver. OCTOBER Consumer complaint that a 2007 Chevrolet HHR stalls, and in one instance, while crossing train tracks. DECEMBER Consumer complaint that a 2006 Ion’s air bags did not deploy in a high-speed frontal crash, injuring the driver and passenger. JANUARY The air bag investigator submits his analysis of the fatal incident involving a 2005 Chevrolet Cobalt to the Defects Assessment Division and Early Warning Division Chiefs. Exhibit E. Timeline of Select Data NHTSA Received on the GM Ignition Switch Defect and Related Actions 36 Data NHTSA Received NHTSA Actions 2009 OCTOBER Consumer complaint that a 2006 Chevrolet HHR stalled three times in 6 months when driving over potholes or bumps in the road and that the dealership was not able to determine why the vehicle stalled. OCTOBER Consumer complaint that a 2006 Cobalt stalled four times in 3 months and that the dealership was not able to replicate or fix the problem. DECEMBER Consumer complaint that a 2008 Chevrolet HHR’s air bags did not deploy in a frontal collision, injuring the driver. APRIL NHTSA launches a special crash investigation of an April 2009 fatal crash involving a 2005 Cobalt. The air bags did not deploy, and the ignition switch was in the “accessory” position. ODI learns of the incident but does not follow up. 2010 MARCH Consumer complaint that a 2008 Pontiac G5’s air bags did not deploy after the driver lost control and crashed into a guard rail, injuring and hospitalizing three passengers. MARCH Consumer complaint that a 2006 Ion stalled and lost electrical systems, steering, and brakes after hitting a bump, requiring the driver to “force” the car to the side of the road to restart it. JULY Consumer complaint that a 2005 Cobalt stalls regularly while driving and that the dealership was not able to identify the cause of the problem. OCTOBER Consumer complaint that a 2006 Cobalt’s air bags did not deploy in a frontal accident, injuring the driver. FEBRUARY Two ODI staff members review the findings of the special crash investigation report on the April 2009 incident, but neither report the results of their reviews. FEBRUARY New complaints prompt a screener to revisit an August 2007 investigation proposal on air bag nondeployments in Cobalts and Ions. He ultimately decides not to resubmit the proposal because an air bag investigator identified a downward trend in the rate of air bag-related complaints for the vehicles. 2011 APRIL Consumer complaint that a 2005 Cobalt’s air bags did not deploy in a frontal accident, injuring the driver. JULY Consumer complaint that a 2007 Chevrolet HHR’s air bags did not deploy in a frontal accident, injuring the driver, and noted that a GM investigator inspected the vehicle after the accident. NOVEMBER Consumer complaint that a 2006 Cobalt’s air bags did not deploy in a frontal accident, injuring the driver. DECEMBER Consumer complaint that a 2007 Ion lost all power and “shut itself down” on multiple occasions while the car was being driven. MAY ODI’s early warning reporting analysis of 22 vehicles finds relatively high numbers of injury and death incidents attributed to air bags in the 2005 to 2010 Cobalts. The Cobalt ranked 15th in consumer complaints, but ranked 4th for early warning reporting of deaths and 2nd for injuries. 2012 MARCH Consumer complaint that a 2007 Cobalt’s air bags did not deploy after it crashed into another vehicle, injuring the driver. MARCH Consumer complaint that a 2005 Cobalt had shut off multiple times when driving over a speed bump or pothole. APRIL Consumer complaint that a 2005 Cobalt lost power sporadically while being driven. MARCH A screener reviews a March 2012 complaint about air bag non-deployment in a 2007 Cobalt and concludes “no actionable trend indicated” and “no further action required at this time.” The screener did not document the justification for this conclusion. MAY Consumer complaint that a 2006 Chevrolet HHR stalled and lost power on a ramp, causing the driver to lose control and hit the curb. Exhibit E. Timeline of Select Data NHTSA Received on the GM Ignition Switch Defect and Related Actions 37 Data NHTSA Received NHTSA Actions 2013 FEBRUARY Consumer complaint that a 2006 Ion’s air bags did not deploy during a frontal collision with a pole, severely injuring the driver and passenger. FEBRUARY Consumer complaint that a 2006 Cobalt’s air bags did not deploy in a crash with another car and that the driver hit his head on the steering wheel. JUNE Consumer complaint that a 2005 Ion’s air bags did not deploy in a crash with another car and that the driver hit his head on the steering wheel. JULY Consumer complaint that a 2007 Ion’s air bags did not deploy in a crash with another vehicle and that the driver fractured an eye socket. JULY A screener reviews a June 2013 complaint about air bag non-deployment in a 2005 Ion and concludes “no actionable trend indicated” and “no action at this time.” The screener did not document the justification for this conclusion. AUGUST A screener reviews a July 2013 complaint about air bag non-deployment in a 2007 Ion and concludes “no actionable trend indicated” and “no action at this time.” The screener did not document the justification for this conclusion. 2014 FEBRUARY GM notifies NHTSA that it is conducting a safety recall for the ignition switch of certain 2005 to 2007 Cobalts and 2007 Pontiac G5 vehicles; two and a half weeks later, GM revises the recall to include the 2003 to 2007 Ion, 2006 to 2007 Chevrolet HHR and Pontiac Solstice, and 2007 Saturn Sky. MARCH GM notifies NHTSA that it is again expanding the scope of the recall to include the 2008 to 2010 Cobalt, 2008 to 2011 Chevrolet HHR, 2008 to 2010 Pontiac Solstice, 2008 to 2010 Pontiac G5, and 2008 to 2010 Saturn Sky. Exhibit E. Timeline of Select Data NHTSA Received on the GM Ignition Switch Defect and Related Actions 38 EXHIBIT F. MAJOR CONTRIBUTORS TO THIS REPORT Name Title Wendy Harris Program Director Betty Krier Program Director Jay Borwankar Project Manager Olivia Starr Senior Analyst Michael English Senior Analyst James Lonergan Senior Analyst Jerrod Sharpe Senior Economist Brian McNamara Senior Economist Arturo Loya Analyst Seth Kaufman Senior Counsel Tom Denomme Consultant Karen Sloan Communications Officer Christina Lee Writer-Editor William Savage IT Specialist Exhibit F. Major Contributors to This Report 39 APPENDIX. AGENCY COMMENTS Memorandum U.S. Department of Transportation National Highway Traffic Safety Administration _______________________________________________________________________ Subject: From: To: INFORMATION: Management Comments – Office of Inspector General (OIG) Draft Report on NHTSA's Efforts To Identify and Investigate Vehicle Safety Concerns Date: June 16, 2015 Mark R. Rosekind, Ph.D. Administrator, National Highway Traffic Safety Administration Mitchell Behm Assistant Inspector General for Surface Transportation Audits The National Highway Traffic Safety Administration (NHTSA) Office of Defects Investigation (ODI) leads the world in protecting the driving public from vehicle safety defects. Over the last decade alone, ODI has conducted 1,060 defect investigations, resulting in 1,889 recalls, involving more than 129 million vehicles and items of equipment. During this time, the ODI staff of 8 defect screeners, 4 early warning data analysts, and 16 investigators received 1,617,245 different records reflecting the 265 million vehicles on U.S. roadways. Efforts to enhance safety never end and examining lessons learned is critical to improving NHTSA’s effectiveness in pursuing the agency’s vital mission. After General Motors (GM) submitted its defect notice on the first ignition switch recall in 2014, NHTSA (with assistance from the Department of Transportation’s Office of the General Counsel) initiated its own due diligence review. This review led to the development of two documents: 1) NHTSA’s Path Forward is a critical look at the GM ignition switch defect issue, and outlines process improvements that go beyond this specific recall; and 2) NHTSA’s Workforce Assessment: The Future of NHTSA’s Defect Investigations identifies staffing and training needs for both near and long term enhancements. Based on ongoing efforts to enhance NHTSA and ODI effectiveness, as well as the most recent examination of lessons learned in the GM ignition switch recall, improvements have already been instituted in pre-investigative, investigation, and recall completion processes. NHTSA also concurs with OIG’s 17 recommendations, as written and will aggressively implement them (see chart on page 3). Additionally, extensive changes have already been implemented with many others underway or planned. Here are examples of such actions: • Tracking pre-investigative work in a dedicated case management system that connects data from various sources that concern each issue being evaluated. Appendix. Agency Comments 40 • • • • • • • • • • • Maintaining detailed records of issues presented to a defects panel for an investigation decision, showing the panel’s date, attendance, and disposition of each issue. Special Crash Investigations (SCI) staff is present at all defect panel meetings to ensure the discussion includes the latest crash investigation information (relevant to GM issue). Continuing to follow up on all Early Warning Reporting (EWR) reports involving fatal incidents and making improvements to its handling and documentation of these incidents. When necessary, NHTSA will send manufacturers pre-investigative notices when the potential severity is very high but there is insufficient evidence to open an investigation (relevant to GM issue). Use of a systems safety approach to look for possible relationships between a symptom in one vehicle system and a possible critical failure of another system and to consider possible defect theories that do not fit with previously-held assumptions (relevant to GM issue). Use of a detailed documentation checklist for investigations to ensure that all relevant documents are identified and stored in a consistent manner. A new training plan for staff, focused on the pre-investigative and investigative divisions, to gain proficiency in new automotive and investigative technologies. Working with NHTSA’s Office of Vehicle Safety Research, ODI is increasing its interactions with manufacturers regarding the latest automotive technologies. To increase identification of potential defects in new safety technologies, ODI is developing standardized inquiries for screening and investigating data about those new systems. Increased contacts with government counterparts in foreign countries to enhance cooperation on common defect issues. Several improvements to increase ODI recall completion rates have been made, including: o Development of a consumer option to sign up for immediate email notification of new recalls. o Enhancement to recall notice envelopes to focus attention to their safety-critical nature. o Design of a Vehicle Identification Number (VIN) lookup website allowing consumers the ability to quickly determine whether their vehicle has an open recall. o Launching of smart phone mobile apps for consumers to receive recall notifications, file complaints, and lookup VINs. In addition to the improvements already implemented, more are being developed, to include: • • • • • Providing more clarity to manufacturers about the EWR requirements and assisting manufacturers as they implement best practices to comply. Determining a mechanism to obtain detailed audits of manufacturers’ internal processes for finding defects. Creating ways for consumers to provide more complete information to the agency, including making it easy to upload supporting documentation to the complaint. Establishing a Safety System Team, a small group of outside safety experts, to help implement changes and recommendations from internal and external reports. Creating an internal risk identification and control team to ensure that pertinent changes are implemented and established for the long term. Appendix. Agency Comments 41 • • • Continuing the development and use of the computerized Corporate Information Factory (CIF), business intelligence software that will allow screeners and investigators to identify and data-mine information across NHTSA’s data sets by integrating multiple databases. Developing a CIF process to track SCI reports throughout the review process and quickly bring crash investigation reports changes to the attention of appropriate ODI staff. Developing and implementing internal controls that require the defect assessment panel to revisit an issue or open a formal investigation if certain criteria are met. Below is a chart showing NHTSA’s anticipated completion dates for the OIG recommendations. Estimated Completion Date September 30, 2015 October 30, 2015 November 30, 2015 January 31, 2016 April 30, 2016 May, 30, 2016 June 30, 2016 OIG Recommendation Number(s) 6 13, 15 17 3*, 10 2, 5 1*, 11, 14 4, 7, 8, 9, 12, 16 *Unless rulemaking is required NHTSA will pursue any efforts that can enhance the agency’s effectiveness in achieving its safety mission and appreciates the opportunity to provide this input to the OIG draft report. Please contact Frank S. Borris, Director, Office of Defects Investigations, at (202) 366-8089 with any questions or additional details about these comments. Appendix. Agency Comments