BOEING 787–8 DESIGN, CERTIFICATION, AND MANUFACTURING SYSTEMS REVIEW BOEING 787–8 CRITICAL SYSTEMS REVIEW TEAM March 19, 2014 Prepared for Ms. Dorenda D. Baker Director, Aircraft Certification Service Federal Aviation Administration Washington, DC Mr. Daniel P. Mooney Vice President of Boeing South Carolina Design Center Boeing Commercial Airplanes Seattle, Washington BOEING 787–8 CRITICAL SYSTEMS REVIEW TEAM Michael Kaszycki, Co-chair Rich Ptacin, Co-chair Manager, Transport Standards Staff FAA Aircraft Certification Service Director, 787 Deputy Chief Project Engineer Boeing Commercial Airplanes Christopher B. Bergen Stephen P. Boyd Manager, Phoenix and Van Nuys FAA Manufacturing Inspection District Offices Manager, Airplane and Flight Crew Interface Branch FAA Transport Airplane Directorate TJ Ginthner Jerome R. Hulm Sr. Manager, Regulatory and Quality System Oversight Boeing Commercial Airplanes Associate Technical Fellow Boeing Commercial Airplanes James Mitchell Christopher R. Parker Technical Fellow, Flight Control System Design Boeing Commercial Airplanes Aerospace Engineer, Transport and Rotorcraft Program Manager FAA Seattle Aircraft Certification Office Bradford A. Moravec Brett E. Portwood Chief Engineer, Mission Assurance Propulsion Systems Division Boeing Commercial Airplanes Technical Specialist for Safety and Integration FAA Los Angeles Aircraft Certification Office Jeff Shario Michael H. Song Chief Engineer, Propulsion Safety and Airworthiness Boeing Commercial Airplanes Director, Quality Engineering Boeing Commercial Airplanes Ian Y. Won Aerospace Engineer Airframe/Cabin Safety Branch, FAA Transport Airplane Directorate CSRT Program Support: Kristin Grimlund, Airplane Programs Business Operations, Boeing; and Matthew Bentley, Airplane Level Integration, Boeing Boeing 787–8 Critical Systems Review Team Report Page ii TABLE OF CONTENTS Executive Summary .......................................................................................................... vii 1. Introduction ................................................................................................................... 1 1.1. Background .......................................................................................................... 1 1.2. Scope of This Review .......................................................................................... 2 1.3. CSRT Membership and Activity ......................................................................... 2 1.3.1. CSRT Meetings .............................................................................................. 3 1.3.2. CSRT Phased Approach ................................................................................ 3 1.3.3. Review of Airplane Design and Design Processes ........................................ 4 2. CSRT Review Analysis—Key Messages ..................................................................... 7 2.1. Introduction .......................................................................................................... 7 2.2. B787 Entry Into Service—Comparison to Other Boeing Airplane Models ........ 7 2.2.1. Data Analysis ................................................................................................. 8 2.2.2. Summary ...................................................................................................... 10 2.3. Component Removals ........................................................................................ 11 2.4. Novel Technologies ........................................................................................... 11 2.5. Business Model .................................................................................................. 12 2.5.1. B787 Systems and Equipment ..................................................................... 12 2.5.2. Manufacturing/Quality................................................................................. 13 2.5.3. FAA Certificate Management of Large Production Approval Holders ....... 14 2.6. Design Requirements ......................................................................................... 15 2.6.1. Background .................................................................................................. 15 2.6.2. Requirements Issues..................................................................................... 15 2.7. Inspection Delegation ........................................................................................ 17 2.8. Regulatory Oversight—FAA Review and Acceptance Policy .......................... 18 2.9. FAA Engineering Conformity ........................................................................... 19 3. Conclusions ................................................................................................................. 21 3.1. B787 Met Expectations in Service ..................................................................... 21 3.2. Improvement Needed ......................................................................................... 22 3.2.1. Manufacturing/Quality Business Model ...................................................... 22 3.2.2. Design requirements .................................................................................... 22 3.2.3. Inspection Delegation .................................................................................. 23 Boeing 787–8 Critical Systems Review Team Report Page iii 3.2.4. FAA Regulatory Oversight .......................................................................... 23 4. FAA Recommendations .............................................................................................. 24 Appendix A—Deep-Dive Review Summaries .............................................................. A‒1 A.1. Introduction ..................................................................................................... A‒1 A.2. Systems ........................................................................................................... A‒2 A.2.1. Selection Process Summary ...................................................................... A‒5 A.2.2. Observations Summary ............................................................................. A‒5 A.2.3. Conclusions ............................................................................................. A‒11 A.3. Propulsion ..................................................................................................... A‒12 A.3.1. Scope of Subteam Review ...................................................................... A‒15 A.3.2. Selection Process Summary .................................................................... A‒15 A.3.3. Observations Summary ........................................................................... A‒15 A.3.4. Conclusions ............................................................................................. A‒17 A.4. Structures ...................................................................................................... A‒17 A.4.1. Selection Process Summary .................................................................... A‒18 A.4.2. Observations Summary ........................................................................... A‒19 A.4.3. Conclusion .............................................................................................. A‒19 A.5. Manufacturing/Quality.................................................................................. A‒20 A.5.1. Selection Process Summary .................................................................... A‒20 A.5.2. Observations Summary ........................................................................... A‒21 A.5.3. Conclusion .............................................................................................. A‒26 A.6. CSRT Deep-Dive Review Summaries—Observations and Recommendations .................................................................................. A‒27 Appendix B—Acronyms ................................................................................................ B‒1 Cover image ©2013 Boeing. All rights reserved. Boeing 787–8 Critical Systems Review Team Report Page iv LIST OF FIGURES AND TABLES Table 1. CSRT Supplier Visits........................................................................................... 5 Figure 1. Schedule Reliability............................................................................................ 8 Figure 2. B787 vs. B777 EE–1s Since Entry Into Revenue Service.................................. 9 Figure 3. 14 CFR § 21.3 EIS Reporting........................................................................... 10 Figure A–1. B787 Components/Systems Selected for Deep-Dive Review .................. A–1 Figure A–2. Electric Power Generation & Start System .............................................. A–3 Figure A–3. Spoiler Electromechanical Actuators ....................................................... A–4 Figure A–4. Elevator Actuator...................................................................................... A–4 Figure A–5. Valve Actuator Micro-Switches and Position Indication ....................... A–13 Figure A–6 Main Wing Fuel Tank Access Door Types and Locations ...................... A–14 Figure A–7. Full Flexible Coupling ............................................................................ A–14 Figure A–8. Horizontal Stabilizer and Aft Fuselage Sections 46, 47, and 48 ............ A–18 Figure A–9. Boeing 787 Value Stream Review.......................................................... A–21 Figure A–10. Traditional vs. Nontraditional Supply Chains ...................................... A–23 Boeing 787–8 Critical Systems Review Team Report Page v LETTER FROM THE BOEING 787–8 CRITICAL SYSTEMS REVIEW TEAM August 16, 2013 Ms. Dorenda D. Baker Director, Aircraft Certification Service Federal Aviation Administration 800 Independence Avenue SW. Washington, DC 20591 Mr. Daniel P. Mooney Vice President of Boeing South Carolina Design Center Boeing Commercial Airplanes P.O. Box 3707 Seattle, WA 98124-2207 Dear Ms. Baker and Mr. Mooney, On January 31, 2013, the Federal Aviation Administration (FAA) and Boeing Commercial Airplanes (Boeing) tasked the Boeing 787‒8 Critical Systems Review Team (CSRT) to perform a comprehensive review of the Boeing 787‒8 critical systems, including the airplane’s design, manufacture, and assembly. The CSRT is pleased to submit to you our report on the Boeing 787–8 Design, Certification, and Manufacturing Systems Review. We believe the observations and recommendations presented in this report will enhance an already robust airplane certification and safety monitoring process. We anticipate our key messages will be useful to the steering group and ultimately to the Secretary of Transportation, the FAA Administrator, and Boeing leadership. During its 6-month task, the CSRT members used their expertise and exercised independent judgment to validate the work conducted during the Boeing 787‒8 certification process. The CSRT met extensively with Boeing suppliers and experienced exceptional cooperation during the review. Without such supplier assistance, this analysis would not have been possible. On behalf of the CSRT, we thank the steering group for selecting us to be a part of this important effort and allowing us to provide our insight. Sincerely, Mr. Michael Kaszycki Manager, Transport Standards Staff Aircraft Certification Service Federal Aviation Administration Boeing 787–8 Critical Systems Review Team Report Mr. Richard R. Ptacin Director, 787 Deputy Chief Project Engineer Boeing Commercial Airplanes Page vi EXECUTIVE SUMMARY BACKGROUND This final report is in response to the Federal Aviation Administration’s (FAA) and Boeing Commercial Airplanes’ (Boeing) assignment to validate the work conducted during the Boeing 787 (B787) certification process and further ensure the airplane meets the intended level of safety. On January 31, 2013, the FAA and Boeing jointly formed the B787 Critical Systems Review Team (CSRT) to conduct a comprehensive review of the B787’s critical systems, including the airplane’s design, manufacture, and assembly, and provide recommendations. From February 1, 2013, to July 31, 2013, the CSRT, composed of FAA and Boeing subject matter experts, conducted in-depth reviews of B787 critical systems based on in-service data and using safety risk management principles. These subject matter experts have backgrounds in both engineering (systems, structures, and propulsion) and manufacturing/quality. The CSRT used in-service and in-production issues to focus its review. To further define the scope of its activities, the CSRT employed a safety-risk methodology to prioritize areas for review. The CSRT then conducted a phased review of the following critical systems, assemblies, and related processes or facilities, as appropriate: Engineering • Variable Frequency Starter Generators (VFSG) • Generator Control Units (GCU) • Primary Electrical Power Panels • Spoiler Electromechanical Actuators • Primary Flight Control System Hydraulic Actuators • Manufacturing Fuel Tank Access Doors o o • Electromagnetic Effects Bonding • VFSG (manufacturing only) • Final Assembly • Aft Fuselage Sections 47/48 Impact-Resistant Wing Fuel Tank Skin Surfaces— Electromagnetic Effects Protection o Boeing South Carolina Motor-Operated Ball Valves Fuel Line Couplings • Aft Fuselage Sections 46/47/48 • Fuel Line Coupling (manufacturing only) • FAA Oversight Processes • Horizontal Stabilizer Mid-Body Fuselage Sections 44/46 Horizontal Stabilizer • GCU (manufacturing only) • • • • o Alenia Aermacchi SPA Appendix A, Deep-Dive Review Summaries, describes the functions of these components, assemblies, and related processes. Boeing 787–8 Critical Systems Review Team Report Page vii CSRT ANALYSIS Upon completing its review, coordinating observations, and conducting an analysis, the CSRT concluded the B787 meets its intended level of safety based on (1) the fundamental soundness of the airplane’s overall design and (2) the effective processes that have been defined and implemented to correct issues that arose during and after certification. In performing the in-depth “deep-dive” reviews of components, assemblies, and related processes, the CSRT identified deficiencies that were either 1) already being addressed by FAA/Boeing continued operational safety (COS) processes or 2) mitigated by the B787’s redundant system architecture. The CSRT validated Boeing’s established compliance design and manufacturing processes work in concert with FAA regulatory requirements and processes to provide a high level of safety for the B787. The CSRT made four recommendations to Boeing to address the issues noted during its review, and made three observations related to FAA policy and guidance issues. The following is a discussion of the CSRT’s observations of the B787’s critical systems that led to its recommendations for Boeing and its observations on FAA oversight. ENGINEERING The CSRT did not observe any significant issues associated with component and/or system design processes. However, the CSRT identified some issues and determined they were being dealt with using standard practices. Although the FAA and Boeing expect first-time quality for every piece designed and built, reality and history show that defects occur. The aviation industry has standards and practices to ensure that even when such defects occur, they are identified, understood, addressed, and not repeated. Because of the B787’s conservative design and redundant systems architecture, the B787 program was found to be operating within these expectations. The deficiencies the CSRT observed are typical of a new airplane model entering service and are being addressed or have been addressed by Boeing’s product improvement processes or the FAA and Boeing COS processes. The following CSRT observations applied across the engineering areas/disciplines (systems, structures, and propulsion) reviewed: • Requirements flowdown. The CSRT identified inconsistencies in design requirements flowdown and design verification. For example, in some cases complete and accurate design requirements did not flow down from Boeing to its primary supplier and then to the involved subtier suppliers. Boeing had established design requirements, but these requirements were inadequately verified and/or validated, resulting in inconsistency in parts manufacturing, part failures, and operational disruptions such as turn backs and diversions. • Responsibility. The CSRT identified communication and verification issues along the supply chain. In some cases, these issues occurred because Boeing or its major suppliers with integration responsibilities did not clearly establish which subsupplier providing components for an integrated system was responsible for a specific detailed design requirement. Boeing 787–8 Critical Systems Review Team Report Page viii • Design review process/industry design standards. The CSRT did not identify significant issues with design requirements or certification processes; however, it did find instances in which additional Boeing oversight likely would have helped ensure specified design processes were followed, especially when designs evolved over time. For example, a design feature made it impossible to install a non-impact-resistant fuel tank access door in locations where an impact-resistant door was required. During subsequent improvements to the door design, a poorly executed design change process allowed deletion of that design feature, thereby making the doors interchangeable. The team also noted that when design requirements or processes were unclear, companies along the supply chain made incorrect assumptions and did not always default to their own or industry design standards. In some cases, requirements ambiguity led suppliers to incorrectly assume they successfully met all the requirements. However, the actual requirements had not been satisfied. The suppliers made these determinations independently, without consulting Boeing or the higher-tier supplier. • New technologies. The B787 employs many new technologies and innovative designs. The CSRT assessed whether technological innovations contributed to the in-service issues reviewed. Although some of the issues the CSRT investigated were associated with new technologies, it determined the primary cause was not the novelty of the technologies. For example, the cause of a given issue may have been improper implementation of a correct design requirement. The technology was well understood, but some aspect of the design did not meet the requirements. • New applications of existing technology. Design requirements for new applications of traditional components were not consistently verified and/or validated where these components were installed. The design feature was assumed to be already proven and tested, and the design weaknesses were found when the airplane was in service. Similarly, previous experience with how a system would perform led to inadequate design requirements for the new system. • Business model. Boeing is responsible for demonstrating and maintaining compliance with FAA regulations. Boeing’s B787 business model uses several levels of suppliers for design and production responsibility. The CSRT assessed several large components or integrated systems designed by suppliers and integrated by Boeing into the airplane. Boeing’s approach for the B787 was different from previous airplane programs where it retained more of the detailed design responsibilities. The CSRT noted the unique aspects of this approach did not directly contribute to the underlying cause of in-service issues reviewed. The causes typically related to the basic communication and coordination issues that any large and complex new airplane development program may encounter. Boeing 787–8 Critical Systems Review Team Report Page ix MANUFACTURING The CSRT noted Boeing suppliers experienced some startup issues with the new business processes for the B787. Under the new approach, certain suppliers manufactured major sections of the airplane, then installed many components and systems into those sections. This new manufacturing strategy magnified the challenges for the manufacturing quality area resulting from the extraordinary number of new manufacturing and assembly processes required for the B787 type design. Before the CSRT review, Boeing addressed these business process issues and made improvements to its quality system. The CSRT recognized the Boeing business model, which uses several levels of suppliers for design and production responsibility, is not new to the worldwide aerospace industry, but noted no other U.S. aircraft manufacturer has shared such responsibility on such a large scale. This led the CSRT to identify areas where FAA policy does not align with Boeing’s new aircraft manufacturing environment that includes intricate international supply chains, novel technologies, and risk management. The following CSRT observations applied to the manufacturing/quality area reviewed. • Business model. Suppliers experienced a learning curve when using these new manufacturing and assembly processes with Boeing. In some cases, this learning curve affected the production rate of components. Boeing has since increased support to its suppliers (a large staff of Boeing employees is onsite at some suppliers) and is working to remedy supply chain issues. The CSRT determined suppliers would better be able to assess risk and implement appropriate mitigation plans with a closed-loop system to define and describe the issues to be solved, identify causes, test and validate solutions, and implement and sustain the solutions. • Structures. The CSRT did not find systemic engineering or design/certification issues during its examination of the horizontal stabilizer and aft fuselage sections, but did review some structural shimming issues directly related to various aspects of the assembly and manufacturing processes. • Inspection delegation. The FAA and Boeing use inspection delegation, in which inspection responsibilities are entrusted to another party—in this case, lower level suppliers. For some FAA-required inspections, the FAA may delegate the FAA inspection to Boeing’s organization designation authorization (ODA) or supplier designees. For other inspections, such as first article inspections to meet Boeing internal requirements, Boeing may delegate the inspection responsibility to suppliers. The CSRT noted that overall, inspection delegation worked well throughout the supply chain. However, the CSRT observed there are industry standards for inspection delegation (for Boeing’s internal inspections) that include training, testing, and currency requirements for inspectors, but not all B787 suppliers follow these standards. • FAA policy. Current FAA policy on acceptance of an aircraft manufacturer’s production capability applies a similar assessment methodology to both the manufacturer of small, less complex aircraft as well as the manufacturer of large, complex transport aircraft with extended international supply chains. The CSRT Boeing 787–8 Critical Systems Review Team Report Page x observed current FAA certificate management policy does not ensure the use of a comprehensive risk-based plan. OTHER OBSERVATIONS Entry-Into-Service Data As one indicator of the B787’s intended level of safety, the CSRT compared B787 entry-into-service (EIS) operational reliability data—schedule reliability data, Extended Operations (ETOPS) data on maintenance issues, and reports of certain occurrences Boeing must submit to the FAA under Title 14, Code of Federal Regulations § 21.3(c), Reporting of failures, malfunctions, and defects—with similar data on previous Boeing airplane models. The CSRT determined the B787 EIS reliability performance is comparable to that of other new Boeing transport airplanes entering into service during its initial 16 months of service. Component Removals The CSRT reviewed in-service data on removed components to determine the cause of critical system component failures and their effect on the airplane’s safety. The CSRT observed B787 operators often remove and replace all potential sources of failure so the airplane can be quickly returned to service. As a result, no fault was found when many of the items identified as failed and removed from service were later checked. Therefore, this practice of removing all possible sources of a failure inflated the failure rate data for specific components. The CSRT concluded that some of the in-service data examined involving removed components did not indicate system reliability issues, but rather was the result of this airline maintenance practice. Engineering Conformity The CSRT reviewed Boeing data showing the FAA (or the Boeing ODA) performed substantially more engineering FAA conformity inspections for the B787 certification program than for the Boeing 777 (B777) certification program. Conformity inspections are done to ensure tested parts and subassemblies match the engineering specifications. Manufacturers are required to perform 100-percent conformity inspections for all tests and inspections used to show compliance with the regulations. The FAA or airworthiness representatives (on the FAA’s behalf) perform FAA conformity inspections to verify the manufacturer’s conformity reports. The CSRT observed varied FAA guidance on the requirements for FAA engineering conformity inspections. The CSRT also observed that 1) the FAA designee system changed during B787 certification program, and 2) the use of novel technologies, design processes, and manufacturing processes on the B787 introduced additional risks regarding conformity of test articles. All of these factors may have contributed to an increase in the number of FAA engineering conformities, but the CSRT was unable to determine the exact cause of the increase. The CSRT noted, however, that FAA orders are not clear and consistent on whether FAA engineering conformity inspections are required on all compliance test articles or on a selected subset. Boeing 787–8 Critical Systems Review Team Report Page xi CORRECTIVE ACTIONS For each in-service and in-production issue the CSRT reviewed, it also evaluated any implemented corrective actions. For engineering issues, these corrective actions typically took the form of design changes. In some cases, the changes were incorporated immediately into the units being produced; in other cases, the changes were scheduled for incorporation at the next planned system update. For most of the issues, a retrofit for in-service airplanes was also developed (or in work). To address safety issues in a few cases, Boeing, its suppliers, and the FAA worked together on a plan for mandatory incorporation of the design changes via airworthiness directive. All of these actions were accomplished using COS and product improvement processes. In all cases reviewed, the CSRT found an acceptable corrective action had been initiated and the appropriate level of urgency had been established for the corrective actions. For manufacturing issues, Boeing’s and its suppliers’ quality systems already had identified the issues and initiated or fully implemented corrective actions. In some cases, the issue had been identified and addressed on each individual airplane before the FAA issued its Certificate of Airworthiness. For those in-service issues traced to manufacturing issues, corrective actions had been initiated in accordance with established quality system and COS processes. In summary, the CSRT found existing processes for problem reporting, product improvement, manufacturing quality assurance, and COS to be effective in addressing the issues investigated. CONCLUSION The CSRT determined the B787 meets its intended level of safety based on (1) the fundamental soundness of the airplane’s overall design and (2) the effective processes that have been defined and implemented to correct issues that arose during and after certification. Although design issues have occurred, the CSRT found their causes tended to represent individual escapes in the design or manufacture of the airplane. In the judgment of the CSRT, a certain number of such escapes are to be expected in the development of a complex product such as a large airplane, due to state-of-the-art limitations in current design, manufacturing, and certification processes. For manufacturing, early issues with suppliers implementing the new business processes are being addressed, and improvements are in progress throughout the supply chain. The FAA’s and Boeing’s COS processes effectively evaluated and addressed any safety risk associated with each in-service event reviewed. The CSRT noted that Boeing’s internal product improvement processes are addressing the non-safety problems that primarily affected airplane economics and customer satisfaction. The process improvements presented in the CSRT’s recommendations to Boeing, when implemented, will improve performance, reduce risk, and help reduce the occurrence of future in-service events for the B787 program and future airplane programs. Boeing 787–8 Critical Systems Review Team Report Page xii Finally, the CSRT determined that appropriate corrective actions have been implemented or initiated for each issue investigated. Normal problem reporting, quality assurance, product improvement, and COS processes have been effective and are expected to adequately address any new issues that may arise over the life of the B787 fleet. CSRT RECOMMENDATIONS/OBSERVATIONS The following is a list of the CSRT’s recommendations to Boeing, followed by a list of the CSRT’s observations on FAA policy and guidance issues: RECOMMENDATIONS Recommendation No. 1: Boeing should establish a means to ensure suppliers identify realistic program risks and complementary mitigation plans through a closed-loop flowdown validation of requirements. 1 (Also see Recommendation No. 2 regarding allocation of sufficient resources.) Recommendation No. 2: Boeing should continue to implement and mature the gated design and production processes with sufficient resources for development programs, and to minimize risks throughout the life cycle of the program. In these processes, a series of programmatic “gates” are established at various points during the development program. Each gate has specific criteria for proceeding to the next development phase. Any criteria that have not been satisfied at a given gate must be addressed or mitigated before proceeding to the next phase. (Boeing is realizing improved performance in the Boeing 737 MAX, Boeing 787‒9, and Boeing 767‒2C programs from using a gated approach.) Recommendation No. 3: Boeing should ensure suppliers are fully aware of their responsibilities, including integration responsibilities and accountability for subtier performance. The gated design processes should include supplier planning, performance, and reporting, using measurable and appropriate performance criteria that include the scope and effectiveness of design reviews, and other airplane life-cycle activities. Recommendation No. 4: Boeing should require its suppliers to follow industry standards for the training, qualification, and certification of supplier personnel performing Boeing-required (non-FAA) inspections. 1 Closed-loop corrective action is a process in which suppliers define and describe the problem to be solved, identify causes, test and validate solutions, implement the solutions, sustain the solutions, and monitor results to ensure the solutions yield the intended improvements. Boeing 787–8 Critical Systems Review Team Report Page xiii OBSERVATIONS Observation No. 1: The FAA policy regarding certificate management of production approval holders addresses only risk at the manufacturer’s top system level. The policy does not require development of tailored certificate management plans that specifically identify and target risk wherever responsibilities and expectations exist within complex supply chains. FAA policy also does not encourage FAA manufacturing oversight offices to conduct surveillance at critical subtier suppliers when first-tier suppliers are major integrators, even though they may not manufacture a significant portion of the assembly. Observation No. 2: The FAA policy regarding production approval procedures does not recognize the differing levels of complexity of manufacturing systems and technologies between small, relatively simple aircraft manufacturers and large-scale, complex aircraft manufacturers with extended supply chains. The production approval process does not focus on aircraft complexity and critical technologies (both innovative and existing) using a comprehensive risk-based plan. Observation No. 3: FAA ODA policy does not provide adequate guidance to ensure risk-based conformity inspection plans. FAA RECOMMENDATIONS Based on the CSRT’s three observations noted above, the FAA CSRT team members recommend the following changes be made to FAA policy and guidance documents. FAA Recommendation No. 1: The FAA should revise chapters 3 and 4 of FAA Order 8120.23, Certificate Management of Production Approval Holders, to recognize new aircraft manufacturing business models and their potential impact on safety, complexity, risk, and mitigating actions. FAA Recommendation No. 2: The FAA should revise chapter 3 of FAA Order 8120.22, Production Approval Procedures, to recognize the changing aircraft manufacturing environment and to more fully address complex, large-scale aircraft manufacturers with extended supply chains, expectations, and production capabilities. FAA Recommendation No. 3: The FAA should revise FAA Order 8110.4C, Type Certification, and FAA Order 8100.15B, Organization Designation Authorization Procedures, to provide clear and consistent guidance to ensure FAA engineering conformity inspections for all projects (including ODA projects) are based on risk. The orders should require FAA (or ODA) approval of the risk-based conformity plan. Boeing 787–8 Critical Systems Review Team Report Page xiv 1. INTRODUCTION 1.1. BACKGROUND On January 11, 2013, based on a series of in-service events, Federal Aviation Administration (FAA) Administrator Michael P. Huerta announced the FAA and Boeing Commercial Airplanes (Boeing) would perform a comprehensive review of the Boeing Model 787‒8 2 critical systems, including the airplane’s design, manufacture, and assembly. The review was to validate the work conducted during the certification process and further ensure the airplane meets the intended level of safety. On January 31, 2013, the FAA and Boeing jointly established the B787 Critical Systems Review Team (CSRT) to conduct this comprehensive review. The CSRT was composed of FAA and Boeing technical specialists representing engineers, manufacturers, and quality inspectors. To help provide a broad understanding of the in-service events, the CSRT initiated this review using a data-driven approach and safety risk management processes designed to identify possible systemic airplane issues instead of focusing only on individual events. The CSRT evaluated the individual causes of the selected in-service events to determine whether shared or overlapping causes existed that would need to be addressed to safeguard against similar events in the future. THE BOEING 787–8 AIRPLANE The B787‒8 is the first member of the B787 family of airplanes. B787 series airplanes are characterized by a composite fuselage, fly-by-wire flight controls, advanced flight deck features, composite wing airfoils, and General Electric or Rolls-Royce engines. In addition, this airplane minimizes the use of bleed air from the engines and extensively incorporates electrically powered systems, rather than conventional pneumatically powered systems. 3 On August 26, 2011, the FAA issued a type certificate for the B787 and amended Boeing’s Production Certificate No. 700 to include the B787‒8. The first airplane delivery to an airline occurred approximately 1 month later. In accordance with the type certification process, the regulatory requirements applied to the B787 were those requirements in effect on the date Boeing applied for the type certificate as well as the additional amendments in effect on the date of Boeing’s request for a schedule extension. Those requirements, referred to as the certification basis, were Title 14, Code of Federal Regulations (14 CFR) part 25, Airworthiness Standards: Transport Category Airplanes, 2 Hereinafter referred to as the B787 unless specifically noted otherwise. See http://www.boeing.com/boeing/commercial/787family/specs.page? for the airplane’s technical specifications. 3 Boeing 787‒8 Critical Systems Review Team Report Page 1 through amendment 25‒128 (issued on May 22, 2009) and special conditions 4 developed to support state-of-the-art technologies incorporated by Boeing. 1.2. SCOPE OF THIS REVIEW The CSRT used in-service and in-production issues as the means to focus its review. It is important to note that FAA and Boeing standardized processes for problem reporting, continued operational safety (COS), manufacturing quality assurance, and product improvement were the basis for the FAA and Boeing responses to the B787 in-service and in-production events. Although the information developed during the CSRT review may be used to support those processes, it was not dependent on them. The CSRT focused on B787 design, manufacture, and assembly. Given the timeframe for the CSRT to review the airplane’s systems, it elected not to examine the B787 engines as part of its review. Also, the engines are certificated under their own type certificate and are subject to their own set of airworthiness directives distinct from the airplane type certification. This review did not duplicate any ongoing incident investigations and relied on information from those activities as appropriate. Currently, the National Transportation Safety Board and the Japanese Transportation Safety Board are investigating separate lithium-ion battery overheat events. Also, the United Kingdom Aviation Accident Investigation Board is investigating an on-ground fire event that may have involved the emergency locator transmitter. Because these investigations are not yet completed, this report does not address any issues specific to those events. All B787 events are being addressed by either Boeing product improvement processes or FAA and Boeing COS processes as appropriate. 1.3. CSRT MEMBERSHIP AND ACTIVITY The joint FAA-Boeing team was composed of 2 co-chairs and 11 subject matter experts (SME). The co-chairs and the 11 SMEs are listed on page ii of this report. The CSRT reported to the steering group, which oversaw CSRT activities and provided administrative and technical guidance and other support as needed. 4 A special condition is a rulemaking action specific to an aircraft make and model and often concerns the use of new technology that the Code of Federal Regulations does not yet address. Special conditions are an integral part of the certification basis and impose appropriate requirements to build the aircraft, engine, or propeller with additional capabilities not referred to in the regulations. Boeing 787‒8 Critical Systems Review Team Report Page 2 To conduct its review, the CSRT divided into the following four subteams representing the disciplines under review: • Systems, • Propulsion, • Structures, and • Manufacturing/Quality. Each subteam performed in-depth “deep-dive” reviews of manufacturing, assembly, flight test, and service data, and identified focus areas using the CSRT’s research and risk assessment process. See section 1.3.2 of this report for a discussion of the CSRT’s phased approach. 1.3.1. CSRT MEETINGS The CSRT held its kickoff meeting on February 25, 2013, at Boeing’s facility in Everett, WA. Over the following 6-month period, it held 7 full team meetings and multiple teleconferences and subteam meetings. The CSRT visited 11 supplier and/or final assembly facilities, plus the Boeing Everett Modification Center. CSRT program support prepared minutes of the CSRT meetings and tracked team action items. The CSRT co-chairs also presented bimonthly status updates on the CSRT’s progress to the steering group. 1.3.2. CSRT PHASED APPROACH The CSRT established a strategy to perform its review using a phased approach, with a July 31, 2013, target date 5 for completing its review and submitting its report to the steering group. PHASE 1 Phase 1 consisted of data gathering for CSRT review using the following data sources: • Component reliability reports; • 14 CFR § 21.3, Reporting of failures, malfunctions, and defects/ COS process reports/Extended Operations (ETOPS) in-service events (EE–1); • Notices of escapement (nonconformance); • Rejection tags (nonconformance) from the Material Review Board; 6 • Build verification test 7 data; 5 The CSRT received an extension from the steering group to complete its review and submit its report by August 23, 2013. 6 The MRB meets regularly to disposition discrepant material that fails inspection. Boeing 787‒8 Critical Systems Review Team Report Page 3 • FAA Aviation Safety Information Analysis and Sharing (ASIAS) incidents; • Operator reports/dispatch reliability; and • Other data as identified by each subteam. PHASE 2 During Phase 2, the CSRT developed a model that— • Identified a sorting/risk assessment method. • Defined criteria to measure potential scope areas. • Documented explanations for in-scope/out-of-scope areas. The CSRT entered the data from Phase 1 into the Phase 2 model to identify the in-scope areas for the Phase 3 deep-dive review process. The deep-dive selection criteria included a review of (1) the in-service record, (2) effects of the business model, (3) the novelty of the design, and (4) the complexity of the component’s integration. PHASE 3 Phase 3 consisted of a deep-dive review of areas identified as in-scope. Once the CSRT selected a component for deep-dive review, the affiliated subteam outlined a plan for a systematic review of that component’s engineering or manufacturing process, as applicable. The subteams then performed their reviews, organized their observations, and looked for systemic trends across the deep-dive reviews. PHASE 4 During Phase 4, the CSRT coordinated observations, conclusions, recommendations to Boeing, and lessons learned from the deep-dive assessments, and drafted its report. 1.3.3. REVIEW OF AIRPLANE DESIGN AND DESIGN PROCESSES The CSRT reviewed the B787‒8 airplane design and the processes used to create the airplane as follows: 1. From February 25, 2013, to March 1, 2013, Boeing presented briefings on the overall design of B787‒8 critical systems. Major topics of the briefings included the following: o An airplane design and manufacturing overview, o Design for safety, o The B787 supply chain, 7 A build verification test is a test performed to verify the component being produced can be passed to the next build stage. Boeing 787‒8 Critical Systems Review Team Report Page 4 o Boeing quality management systems, o Supplier quality management, o The B787 production system, o A summary of the B787 type certification program, o B787 reliability data since entry into service (EIS), o Boeing in-service safety processes, o Condition of assembly (level of completeness of major assemblies on arrival at the final assembly location), o Case studies of specific technical and/or production issues, o Boeing “gated” design and production processes, 8 and o The functional integration process. 2. The CSRT received detailed briefings on specific aspects of airplane design and/or production for those issues. In some cases, the CSRT traveled to the supplier’s location (see table 1) to receive detailed briefings and conduct in-depth reviews of the design and manufacture of the respective equipment, systems, or structural components. Details of the issues and systems reviewed are provided in appendix A. Table 1. CSRT Supplier Visits Supplier Location Manufacturing Engineering Component Alenia Aermacchi SPA Grottaglie, Italy X Mid-Body Fuselage Sections 44/46 Alenia Aermacchi SPA Foggia, Italy X Horizontal Stabilizer Boeing Everett, WA Boeing–Final Assembly Everett, WA Boeing/ Moog Inc Everett, WA Boeing South Carolina North Charleston, SC X X X Electromagnetic Effects/ Fuel Coupling X X Power Panel Electro-Hydraulic Servo Valves Aft Fuselage Sections 47/48 and Final Assembly, Interiors Responsibility Center 8 In this process, a series of programmatic “gates” are established at various points during the development program. Each gate has specific criteria for proceeding to the next development phase. Any criteria that have not been satisfied at a given gate must be addressed or mitigated before proceeding to the next phase. Boeing 787‒8 Critical Systems Review Team Report Page 5 Supplier Ducommun LaBarge Technologies GSE Industria Aeronautica SRL Location Manufacturing Phoenix, AZ Brindisi, Italy V La Gatta SRL Pomigliano, Italy Moog Inc North Charleston, SC Component X Wire Harnesses for B787 Auxiliary Power Unit X Electronics and Equipment Racks, Cradles, Composite Clips (787–8 Section 46) X Cargo Door Surround and Passenger Door Surround (787–8 Section 46) Salt Lake City, UT New Breed Logistics Inc Engineering X Spoiler Electric Actuator X Third Party Inventory Management Rexnord Corp Wheeling, IL X Variable Frequency Starter Generator (VFSG) Seal Sicamb SPA Latina, Italy X Auxiliary Spar Box (787–8 Horizontal Stabilizer) United Technologies Aerospace Systems, UTAS (formerly HamiltonSundstrand Corp). Rockford, IL X UTAS Rockford, IL UTAS Phoenix, AZ Boeing 787‒8 Critical Systems Review Team Report High-Power VFSG X X X Generator Control Unit (GCU) GCU Page 6 2. CSRT REVIEW ANALYSIS—KEY MESSAGES 2.1. INTRODUCTION The CSRT analyzed its deep-dive reports and noted several similar issues surfaced across the four disciplines reviewed. The deep-dive reports also identified some unique issues warranting further discussion. The CSRT grouped those topics and presents them in this chapter as the key messages stemming from its review. The key messages document the following: • The B787’s EIS reliability performance as compared to other Boeing airplane models; • The impact of— o Component removals on airplane safety, o Novel technologies on in-service issues, and o The B787 business model on in-service issues; • Issues associated with design requirements; • FAA oversight, supplier oversight, conformity, and inspection delegation issues; and • Whether the B787 meets its intended level of safety (meaning the B787, as a result of the combined efforts of Boeing and the FAA in the certification and post-certification processes, meets the level of safety intended by Boeing, the FAA, and the flying public). 2.2. B787 ENTRY INTO SERVICE—COMPARISON TO OTHER BOEING AIRPLANE MODELS Operational reliability data can denote whether there are systemic failures in design requirements, assembly, or manufacturing, thereby indicating an airplane’s intended level of safety. The CSRT analyzed operational reliability data for the B787 from its EIS operations, specifically data on schedule reliability, ETOPS, and certain required regulatory reports. The CSRT compared this data with similar data for other Boeing airplane models to determine whether the B787 meets or exceeds the performance standards of previous Boeing models. The available data the CSRT reviewed shows the B787 operational reliability during its EIS operations is very similar to the comparable data for previous Boeing airplane models. Boeing 787‒8 Critical Systems Review Team Report Page 7 2.2.1. DATA ANALYSIS Figure 1. Schedule Reliability SCHEDULE RELIABILITY The CSRT reviewed comparable data between the EIS performance of the B787 and the Boeing 777 (B777), as well as EIS performance data from other Boeing airplanes. The data reviewed included schedule reliability, a direct measure of the airplane’s ability to successfully complete each assigned flight segment on schedule. Figure 1 compares the B787 EIS data with equivalent data from the EIS of previous Boeing airplane models. The data shows that except for the Boeing 747‒400 EIS, the initial EIS schedule reliability performance of Boeing airplanes has consistently been higher than 96 percent. As the data demonstrates, the B787 is following this trend. Boeing 787‒8 Critical Systems Review Team Report Page 8 EXTENDED OPERATIONS Because the B787 is designed for long-range service, another measure of operational reliability is the airplane’s in-service performance against ETOPS criteria. EE–1s (reportable events agreed to by the FAA and Boeing), or ETOPS events, are airplane maintenance issues specifically related to airplane requirements and capability for ETOPS. Because these systems include the airplane’s electrical system (a critical system under review), the CSRT compared the B787’s EE–1 performance with the B777’s performance (which was proposed as a standard to achieve when measuring EIS and other performance indicators). Figure 2. B787 vs. B777 EE–1s Since Entry Into Revenue Service 9 After 15 months, there were 49 B787s and 37 B777s in service. The data in figure 2 illustrates that the B787’s EE–1 performance has been consistently equal to or better than the B777’s performance during the same period after EIS (fewer EE–1s), despite there being more B787 airplanes in service. 9 Federal Aviation Regulation (FAR) 21.3 is a Boeing reference to 14 CFR § 21.3. Boeing 787‒8 Critical Systems Review Team Report Page 9 14 CFR § 21.3 REPORTING As part of COS, all type certificate holders, including Boeing, are required by § 21.3 to report to the FAA any failure, malfunction, or defect in any product, part, process, or article they manufacture that they determine has resulted in occurrences listed in § 21.3(c). Figure 3 illustrates that the B787’s § 21.3 EIS performance has been consistently equal to or better than the B777’s § 21.3 EIS performance. 777 & 787 FAR 21.3 Reporting During Entry Into Service Figure 3. 14 CFR § 21.3 EIS Reporting 2.2.2. SUMMARY Based on its analysis of the schedule, ETOPS reliability data, and § 21.3 reporting data, the CSRT determined the B787 EIS reliability performance is comparable to that of other Boeing models. The CSRT used the B787’s positive reliability record as one indicator of whether the B787 meets its intended level of safety. Boeing 787‒8 Critical Systems Review Team Report Page 10 2.3. COMPONENT REMOVALS The CSRT reviewed in-service data on returned components to determine the cause of critical system component failures and their effect on the airplane’s safety. The CSRT noted the potential exists on any airplane for in-service events involving failures that create a maintenance cost/burden but do not compromise safety or regulatory compliance. For example, the airplane’s onboard systems may detect an equipment failure, which might generate a message for maintenance (and/or the pilot) at airplane startup indicating the system must be fixed before dispatching the airplane. The primary effect of these types of failures is economic—the flight may be delayed or cancelled while corrective maintenance is performed. When such failures must be resolved before airplane dispatch, operators will often remove and replace all potential sources of the failure so the airplane can be quickly returned to service. 10 This practice is conservative from a safety perspective and minimizes airline schedule disruptions. However, based on detailed explanations from Boeing and system suppliers, the CSRT learned that when many of the allegedly failed items were checked, no fault was found and the removal of all possible sources of a failure inflated the failure rate data for specific components. In some cases, the CSRT determined the in-service return data for several systems it reviewed was significantly inflated by this maintenance practice. Therefore, the CSRT concluded that not all part removals provide direct insight into part reliability, and that sometimes the removal can be a precautionary measure taken to ensure a timely return to service. 2.4. NOVEL TECHNOLOGIES The B787 employs many new technologies and innovative designs and architectures. The CSRT assessed whether technological innovations contributed to the in-service issues reviewed and ultimately determined novel technologies were not the cause of in-service issues. SIGNIFICANT TECHNOLOGICAL INNOVATIONS The CSRT selected several significant technological innovations on the B787 for further review. These included— • VFSGs, • The high-power electrical system, 10 In highly integrated and complex systems, it is not always feasible to quickly narrow down a specific component failure using on-airplane diagnostic systems. For example, when there is a failure of a VSFG, the fault could be in the VSFG or in the GCU that manages it. In some cases, the onboard diagnostic systems cannot make a reliable component-level failure determination, so operators chose to replace both the VFSG and GCU rather than delay the flight to perform component-level troubleshooting. The airline then returns both components to the supplier for evaluation and/or repair. Boeing 787‒8 Critical Systems Review Team Report Page 11 • Electrically powered spoiler actuators, and • Composite fuselage manufacturing. See appendix A for a detailed discussion of these areas. In each case, the CSRT determined that although the technology was novel, novelty did not cause the in-service issues that triggered the events and the associated challenges discovered during the deep-dive reviews. For example, the CSRT noted one case of internal short-circuiting on an electrical panel’s printed circuit boards. Although a design standard to prevent the short-circuiting existed, the issue arose because the design standard was not followed. New technology did not cause the problem in this example; the cause was improper implementation of established design requirements (see further discussion of this specific issue in section 2.6 of this report under Industry Design Standards). The CSRT observed the B787’s use of composites for the primary fuselage and wing structures is unprecedented, but it noted the novel use of composites was not the source of the manufacturing issues it reviewed. Therefore, using the above examples and other similar instances from the deep-dive process, the CSRT concluded that novel technology has not significantly contributed to the B787 in-service issues that prompted this review or compromised the safety of the airplane. 2.5. BUSINESS MODEL Boeing’s business model for aircraft design and production shares design and production responsibility with numerous suppliers. A perception occasionally associated with the B787 is that subcontracting various parts of the airplane to other companies resulted in a deficient or subpar product. The CSRT found that for B787 systems and equipment, the business model did not cause the in-service issues reviewed. However, the business model presented some unique challenges for the manufacturing/quality area and the FAA. 2.5.1. B787 SYSTEMS AND EQUIPMENT The CSRT noted Boeing gave suppliers a larger role in developing, testing, and certifying integrated systems for the B787. The CSRT assessed integrated systems components and found no evidence the B787 business model directly contributed to the underlying causes of the in-service issues. Instead, the causes were typically linked to basic communication and coordination issues encountered with any large and complex new airplane development program. Additionally, the CSRT found that although there were individual issues with parts of certain B787 systems, the systems were found to have robust architecture and adequate redundancy, which enabled them to continue functioning safely following component failures. Boeing 787‒8 Critical Systems Review Team Report Page 12 2.5.2. MANUFACTURING/QUALITY The CSRT found the business model presented challenges for manufacturing/quality. The B787 introduced novel manufacturing and assembly processes to the manufacturing environment that were unprecedented in scope and scale. The CSRT observed that Boeing’s implementation of new manufacturing and assembly processes during the B787’s design and production created unanticipated manufacturing challenges. During the B787’s development, the supply chain faced a learning curve as suppliers discovered how to work with interfaces for the new processes. In addition, late engineering changes during the B787 build process affected the suppliers’ ability to meet Boeing’s specified production rates for components. The CSRT observed that without increased support from Boeing, some suppliers had difficulty meeting their schedule commitments and integrating the late engineering changes. This resulted in a high amount of “travelled work” (incomplete airplane elements shipped to Boeing for final assembly), part shortages, and nonconformances in the initial build phase. The CSRT noted that Boeing’s production system and supplier management have improved significantly over the past 2 years. The CSRT observed that Boeing has invested considerable resources to align supplier performance with necessary expectations to support the B787, and continues to have a major presence at select supplier sites. The CSRT determined that production along the entire supply chain is maturing. This is evidenced by the reduced number of interventions, negligible changes to the current processes, and higher quality of output coupled with an increased production rate. LESSONS LEARNED • Boeing’s B787 program was not initially set up to manage unanticipated challenges from suppliers unfamiliar with the new manufacturing environment. Additionally, Boeing did not intervene early enough in the process to assist the suppliers. However, Boeing has since increased supply chain support to assist struggling suppliers, and this has mitigated many supply chain issues. • The CSRT noted that at the start of large, complex airplane development programs, production certificate holders and suppliers should ensure controls are in place for critical process completion based on product and/or process risk assessment at every level of the supply chain. A closed-loop system should minimize misinterpretation and gaps at the first tier of the supply chain, creating a reliable flow of information to lower tier suppliers. Boeing 787‒8 Critical Systems Review Team Report Page 13 Recommendation No. 1: Boeing should establish a means to ensure suppliers identify realistic program risks and complementary mitigation plans through a closed-loop flowdown validation of requirements. 11 (Also see Recommendation No. 2 regarding allocation of sufficient resources.) 2.5.3. FAA CERTIFICATE MANAGEMENT OF LARGE PRODUCTION APPROVAL HOLDERS Boeing’s B787 business model altered the traditional Boeing supply chain and set a new precedent from a manufacturing and design perspective as well as an FAA regulatory oversight perspective. The FAA had not previously overseen such a large, complex supply chain with multiple subtier suppliers. The CSRT observed that FAA manufacturing certificate management policy 12 does not align with the current B787 supply chain environment, nor will it adequately accommodate future aircraft manufacturing surveillance using alternative business models (such as Boeing’s business model). The CSRT observed that current FAA certificate management policy lacks the flexibility to adequately focus resources in a standardized fashion to new areas of inherent risk in business models unknown or unfamiliar to FAA Certificate Management Offices (CMO). This observation is substantiated using Boeing’s business model, which elevates first-tier suppliers to a new level of responsibility for component design and moves the actual component manufacturing farther down the supply chain to lower tier suppliers. This creates new oversight challenges, thus increasing the need for FAA resource management flexibility. Observation No. 1: The FAA policy regarding certificate management of production approval holders addresses only risk at the manufacturer’s top system level. The policy does not require development of tailored certificate management plans that specifically identify and target risk wherever responsibilities and expectations exist within complex supply chains. FAA policy also does not encourage FAA manufacturing oversight offices to conduct surveillance at critical subtier suppliers when first-tier suppliers are major integrators, even though they may not manufacture a significant portion of the assembly. 11 Closed-loop corrective action is a process in which suppliers define and describe the problem to be solved, identify causes, test and validate solutions, implement the solutions, sustain the solutions, and monitor results to ensure that the solutions yield the intended improvements. 12 FAA Order 8120.23, Certificate Management of Production and Approval Holders. Boeing 787‒8 Critical Systems Review Team Report Page 14 2.6. DESIGN REQUIREMENTS The CSRT found few issues related to the B787 design requirements and identified the causes of these issues as unique. Throughout this review, the CSRT also found continuous improvement processes in use as part of established procedures intended to advance and improve both airplane parts and systems. As noted in appendix A, the CSRT found no fundamental weaknesses in the overall design process or in the general methods used to develop design requirements. 2.6.1. BACKGROUND Requirements development is an inherently challenging task for large, complex, safety-critical systems such as transport airplanes. For B787 requirements development, tens of thousands of requirements are applied to millions of parts. In addition, every part, component, and system is subject to multiple requirements. The CSRT noted Boeing uses requirements development processes that meet or exceed industry standards. Boeing has processes in place to manage and confirm the requirements by tracking and validating them, then verifying the design meets each one. Although Boeing has a B787 requirements development process in place, the CSRT noted this process does not guarantee against deficiencies in requirements (which may result in design issues). A robust requirements development process provides sufficient verification and validation so risk is managed appropriately and requirements issues can be found and addressed. 2.6.2. REQUIREMENTS ISSUES The CSRT reviewed in-service issues traceable to requirements deficiencies that fell into the following categories: (1) requirements flowdown, (2) responsibility, (3) industry design standards, (4) new application of existing technology, and (5) design review process. REQUIREMENTS FLOWDOWN The CSRT identified inconsistencies in design requirements flowdown 13 and design verification in multiple deep-dive reviews. For example, the CSRT deep-dive reviews revealed cases in which Boeing’s design requirements did not flow down to its primary supplier and then to the involved subtier suppliers. Inconsistency in parts manufacturing, part failures, and operational disruptions (such as turn backs and diversions) could be traced to inadequate verification and/or validation of the established Boeing design requirements. 13 Flowdown is the movement of information down Boeing’s supply chain to lower tier suppliers. Boeing 787‒8 Critical Systems Review Team Report Page 15 RESPONSIBILITY In some cases the CSRT examined, Boeing or its major suppliers with integration responsibilities did not clearly establish which subtier supplier providing components for an integrated system was responsible for the specific detailed design requirement. This resulted in communication and verification issues along the supply chain. INDUSTRY DESIGN STANDARDS The CSRT observed that some suppliers did not follow aerospace industry design standards 14 if Boeing had not established a specific design requirement. In the case of internal short-circuiting on an electrical panel’s printed circuit boards, the CSRT observed the supplier identified a specific item as an optional design aspect and did not use industry design standards when fulfilling its obligation to design this part. Although the supplier assumed it successfully met requirements, the intent had not been satisfied. NEW APPLICATION OF EXISTING TECHNOLOGY The CSRT noted previous experience with similar designs or past engineering practices led to incorrect assumptions about how the systems would perform, and this led to inadequate design requirements for these components. The CSRT noted detailed requirements are driven by how a system is expected to perform, either in normal operations or in failure conditions. In some cases, that expectation is based on past experience with similar designs. Because new designs are rarely identical to previous designs, engineering judgment is needed to determine when the new system can be assumed to perform similarly to previous designs. In addition, the CSRT determined it would be impractical to reverify all previous design practices for every design detail. The CSRT also found incorrect assumptions that proven design solutions would apply to new and/or novel design applications without validation of those assumptions or consideration of their context. In some cases, the fact that the design feature was “proven” led to a conclusion that failures of those components were well understood, fully mitigated, and represented a low design risk. This led to decisions that those failures were not critical cases needing testing, resulting in the design weakness not being found during the test program. Instead, some of the design weaknesses were found when the airplane entered commercial service. 14 Standard aerospace industry design standards are generally accepted as the minimum design standards for aircraft components. Boeing 787‒8 Critical Systems Review Team Report Page 16 DESIGN REVIEW PROCESS The CSRT deep-dive reviews included an examination of several design errors resulting from the established design review process not being followed. For example, a design feature made it impossible to install a non-impact-resistant fuel tank access door in locations where an impact-resistant door was required. During subsequent improvements to the door design, a poorly executed design change process allowed deletion of that design feature, thereby making the doors interchangeable. Component design reviews should have found design errors such as this and identified instances in which subtier suppliers did not follow standard aerospace industry design practices. Lessons Learned • When design requirements cross organizational or design boundaries, Boeing needs to establish the supplier responsible for meeting the requirements to ensure verification and validation of the requirement is appropriately assessed and documented. • Emphasis must be placed on requirements clarity and verification throughout Boeing’s supply chain. Based on the above discussion, the CSRT made the following recommendations: Recommendation No. 2: Boeing should continue to implement and mature the gated design and production processes with sufficient resources for development programs, and to minimize risks throughout the life cycle of the program. In these processes, a series of programmatic “gates” are established at various points during the development program. Each gate has specific criteria for proceeding to the next development phase. Any criteria that have not been satisfied at a given gate must be addressed or mitigated before proceeding to the next phase. (Boeing is realizing improved performance in the Boeing 737 MAX, Boeing 787‒9, and Boeing 767‒2C programs from using a gated approach.) Recommendation No. 3: Boeing should ensure suppliers are fully aware of their responsibilities, including integration responsibilities and accountability for subtier performance. The gated design processes should include supplier planning, performance, and reporting, using measurable and appropriate performance criteria that include the scope and effectiveness of design reviews and other airplane life-cycle activities. 2.7. INSPECTION DELEGATION The CSRT noted one issue with inspection delegation 15 during the B787 build process. Many of Boeing’s suppliers have unique inspection delegation processes, and some of 15 Inspection delegation involves a supplier giving inspection responsibility to lower tier suppliers in the supply chain. Boeing 787‒8 Critical Systems Review Team Report Page 17 these supplier’s processes do not meet minimum industry standards. The FAA does not require suppliers to follow industry standards for internal inspections; however, Boeing assumed the suppliers followed industry standards. Note: This discussion refers to Boeing’s internally driven inspections, not delegated FAA inspections. FAA orders establish qualification requirements for FAA designees performing FAA inspections. The CSRT observed that some manufacturer-delegated employees are certified and recertified to ensure they are competent to perform critical inspections. In addition, many companies provide continued training and alerts through Web-based systems, allowing the manufacturer-delegated employees to be current on the requirements, specifications, and changes occurring to the products under their responsibility to inspect. The CSRT noted that industry standards (such as the G–14 Americas Aerospace Quality Standards Committee AS9015 Supplier Self Verification Process—Delegation Programs document) exist as an aviation best practice for manufacturer inspection delegation that includes training, testing, and currency requirements. Recommendation No. 4: Boeing should require its suppliers to follow industry standards for the training, qualification, and certification of supplier personnel performing Boeing-required (non-FAA) inspections. 2.8. REGULATORY OVERSIGHT—FAA REVIEW AND ACCEPTANCE POLICY The CSRT observed that the Boeing FAA CMO developed a detailed preproduction certificate B787 validation plan that included quality management system compliance review, facility review, and targeted conformity inspections. In making this observation, the CSRT found current FAA policy 16 on acceptance of an aircraft manufacturer’s production capability is outdated because it treats all production facilities the same. This generic FAA policy fosters significant inconsistency between the multiple FAA offices tasked with production certification. Currently, FAA manufacturing offices can approve small, less complex aircraft for production using the same methodology they use for large, complex transport category aircraft manufacturers that work with international supply chains. The CSRT noted the CMO has augmented this policy by independently increasing its exposure to new and novel technologies, manufacturing processes, and quality management system procedures. However, a risk management approach has not been incorporated into FAA orders so that some of the enhanced practices used on the B787 program are institutionalized and employed on future production certificate projects. 16 FAA Order 8120.22, Production Approval Procedures. Boeing 787‒8 Critical Systems Review Team Report Page 18 Observation No. 2: The FAA policy regarding production approval procedures does not recognize the differing levels of complexity of manufacturing systems and technologies between small, relatively simple aircraft manufacturers and large-scale, complex aircraft manufacturers with extended supply chains. The production approval process does not focus on aircraft complexity and critical technologies (both innovative and existing) using a comprehensive risk-based plan. 2.9. FAA ENGINEERING CONFORMITY The CSRT reviewed Boeing data showing the FAA (or the Boeing ODA) performed substantially more engineering FAA conformity 17 inspections for the B787 program than for the B777 certification program. The following FAA orders contain guidance on conformity inspections: • FAA Order 8110.4C, Type Certification, effective during the B777 certification program, provides generally applicable guidance regarding when conformity inspections are needed, including programs involving individual designees. • FAA Order 8100.9A, DAS, DOA, and SFAR 36 Authorization Procedures 18, (now cancelled) provided information on FAA conformities for delegation option authorizations (DOA). • FAA Order 8100.15B, Organization Designation Authorization Procedures, provides information on when FAA conformities are needed for projects managed by an ODA. The CSRT discussed the guidance in these orders with Boeing, the Boeing Aviation Safety Oversight Office, and the FAA Engineering Division, and determined there were differing interpretations of the guidance. The CSRT found that although total applicant engineering conformity is necessary for all tests and inspections to show compliance, FAA guidance is not clear and consistent on the requirements for FAA engineering conformity inspections (which may be conducted by the FAA or delegated to the ODA). Some language has been interpreted to mean that FAA conformity inspections must be conducted on all test articles, while other guidance language suggests the FAA (or the delegated organization) may determine which of the test articles must have an FAA conformity inspection. The CSRT concluded these differing interpretations may have led to a perceived need for more FAA conformity inspections on the B787 than were required on the B777. 17 Conformity means an aspect of the manufactured product matches the engineering data, including (1) the physical aspects of the design, (2) the processes by which the components were constructed, and (3) the installed software. Engineering conformity inspections intend to document/verify the configuration of certain articles undergoing compliance inspections or testing matches the design data. 18 DAS stands for Designated Alteration Station. SFAR stands for Special Federal Aviation Regulation. Boeing 787‒8 Critical Systems Review Team Report Page 19 As a further complication, the B787 program initially used individual designees (similar to the B777 program) to perform authorized functions on behalf of the FAA, converted to an organizational delegation (delegation option authorization), then transitioned to the Boeing ODA, the current form of organizational delegation. The CSRT also noted the use of novel technologies, design processes, and manufacturing processes on the B787 introduced additional risks regarding conformity of test articles, which may have contributed to an increase in the number of FAA engineering conformities. The CSRT was unable to determine which of the noted factors caused the increased number of required FAA conformity inspections on the B787, but concluded the FAA orders are unclear on this issue for FAA offices and delegated organizations. The CSRT noted that requiring FAA engineering conformity inspections for all projects (including ODA projects) based on safety risk would still require FAA (or ODA) approval of the risk-based conformity plan. Observation No. 3: FAA ODA policy does not provide adequate guidance to ensure risk-based conformity inspection plans. Boeing 787‒8 Critical Systems Review Team Report Page 20 3. CONCLUSIONS The CSRT’s critical systems review showed the B787 met expectations in most areas. Additionally, the CSRT found that although there were individual issues with parts of certain B787 systems, the systems were found to have robust architecture and adequate redundancy, which enabled them to continue functioning safely following component failures. However, it also concluded a few areas of the B787 program need improvement, as discussed below. The CSRT highlighted that practices and/or processes governing the B787 program ensure the safety of the airplane. To address the unique issues reviewed, the CSRT made four recommendations to Boeing and three observations on FAA policy and guidance issues, as presented in chapter 2 of this report. Timely implementation of these recommendations and FAA consideration of its observations, in addition to the continuing application of Boeing process improvements and the established COS processes, serves to— • Ensure the B787 continues to meet its intended level of safety. • Improve performance (in-service reliability). • Reduce risk. • Help reduce the occurrence of similar future in-service events for the B787 program and future airplane programs. In summary, the CSRT interpreted its task to “ensure that the aircraft meets its intended level of safety” broadly to mean that the B787, as a result of the combined efforts of Boeing and the FAA in the certification and post-certification processes, meets the level of safety intended by Boeing, the FAA, and the flying public. Despite the issues identified in this report, the CSRT concluded that the B787 meets its intended level of safety. The CSRT reached this conclusion based on (1) the fundamental soundness of the airplane’s overall design and (2) the effective processes that have been defined and implemented to correct issues that arose during and after certification. 3.1. B787 MET EXPECTATIONS IN SERVICE The CSRT determined the B787 is successfully using COS processes to ensure the airplane continues to meet its intended level of safety based on the following: • A review of in-service schedule reliability data, ETOPS systems event reports, and reports of certain occurrences Boeing must submit to the FAA under 14 CFR § 21.3(c) showed the B787 EIS experience is equivalent to or better than Boeing’s previous new model airplanes. • Because the B787 employs many new technologies and innovative designs and architectures, consideration was given to whether technological innovations contributed to in-service issues reviewed. Novel technologies did not inherently cause the in-service issues studied. Boeing 787‒8 Critical Systems Review Team Report Page 21 • Because of system redundancies and other architecture features, most in-service events involved failures that did not compromise safety or regulatory compliance. In many cases where the airline removed a component, the manufacturer’s inspection found there was no fault. • The business model used for engineering and manufacturing B787 systems did not significantly contribute to the underlying cause of the in-service issues. In-service issues were typical of those found during the introduction of any new transport airplane. 3.2. IMPROVEMENT NEEDED The CSRT determined that as with any multi-layered process, there should be a continual process review and enhancement of the B787 program. Although the fundamental processes worked as planned, improvements are needed in the following areas; in some cases, Boeing has already taken steps to implement these improvements. 3.2.1. MANUFACTURING/QUALITY BUSINESS MODEL Early on, suppliers experienced difficulty learning to work with the new business processes, especially in handling late engineering changes from Boeing. Boeing has increased supply chain support to mitigate supply chain issues, and supplier management has improved significantly. The CSRT recommends manufacturers ensure suppliers use a closed-loop system to capture realistic program risks and complementary mitigation plans. 3.2.2. DESIGN REQUIREMENTS Although the CSRT did not find issues with the general methods used to develop design requirements, it noted the following issues with identified causes: • Communication of the requirements (flowdown), • Ownership of the requirements, • SMEs following the established design review process, and • Inadequate design requirements because of incorrect assumptions about how systems would perform based on previous experience with a similar design. The CSRT lessons learned for this area noted emphasis must be placed on requirements clarity and verification throughout the supply chain. In addition, the owner of the design requirement needs to be clear. The CSRT recommends that Boeing implement and advance its gated design and production processes and clarify supplier accountability. Boeing 787‒8 Critical Systems Review Team Report Page 22 3.2.3. INSPECTION DELEGATION The CSRT found inspection delegation in widespread use among the suppliers. The CSRT noted many of Boeing’s suppliers have unique inspection delegation processes, some of which do not meet minimum industry standards. The CSRT recommends that Boeing require all suppliers to follow industry standards for inspection delegation including certification and recertification. 3.2.4. FAA REGULATORY OVERSIGHT The CSRT observed that several FAA orders do not align with current practices. Specifically, the FAA orders do not— • Encourage surveillance at critical subtier suppliers and require risk management models to allow assigning risk and surveillance requirements at integrator tier suppliers. • Recognize the differing levels of complexity of aircraft manufacturing systems and technologies (small, relatively simple aircraft manufacturers versus large-scale, complex aircraft manufacturers with extended supply chains.) • Describe a production approval process that focuses on aircraft complexity and critical technologies using a comprehensive risk-based plan. • Establish engineering conformity based on a risk-based conformity inspection plan. Boeing 787‒8 Critical Systems Review Team Report Page 23 4. FAA RECOMMENDATIONS Based on the CSRT’s observations of FAA oversight policy and guidance issues in chapter 2 of this report, the FAA CSRT team members recommend the following changes be made to FAA policy and guidance documents. • FAA Recommendation No. 1: The FAA should revise chapters 3 and 4 of FAA Order 8120.23, Certificate Management of Production Approval Holders, to recognize new aircraft manufacturing business models and their potential impact on safety, complexity, risk, and mitigating actions. • FAA Recommendation No. 2: The FAA should revise chapter 3 of FAA Order 8120.22, Production Approval Procedures, to recognize the changing aircraft manufacturing environment and to more fully address complex, large-scale aircraft manufacturers with extended supply chains, expectations, and production capabilities. • FAA Recommendation No. 3: The FAA should revise FAA Order 8110.4C, Type Certification, and FAA Order 8100.15B, Organization Designation Authorization Procedures, to provide clear and consistent guidance to ensure FAA engineering conformity inspections for all projects (including ODA projects) are based on risk. The orders should require FAA (or ODA) approval of the risk-based conformity plan. Boeing 787‒8 Critical Systems Review Team Report Page 24 APPENDIX A—DEEP-DIVE REVIEW SUMMARIES A.1. INTRODUCTION The Critical Systems Review Team (CSRT) divided into four subteams to organize and perform its deep-dive reviews of critical systems, structures, and manufacturing. The Systems, Structures, Propulsion, and Manufacturing/Quality Subteams selected each component, assembly, or process for deep-dive review. Each subteam examined its assigned deep-dive areas to evaluate the causes of the identified issues, review corrective actions already taken, identify potential gaps, and make recommendations for future actions. The Systems, Structures, and Propulsion subteams forwarded issues not identified as design issues to the Manufacturing/Quality Subteam for review and disposition. Each subteam also factored the level of system complexity, level of supplier responsibilities, application of new technology, and novel applications of existing technology into its selection decision. The figure below depicts the areas of the Boeing 787 (B787) selected for review. The specific items reviewed are shown in black text. Figure A–1. B787 Components/Systems Selected for Deep-Dive Review Boeing 787–8 Critical Systems Review Team Report Page A–1 This chapter contains a summary of common themes identified during the subteams’ individual deep-dive reviews, or in some cases, a discussion of issues that emerged during a subteam’s deep-dive review. The components and subsystems reviewed, selection process, discussion topics, observations, recommendations (if provided), and conclusions are presented for each subteam (Systems, Propulsion, Structures, and Manufacturing/Quality). A.2. SYSTEMS The Systems Subteam selected the following components and subsystems for in-depth review: • Variable frequency starter generators (VFSG). o • Generator control units (GCU). o • The primary power panels house the engine generator GCUs and contactors used to control and distribute the power from the engine VFSGs to the rest of the airplane. There are two primary power panels located in the aft electronics bay, each receiving power from the two generators on each engine. Spoiler electromechanical actuators (SEMA). o • There are six GCUs on the B787, one for each generator (four main engine generators and two auxiliary power unit generators). Each GCU controls power from its respective generator and reconfigures the power system to ensure airplane capability is maintained if an engine or generator fails. Each GCU also provides voltage regulation and protection of the respective generator. Primary power panels. o • There are four VFSGs on the B787, two mounted on each main engine and connected through a gearbox. The VFSGs perform two primary functions: electric starting of the main engines and, once the engine is started, providing electric power to the airplane. The frequency of the VFSG alternating current output varies with the speed of the engine. SEMAs control two of the seven spoiler pairs on the wing surfaces and provide roll control, air/ground speedbrake, and droop capabilities similar to the hydraulic actuators used on the remaining spoiler surfaces. Primary flight control system hydraulic actuators. o Hydraulic actuators are used to position all of the primary airplane control surfaces (ailerons, flaperons, elevators, and rudders). VFSG, GCU, and primary power panels. The VFSG, GCU, and primary power panels are all part of the airplane electrical power generation and start system (EPGSS) as illustrated in figure A‒2 below. These components comprise the equipment necessary to generate and distribute the variable frequency electrical power used on the airplane. Boeing 787–8 Critical Systems Review Team Report Page A–2 Figure A–2. Electric Power Generation & Start System 19 SEMA. Figure A‒3 shows a SEMA installed at the wing rear spar and also shows the associated electronic motor control unit (EMCU) that controls the operation of the SEMA. This is the first use of an electromechanical actuator on the primary flight control surface of a production civil transport airplane or military aircraft. 19 SG stands for starter generator. QAD stands for quick attach/detach. CEI stands for Common Electronics Initiative. Boeing 787–8 Critical Systems Review Team Report Page A–3 Wing Rear Spar SEMA EMCU Figure A–3. Spoiler Electromechanical Actuators Primary flight control system hydraulic actuators. The hydraulic actuators control the position of the primary control surface actuators in response to commands from the flight control electronics and remote electronic units (REU). Figure A–4 provides an illustration of the elevator actuator, which is typical of all the primary hydraulic actuators. Figure A–4. Elevator Actuator Boeing 787–8 Critical Systems Review Team Report Page A–4 A.2.1. SELECTION PROCESS SUMMARY The subteam selected the components/subsystems listed previously because proper operation at the airplane level is critical. In addition, these components/subsystems— • Experienced multiple in-service events that were being tracked by Boeing’s continued operational safety (COS) process. • Were reportable under Extended Operations (ETOPS) reporting requirements. • Were leading contributors to service interruption/reliability statistics. A.2.2. OBSERVATIONS SUMMARY The following sections provide observations grouped by requirements, design, test, analysis, and in-service issues that are based on the Systems Subteam’s deep-dive reviews. Observations that do not have a common theme are not discussed in this section. During its review, the Systems Subteam also noted an issue with the number of engineering conformity inspections on the B787. The subteam’s discussion of this issue and its observation are at the end of section A.2.2. REQUIREMENTS Generating requirements is an extremely large and challenging task. There are tens of thousands of requirements for an airplane, and these requirements are levied against millions of components and systems. Several instances of deficiencies in requirements definition were identified during the review. These deficiencies, when translated through the design and implementation of the component, led to the issues identified during the investigation of the in-service issue. Although the cause of the requirements deficiencies identified during this review were related to one another at a high level, the subteam determined that at a detailed level each one was unique and isolated. The following topics characterize the requirements issues the subteam identified. Topic 1: Requirements flowdown. There were instances in which requirements flowed down from Boeing to a primary supplier and then to subtier suppliers. An example is the primary electrical power panel requirements that flowed down from Boeing to United Technologies Aerospace Systems (UTAS), then from UTAS to its subtier supplier Equipment et Construction Electrique (ECE), and then on to the printed circuit board component suppliers. The application and verification of the requirement was not adequately defined as the requirement passed through each organization. This resulted in (1) designs that were shown to be deficient in various aspects once the part entered service, (2) variability in the manufacturing, or (3) operation of the part leading to anomalous behavior or failures. Boeing 787–8 Critical Systems Review Team Report Page A–5 Observation: Even though Boeing requirements existed, verification/validation of the requirements did not always occur between Boeing and/or subtier suppliers. Topic 2: Requirements assumptions. There were instances in which a requirement was assumed to be correct based on past engineering practice or knowledge. The assumption was that because the requirement was sufficient in a past application, it remained a valid requirement for the current application. Due to either changes in the environment (for example, the change in operating environment for the SEMA motor resolver) or application of the device, an incorrect assumption was made about how the device would perform, which was then shown to be incorrect once the part had sufficient in-service use. Observation: In some cases, previous experience with similar designs led to assumptions that were not validated. Topic 3: Requirements ownership. There were instances in which requirements flowed down from Boeing to a primary supplier and then to subtier suppliers, or flowed across and between two Boeing disciplines to their respective primary suppliers. In these cases, the owner/verifier was not explicitly defined, resulting in each organization assuming the other was the owner. An example of this situation was the design of the VFSG air-oil heat exchanger circuit; different parts of the circuit were owned by different suppliers with no clear ownership of the integrated assembly. When requirements cross either organizational or design boundaries, the owner of the requirement needs to be established to ensure verification and validation of the requirement is appropriately assessed and documented. Observation: With multiple suppliers involved, either in the same supply chain or across multiple platforms/commodities, the lack of a defined owner resulted in a requirement not being adequately communicated and/or verified. DESIGN Topic 1: Incorrect assumption that proven design solutions would be equally applicable to new and/or novel design applications without validation of those assumptions or their context. In some cases, incorrect design assumptions resulted in functional or performance shortcomings in components or systems that were not discovered until later in the airplane development life cycle. These late discoveries resulted in airplane development Boeing 787–8 Critical Systems Review Team Report Page A–6 schedule disruptions. Examples include the variable oil pressure in the VFSG oil cooling system and its effects on the dynamics of that system, and the contribution of variable frequency to the effects of failure modes of the VFSG rotating diodes on the GCU transorbs. Although these issues may disrupt schedules and create burdens for airlines, they do not individually represent a safety risk. Observation: Indications that incorrect assumptions survived peer and subject matter expert (SME) design review demonstrate that additional oversight and diligence in following formal design and design review processes, including the use of design checklists, is required. Topic 2: Appropriate industry design standards not followed. In the absence of specific design requirements called out by Boeing, the CSRT determined that in certain cases, the subtier suppliers did not follow their own or industry design standards. Industry design standards provide a means for documenting the collective experience of the industry and thus provide protection against common design errors. For example, industry standards were not followed with the design of the power panel printed circuit boards. Observation: There were cases where a tier 1 supplier did not correctly flow down specific Boeing requirements to a subtier supplier(s). In these cases, it was expected that industry standard design practices would be followed. Because there was no specific requirement, the supplier considered that aspect of the specification to be optional and made an inappropriate design decision. The supplier incorrectly assumed it successfully met all the requirements, but the actual requirements had not been satisfied. The supplier made these determinations independently and did not consult Boeing or the first-tier supplier. It is likely this issue would have been identified and mitigated (with a design change) if the decision had been explicitly discussed during design reviews. TEST Topic: Insufficient test design, either in inadequate test coverage or unrepresentative test environment. Insufficient test design resulted in the failure to detect component and system design and/or implementation issues. These issues were then discovered later in the development cycle, when they were more difficult and expensive to correct. For example, the test environment for the VFSG oil-cooling circuit was not representative of the airplane, nor was the testing of the SEMA motor brake, which used a laboratory power supply in place of the EMCU. Boeing 787–8 Critical Systems Review Team Report Page A–7 Observation: In some cases, aspects of the design were assumed to be noncritical to a test, which allowed testing to be conducted on a test rig in which those aspects did not adequately represent those in the type design. Some of these portions of the test rig performed in a manner different from the actual parts installed on the airplane, leading to inadequate testing. For many tests, engineering must make decisions about (1) which aspects of the airplane must be replicated exactly, (2) what portions can be simulated using non-type designed equipment, and (3) when such equipment can be used and what the critical operating characteristics must be. In the above cases, incorrect decisions were made about the degree of fidelity necessary for the test configuration. In some situations, the incorrect decision may have been the result of poor communication between Boeing and the various suppliers or among the suppliers whose equipment interacts in the airplane. ANALYSIS Topic: System failure modes were not properly identified in the failure modes and effects analyses (FMEA). The subteam noted two instances in which system failure modes were not properly identified in the FMEAs. First, there was a single instance in which the system effects were not identified by the change impact analysis. For example, a voltage ripple occurred when there was an open circuit in the VFSG rotating diode. The severity of the ripple and the impact on the GCU was not identified in the FMEA. Previous experience with similar, less complex designs led to analysis assumptions that were not validated. The increase in complexity might have contributed to this analysis deficiency. In another case, the FMEAs did not predict the system effects of cosmic radiation. Observation: Previous experience with similar, less complex designs led to analysis assumptions that were not validated. For complex systems, it is not possible to predict all failure modes and their effects with 100-percent precision. It also is not feasible to verify all possible failure modes using fault insertion testing for complex systems. This is why critical airplane systems typically have multiple layers of redundancy. The subteam concluded none of the unintended system effects related to analysis deficiencies resulted in a safety-related effect on the airplane and were mitigated, per design, by the multiple layers of protection built into the system and aircraft architectures. Boeing 787–8 Critical Systems Review Team Report Page A–8 IN-SERVICE Topic: A high number of component failures were cataloged and later identified as “no fault found” by the affected supplier. There were few issues related to the in-service use of the parts reviewed. For example, there was an assumption that the airlines would inspect the part when a particular condition was annunciated. But the airlines chose to remove the part rather than inspect it to ensure rapid gate turnaround and airplane dispatch. For example, when there is a VSFG failure, the fault could be in the VSFG or in the GCU that manages it. Airlines would often replace both the VFSG and the GCU rather than incur a longer delay while performing extended troubleshooting to isolate the failed component. This approach to troubleshooting resulted in numerous component removals where suppliers later determined the component was functioning properly and identified it as “no fault found.” Also, until the product/model is mature and familiar, airlines will take the least time-consuming and most reliable approach to resolve issues to ensure a timely departure, even if it requires the removal and replacement of good parts in the process. Observation: Airlines take a broad-based approach when evaluating a particular fault message and sometimes replace all related parts rather than expend time to further isolate the failed component. This maintenance practice inflates the number of components recorded as “failed,” many of which are later identified as “no fault found” by the affected supplier. ENGINEERING CONFORMITIES Topic: The number of engineering conformities conducted on the B787 program was substantially greater than that of previous airplane certification programs. Conformity20 is a key aspect of any certification program. There are several important types of conformities: • Applicant conformity is typically done by the manufacturer as an internal process, while Federal Aviation Administration (FAA) conformity is a regulatory confirmation of the manufacturer’s (that is, the applicant’s) conformity finding. • Engineering conformities are needed when conducting tests/inspections that are intended to support design approval; manufacturing conformities are needed to show that the manufacturing process is producing parts that conform to the approved design. 20 Conformity means an aspect of the manufactured product matches the engineering data, including (1) the physical aspects of the design, (2) the processes by which the components were constructed, and (3) the installed software. Engineering conformity inspections intend to document/verify the configuration of certain articles undergoing compliance inspections or testing matches the design data. Boeing 787–8 Critical Systems Review Team Report Page A–9 A review of Boeing data showed the FAA (or the Boeing organization designation authorization (ODA)) performed substantially more engineering FAA conformity inspections for the B787 program than for the B777 certification program. The following FAA orders provide guidance on engineering conformity inspections: • FAA Order 8110.4C, Type Certification, effective during the B777 certification program, provides generally applicable guidance regarding when conformity inspections are needed, including programs involving individual designees. • FAA Order 8100.9A, DAS, DOA, and SFAR 36 Authorization Procedures 21, (now cancelled) provided information on FAA conformities for delegation option authorizations (DOA). • FAA Order 8100.15B, Organization Designation Authorization Procedures, provides information on when FAA conformities are needed for projects managed by an ODA. Originally, individual designees performed authorized functions on behalf of the FAA; the designee functions then converted to the Boeing DOA, and later transitioned to the Boeing ODA. These multiple changes in types of designee functions in the B787 program resulted from parallel changes in the FAA’s designee program. A discussion with Boeing, the Boeing Aviation Safety Oversight Office, and the FAA Engineering Division revealed differing interpretations of the guidance. Total applicant engineering conformity is necessary for all tests and inspections needed to show compliance. However, the guidance is not clear and consistent on the requirements for FAA engineering conformity inspections (which may be conducted by the FAA or delegated to the ODA). Some language has been interpreted to mean that FAA conformity inspections must be conducted on all test articles, while other guidance language suggests the FAA (or the delegated organization) may determine which of the test articles must have an FAA conformity inspection. These differing interpretations may have led to a perceived need for more FAA conformity inspections on the B787 than were required on the B777. The B787 also involved the use of novel technologies, design processes, and manufacturing processes, all of which can add risk from a conformity perspective. Some of these novel aspects may have also contributed to an increase in the number of FAA engineering conformities. It is unclear which of these factors increased the number of required FAA conformities on the B787, but it is apparent the FAA orders need clarification on the requirements for FAA engineering conformity inspections. The FAA is adopting a more risk-based approach to determine best use of FAA resources, including resources needed to perform conformity inspections, as it and industry are moving toward adopting more formalized safety management systems. 21 DAS stands for Designated Alteration Station. SFAR stands for Special Federal Aviation Regulation. Boeing 787–8 Critical Systems Review Team Report Page A–10 One-hundred-percent applicant conformity (performed by the manufacturer) is necessary and should continue. However, requiring FAA confirmation of every such conformity is not a risk-based approach. There are cases in which the applicant may have extensive experience with the general design principles and manufacturing processes relevant to a given test, and the FAA may have conducted numerous successful conformity inspections on similar designs for that applicant in the past. In such circumstances, the FAA should be able to rely on the applicant’s conformity statement, rather than expending additional FAA and delegated organization resources to perform an additional FAA conformity inspection. Observation: FAA ODA policy does not provide adequate guidance to ensure riskbased conformity inspection plans. A.2.3. CONCLUSIONS A focus of the B787 review was to validate the certification process and ensure the airplane critical systems meet their intended level of safety. The previous discussions present the results of the Systems Subteam activities. As part of the review process, Boeing and/or the suppliers involved with the components and/or subsystems selected by the subteam for review provided details on what was being done to address the issue(s) associated with the part/system under review. In all instances, the actions being taken by Boeing and the supplier—and overseen by the FAA—to resolve the issue(s) were sufficient and comprehensive in nature. The Systems Subteam did not identify any additional safety issues or actions that needed to be addressed. The results of the reviews were compiled and assessed as detailed above. In reviewing this compilation, although many of the part/system problems were categorized into requirements or design issues, the cause for each of the deficiencies was found to be unique. This does not mean there were no lessons learned by Boeing or the affected suppliers from these events. Chapter 2 of this report presents lessons that Boeing and its suppliers identified and incorporated into their process documents and/or design guides to ensure similar errors are not repeated in the future. The initiative of continuous improvement was evident throughout the review and is part of the process to mature and improve the parts, systems, and airplane. The discovery of the high number of engineering conformity inspections for the B787 as compared to the B777 revealed the need for clear FAA guidance on risk-based conformity inspection plans. Boeing 787–8 Critical Systems Review Team Report Page A–11 A.3. PROPULSION The Propulsion Subteam selected the following components for deep-dive review: • Motor-operated ball valves. o • Fuel tank access doors. o • Impact-resistant fuel doors are fuel tank access doors on the lower surface of the wing located inboard of the engines, and are specifically designed to be resistant to effects from thrown tire tread fragments and small engine rotor fragments. Wing fuel tank skin surfaces—electromagnetic effects protection. o • Fuel tank access doors are removable oval covers (approximately 10 by 18 inches) on the lower surface of the wing, providing manufacturing and maintenance access to the fuel tanks and associated systems in the tanks. Impact-resistant fuel tank access doors. o • Motor-operated ball valves control the distribution of fuel within the fuel tanks and allow for the routing of fuel to desired locations to support engine operation, auxiliary power unit operation, fuel balancing, and fuel jettison. The wing is a composite structure that incorporates the fuel tanks and provides specific design features to protect against external ignition sources such as lightning and electrostatics. Fuel couplings. o Couplings connect fuel transfer tubes and ducts within the airplane fuel system. Boeing 787–8 Critical Systems Review Team Report Page A–12 Motor-operated ball valves. The motor-operated ball valve is a direct current electric-motor-driven actuator mounted on an actuator adapter within the unpressurized wing. The position of the actuator (open or closed) is indicated by end stop microswitches. Figure A–5 shows valve actuator micro-switches and position indication. Figure A–5. Valve Actuator Micro-Switches and Position Indication Boeing 787–8 Critical Systems Review Team Report Page A–13 Fuel tank access doors. There are two types of fuel tank access doors on the B787— titanium and hybrid. The impact-resistant fuel tank access doors and the vent scoop doors are titanium doors; the middle zone access doors and pressure relief doors are hybrid doors. The location of the fuel tank access doors is shown in figure A–6. Figure A–6 Main Wing Fuel Tank Access Door Types and Locations Wing fuel tank skin surfaces—electromagnetic effects protection. Fasteners with dielectric tops (sealant flush to the outer surface of the fuel tank skin) in combination with copper foil protect against external ignition sources such as lightning and electrostatics on the wing tank skin surface by diverting and distributing current. A figure showing a cross section of a fastener with a dielectric top is not included because the drawing is proprietary. Fuel couplings. The engine fuel feed manifold in the strut consists of rigid and full flexible couplings. From wing front spar to engine hook up-fuel line, there are six couplings for the General Electric engine and five couplings for the Rolls-Royce engine. Figure A–7 shows the details of a full flexible coupling. Figure A–7. Full Flexible Coupling Boeing 787–8 Critical Systems Review Team Report Page A–14 A.3.1. SCOPE OF SUBTEAM REVIEW The B787 has experienced in-service events resulting from issues with engine components. The engines are a separate type-certificated product. Many aspects of the engine are certificated by the FAA independently from the aircraft under Title 14, Code of Federal Regulations (14 CFR) part 33, Aircraft Engines. General Electric and Rolls-Royce produce the engines used on the B787 under Engine Type Certificates E00078NE and E00076EN, respectively. As previously noted, this report does not address the certification requirements and activities under part 33. The Propulsion Subteam limited the scope of its review to systems certificated under 14 CFR part 25, Airworthiness Standards: Transport Category Airplanes. Part 25 requirements address engine installation effects on the airframe and ensure the engine airframe combination does not jeopardize safe operation of the aircraft. A.3.2. SELECTION PROCESS SUMMARY The Propulsion Subteam reviewed Material Review Board (MRB) items known as emergent process documents, voluntary disclosures, noncompliances, and flight test data leading to the selection of the following components for deep-dive review: Motor-operated ball valve. The B787 has experienced at least one in-service event in which a motor-operated ball valve did not function as intended. Additionally, the motor-operated ball valve was the subject of a manufacturing quality escape. Fuel tank access doors and wing fuel tank structure. The fuel tank access doors and wing fuel tank structure are areas on the B787 that are made of composite material rather than the traditional aluminum construction used on Boeing’s previous model airplanes. In addition, these areas were affected by a change in certification guidance regarding fuel tank lightning protection. These changes, in combination with manufacturing nonconformance (quality escapes) and noncompliance disclosures, were the basis for selecting the fuel tank access doors and wing fuel tank structure for deep-dive review. Fuel couplings. Fuel couplings have been the subject of manufacturing quality escapes. Recognition of the potential failures associated with misassembling certain fuel couplings was the basis for selecting this component for deep-dive review. Impact-resistant fuel tank access doors. Impact-resistant fuel doors on the B787 experienced a late design change during the certification program. This component was selected for deep-dive review because of in-service events on previous airplane models that resulted in impact and nonimpact fuel tank access doors being mistakenly interchanged. A.3.3. OBSERVATIONS SUMMARY The Propulsion Subteam observations, grouped by requirements and design issues, are presented below. Observations that do not have a common theme are not discussed in this section. Boeing 787–8 Critical Systems Review Team Report Page A–15 REQUIREMENTS Topic: Inadequate verification that the design met requirements. A common thread among the selected components is the importance of requirements clarity, verification, and validation of assumptions. For example, the motor-operated ball valves experienced a design deficiency rooted in a misinterpretation of the requirement during the flowdown from Boeing to the valve supplier. The error resulted in latent failure faults that were not discovered during the design review process or development of the system safety assessment where requirements verification is typically conducted. Instead, the latent failure condition was discovered after the airplane entered commercial service. Likewise, lack of clarity and verification for certain fuel couplings installation requirements resulted in cases of incorrect fuel tube engagement. Although not a focus of the CSRT review, the Propulsion Subteam noted the aircraft maintenance manuals lacked consistency with fuel coupling engineering and production work instructions. The fuel tank access doors and wing fuel tank skin surfaces contained limited instances in which the requirements failed to address and validate system-level interactions of components/design features. These cases were because of incorrect assumptions about the interaction of features in contact with the component. Failure to validate assumptions about the composite access door mating surfaces led to ineffective electrical bonding characteristics of the composite access doors. Additionally, crack growth testing of the dielectric top (sealant flush to the outer surface of the fuel tank skin) incorrectly assumed crack growth characteristics of low moisture composite materials. Observation: Requirements clarity and verification must be emphasized throughout the supply chain; however, there did not appear to be any significant flaws in the development and verification of airplane, system, and component requirements themselves. The subteam noted that although it observed errors related to requirements implementation for the motor-operated ball valves, fuel tank access doors, wing fuel tank skin surfaces, and fuel couplings, it found no pervasive flaw in the requirements themselves. Corrective actions were initiated for these errors after discovery. Features of the motor-operated ball valve actuator and wing fuel tank skin surfaces are being evaluated under the FAA and Boeing COS process. The COS process also was used to evaluate certain fuel couplings, and resulted in the issuance of Airworthiness Directive (AD) 2012‒24‒07 (72 FR 72200, December 5, 2012). In addition, inspections of the engine fuel feed manifold coupling installation were incorporated in production to verify correct installation on production aircraft. The FAA and Boeing are reevaluating the fuel coupling installation to ensure production aircraft have correct fuel tube engagement. These errors underscore the importance of peer review to identify and validate key assumptions and ensure requirements are clearly understood throughout the supply chain. Boeing 787–8 Critical Systems Review Team Report Page A–16 DESIGN Topic: Occasional lack of rigor in adhering to the established design review process. Design errors were reviewed for the motor-operated ball valve, fuel tank access doors, and impact-resistant fuel doors. A common thread of the individual observations was an occasional lack of rigor in following the established design review process. Although the formal design review process is well defined, observations indicate an occasional lack of rigor in both peer and SME design reviews. In these cases, the design reviews failed to identify the position indication failure mode of the motor-operated ball valve. In addition, design reviews failed to identify interference issues between the fuel tank access doors, clamp rings, and exterior wing paint affecting an electrical bonding characteristic. Finally, a design error in the impact-resistant fuel doors resulted from not following the process for assessing design changes and led to the ability to install nonimpact doors in locations requiring impact-resistant doors. Observation: The formal design review process is well-defined; therefore, peers and SMEs must follow the established design review process. The subteam determined that allowing program progression before achieving the necessary design maturity is a contributing factor to the identified issue. Subsequently, Boeing implemented a gated design process that uses non-advocate peer review of key engineering deliverables to ensure an appropriate level of maturity and/or planning for each design phase. In addition, Boeing introduced an airworthiness limitation to require operators to verify impact-resistant doors are installed in the correct location following access door removal and reinstallation at a specified inspection interval. For fuel tank access doors, Boeing is currently adding a bond check to verify adequate bonding for production aircraft. Finally, Boeing is updating service documents to add the bond check to address in-service aircraft. A.3.4. CONCLUSIONS The Propulsion Subteam noted the FAA and Boeing have taken the appropriate steps to ensure a resolution of issues and prevention of future events. They are addressing the identified issues using existing processes such as the COS process, issuing service documents for airline fleets, and improving certain checks to the engineering process. Also, Boeing’s implementation of the enterprise gated design process provides an improved structured and disciplined approach for each design phase. These measures serve to address safety concerns and provide a proactive approach to minimizing future errors. A.4. STRUCTURES The Structures Subteam reviewed data for significant issues affecting structure, the failure of which could adversely affect the structural integrity of the airplane. Boeing 787–8 Critical Systems Review Team Report Page A–17 The components selected for deep-dive review as a result are the horizontal stabilizer and aft fuselage sections 46, 47, and 48. Figure A–8. Horizontal Stabilizer and Aft Fuselage Sections 46, 47, and 48 Horizontal stabilizer. The horizontal stabilizer provides stability for the airplane to keep it flying straight. The horizontal stabilizer prevents up-and-down (pitching) motion of the aircraft nose. The B787 horizontal stabilizer uses a multi-spar, rather than multi-rib, arrangement and a completely co-cured composite torque box. Aft fuselage sections 46, 47, and 48. The aft fuselage is primarily a single-cell tube of vertical oval cross-sectional shape. Fuselage sections 46 and 47 are pressurized and consist of an upper lobe and a lower lobe, separated by the passenger floor. Section 48 is unpressurized and is the structure that supports the empennage (tail). A.4.1. SELECTION PROCESS SUMMARY The subteam used a data-driven approach based on factual data collected to date from the B787 certification program, build/production, test, and operations. The subteam subjected the data to a number of filters to identify potential pervasive/systematic challenges such as test or operational failures, frequency of nonconformance occurrences, and inconsistencies in certification documentation. The subteam further filtered these areas of interest to identify specific issues, parts, and/or subsystems for in-depth review. The subteam review focused on those items identified as having the highest impact on safety risk. The subteam performed a comprehensive review of issues affecting structural elements that contribute significantly to the carrying of flight, ground, or pressurization loads. The integrity of these structural elements is essential in maintaining the overall structural integrity of the airplane. Boeing 787–8 Critical Systems Review Team Report Page A–18 The subteam started with a comprehensive review of all nonconformance records filed against engineering, then compared initial findings with data from COS reports, test data, and the related certification compliance data. All data was filtered to identify common themes, allowing the subteam to focus on specific areas and issues that warranted further investigation. The subteam noted COS reports, nonconformance record data, and test data identified common symptoms of discrepant shimming on the horizontal stabilizer and aft fuselage sections. A.4.2. OBSERVATIONS SUMMARY The Structures Subteam’s observation is presented below. Topic: Discrepant shimming on the horizontal stabilizer and aft fuselage sections. Boeing observed numerous production nonconformances in the form of gaps between structural elements and structure pull-up 22 resulting from improper shimming. In the horizontal stabilizer, Boeing identified gaps common to the horizontal stabilizer rear spar terminal fitting. In the aft fuselage (sections 46, 47, and 48), Boeing identified a significantly higher number of nonconformances related to shimming as compared to other fuselage sections with comparable design features. Aft fuselage shimming issues were identified in production and in the full-scale fatigue test. Observation: No systemic engineering or design/certification issues were identified during the examination of the horizontal stabilizer and the aft fuselage sections. Although the subteam did not identify systemic engineering or design/certification issues, it reviewed shimming issues directly related to various aspects of the assembly and manufacturing processes. The subteam passed its observations to the Manufacturing/Quality Subteam for consideration during its deep-dive review at Boeing South Carolina (BSC) and Alenia Aeronautica. A.4.3. CONCLUSION Boeing addressed the shimming issues identified in fuselage sections 46, 47, and 48 through corrective actions implemented before delivery. However, five airplanes were delivered with potentially discrepant shims in section 48. Boeing issued Alert Service Bulletin No. B787‒81205‒SB530004‒00 with inspection and repair requirements for the affected airplanes in anticipation of an AD; however, all inspections and required corrective actions were completed before AD issuance. 22 Structure pull-up is a gap between two structural elements resulting from the gap being too large or the structure too stiff. The gap occurs as the structural elements are being fastened together by tightening the bolts to close the gap. Boeing applies Boeing Process Specification BAC5430, “Fabrication and Installation of Resin Bonded Laminated Shims and Solid Fillers,” for the assembly of composite structural elements on the B787. This specification establishes the requirements for fabricating and installing resin bonded laminated shims, solid fillers, and radius fillers as necessary to fill gaps between structural elements during part assembly. The gaps occur between structural elements due to typical part build tolerances. Boeing 787–8 Critical Systems Review Team Report Page A–19 A.5. MANUFACTURING/QUALITY The Manufacturing/Quality Subteam selected the following areas for deep-dive review: • VFSG (manufacturing only), • BSC (final assembly), • GCU (manufacturing only), • Fuel coupling (manufacturing only), • FAA oversight, and • Alenia. A.5.1. SELECTION PROCESS SUMMARY The Manufacturing/Quality Subteam received deep-dive assignments from the other subteams. Identification of deep-dive areas was achieved through the assigning team’s review process and as described in the above sections. The subteam also identified other areas for review using a data-driven methodology and an analysis of the value stream depicted in figure A–9. Boeing 787–8 Critical Systems Review Team Report Page A–20 Figure A–9. Boeing 787 Value Stream Review A.5.2. OBSERVATIONS SUMMARY The Manufacturing/Quality Subteam’s observations and proposed recommendations are presented below and addressed under a common topic or a specific issue, as appropriate. BUSINESS MODEL AND REQUIREMENTS FLOWDOWN Boeing has developed and instituted a new business model approach to aircraft design and production. This model shares design and production responsibility with suppliers. Although this model is not new to the aerospace industry, sharing, design, and subtier integration previously had not been done on a scale of this size at a major aircraft manufacturer in the United States. Boeing 787–8 Critical Systems Review Team Report Page A–21 The novel manufacturing and assembly processes led to some startup issues both at Boeing and within its supply chain. The business model inserted a new tier into the traditional production value stream below Boeing final assembly that produces and/or integrates major systems and sections of the B787 (see figure A–10). Tiers below this level are structured in a traditional supplier/prime relationship to the tier 1 suppliers. Consequently, the nontraditional aspects of this business model reside in tier 1 suppliers’ expectations and performance. Although BSC and Boeing Fabrication build approximately 35 percent of the airplane, Boeing’s role is almost exclusively final integration and testing. However, Boeing retains overall responsibility for the design and supply chain. Boeing eventually plans to integrate aspects of this business model across new airplane models and derivative programs. Topic: Implementation of a nontraditional business model at Boeing for the B787’s design and production has created unanticipated challenges in requirements flowdown, supply chain management, and FAA oversight. The B787 business model added complexity to contract flowdown of technical requirements and responsibilities to suppliers. This complexity exacerbated flowdown issues between suppliers and their subtier suppliers, leading to misinterpretations and gaps in requirements flowdown. Boeing and its supply chain have implemented multiple actions to close these gaps and eliminate the risks associated with technical flowdown of requirements. Observation: At the start of large, complex airplane development programs, production certificate holders and their suppliers should ensure controls are in place for closed-loop, objective evidence of critical process completion based on product and/or process risk at every level of the supply chain. A closed-loop system will minimize misinterpretation and gaps at the first tier creating a reliable subsequent flow of information to lower tier suppliers. Examples are— • Validation of First Article Inspection report in compliance with SAE International Standard No. SAE AS9102, • Ensuring appropriate engineering change approvals, and • Implementation of critical manufacturing and inspection plans. Boeing 787–8 Critical Systems Review Team Report Page A–22 Figure A–10. Traditional vs. Nontraditional Supply Chains Boeing 787–8 Critical Systems Review Team Report Page A–23 SUPPLIER QUALITY AND SUPPLIER MANAGEMENT STAFFING AND PROCESSES Topic: High amount of travelled work, part shortages, and nonconformances in the B787’s initial build phase. Supplier quality, management staffing, and management processes were not adequately in place early in the B787 development program to effectively manage unanticipated issues. Observation: Boeing has invested considerable resources to align supplier performance with necessary expectations to support the B787 business model. Observation: Boeing should continue to provide adequate funding and resources to support new development programs and continue to implement and mature the Enterprise Standard Gated Process. The subteam noted production system and supplier management is maturing and has improved significantly over the past 2 years. Boeing continues to have a major presence at supplier sites. INSPECTION DELEGATION Topic: Inspection delegation programs are not standardized and vary widely in approach throughout the industry. The FAA and Boeing use inspection delegation; however, many of Boeing’s suppliers have unique inspection delegation processes, some of which do not meet minimum industry standards. The CSRT noted that industry standards (such as the G–14 Americas Aerospace Quality Standards Committee AS9015 Supplier Self Verification Process— Delegation Programs document) exist as an aviation best practice for inspection delegation that includes training, testing, and currency requirements. Observation: Industry standards should be required for inspection delegation programs that include certification and recertification processes. FUEL COUPLINGS Topic: Variation exists in the use of work instructions and engineering specifications for B787 fuel pylon coupling installations. A lack of clarity and verification for certain fuel coupling installation requirements resulted in cases of incorrect fuel tube engagement. Boeing 787–8 Critical Systems Review Team Report Page A–24 Observation: Boeing must ensure work instructions and appropriate standards are used in compliance with its quality management system (QMS). FOREIGN OBJECT DEBRIS Topic: Foreign object debris (FOD) controls have improved, but opportunities for improvement exist. Despite good intentions, tools and equipment introduced without appropriate paperwork into the manufacturing/assembly process for installation or removal become FOD hazards and should be recognized as such. Observation: FOD controls have improved and generally tend to be generic and not differentiated by criticality of process or product. FAA MANUFACTURING RESOURCES Topic: The Boeing Certificate Management Office (CMO) has taken a proactive approach by significantly increasing its exposure to new and novel technologies, manufacturing processes, and QMS procedures. The FAA developed a clear and detailed preproduction certificate B787 validation plan that included QMS compliance review, facility review, and targeted conformity inspections. However, its effort was heavily weighted toward structures. Boeing’s restructuring of the supply chain responsibilities and expectations placed increased importance on adequate and appropriate FAA policy. Current FAA policy driving review and acceptance of an aircraft manufacturer’s production capability to produce to an approved design and follow appropriate processes and procedures is an antiquated “one size fits all” methodology. The generic and subjective nature of the current policy fosters significant variability between FAA offices tasked with production certification. FAA manufacturing offices can approve for production aircraft manufacturers of relatively simple, low-complexity small aircraft exactly the same as extremely complex, large-scale aircraft manufacturers using complicated international supply chains. Observation: The FAA policy regarding production approval procedures does not recognize the differing levels of complexity of manufacturing systems and technologies between small, relatively simple aircraft manufacturers and large-scale, complex aircraft manufacturers with extended supply chains. Boeing 787–8 Critical Systems Review Team Report Page A–25 FAA POLICY—BUSINESS MODEL Topic: FAA manufacturing certificate management policy does not align with the current B787 supply chain environment nor will it adequately accommodate surveillance of future aircraft manufacturing under similar business models. The Boeing business model has inserted a new tier into the traditional production value stream below Boeing final assembly that produces and/or integrates major systems and sections of the B787. This manufacturing approach will significantly reduce first-tier suppliers to the production approval holder; increase supplier responsibility and accountability; and push actual manufacturing to tier 2 and below. Current FAA certificate management policy does not have the flexibility to adequately focus resources in a standardized fashion to new areas of risk—that is, significantly reduced tier 2 supplier population; significantly increased tier 2 supplier control responsibility and expectations; and significantly increased design control and change management at tier 2. Boeing eventually plans to integrate this business model across new airplane and significant derivative programs. Consequently, with the inherent business advantages associated with this methodology, it is expected that more aircraft manufacturers and large production approval holders will adopt this concept. Observation: The FAA policy regarding certificate management of production approval holders does not recognize the importance of subtier suppliers in modern complex manufacturing business models. Instead, it addresses only risk at the manufacturer’s top system level. A.5.3. CONCLUSION Boeing has mitigated issues related to aircraft assembly and manufacturing using proprietary processes and its QMS. Overall, the subteam noted continued improvement in all areas, as evidenced by the reduced number of interventions, negligible changes to the current processes, and higher quality of output coupled with an increased B787 production rate. Additionally, the subteam observed continuing areas for improvement for the FAA oversight process to ensure a progressive and flexible regulatory environment. Boeing 787–8 Critical Systems Review Team Report Page A–26 A.6. CSRT DEEP-DIVE REVIEW SUMMARIES—OBSERVATIONS AND RECOMMENDATIONS Area Topic Observation Recommendation No. (see chapter 2 of this report) Systems Requirements flowdown. Even though Boeing requirements existed, verification/validation of the requirements did not always occur between Boeing and/or subtier suppliers. Requirements assumptions. In some cases, previous experience with similar designs led to assumptions that were not validated. Requirements ownership. With multiple suppliers involved, either in the same supply chain or across multiple platforms/commodities, the lack of a defined owner resulted in a requirement not being adequately communicated and/or verified. Incorrect assumption that proven design solutions would be equally applicable to new and/or novel design applications without validation of those assumptions or their context. Indications that incorrect assumptions survived peer and SME design review demonstrate that additional oversight and diligence in following formal design and design review processes, including the use of design checklists, is required. Boeing 787–8 Critical Systems Review Team Report 2. Boeing should continue to implement and mature the gated design and production processes with sufficient resources for development programs, and to minimize risks throughout the life cycle of the program. In these processes, a series of programmatic “gates” are established at various points during the development program. Each gate has specific criteria for proceeding to the next development phase. Any criteria that have not been satisfied at a given gate must be addressed or mitigated before proceeding to the next phase. (Boeing is realizing improved performance in the Boeing 737 MAX, Boeing 787‒9, and Boeing 767‒2C programs from using a gated approach.) Page A–27 Area Topic Observation Recommendation No. (see chapter 2 of this report) Appropriate industry design standards not followed. There were cases where a tier 1 supplier did not correctly flow down specific Boeing requirements to a subtier supplier(s). In these cases, it was expected that industry standard design practices would be followed. Because there was no specific requirement, the supplier considered that aspect of the specification to be optional and made an inappropriate design decision. The supplier incorrectly assumed it successfully met all the requirements but the actual requirements had not been satisfied. The supplier made these determinations independently and did not consult Boeing or the first-tier supplier. It is likely this issue would have been identified and mitigated (with a design change) if the decision had been explicitly discussed during design reviews. Insufficient test design, either in inadequate test coverage or unrepresentative test environment. In some cases, aspects of the design were assumed to be noncritical to a test, which allowed testing to be conducted on a test rig in which those aspects did not adequately represent those in the type design. Some of these portions of the test rig performed in a manner different from the actual parts installed on the airplane, leading to inadequate testing. Boeing 787–8 Critical Systems Review Team Report 3. Boeing should ensure suppliers are fully aware of their responsibilities, including integration responsibilities and accountability for subtier performance. The gated design processes should include supplier planning, performance, and reporting, using measurable and appropriate performance criteria that include the scope and effectiveness of design reviews and other airplane life-cycle activities. Page A–28 Area Topic Observation Recommendation No. (see chapter 2 of this report) System failure modes were not properly identified in the FMEAs. A high number of component failures were cataloged and later identified as “no fault found” by the affected supplier. Airlines take a broad-based approach when evaluating a particular fault message and sometimes replace all related parts rather than expend time to further isolate the failed component. This maintenance practice inflates the number of components recorded as “failed,” many of which are later identified as “no fault found” by the affected supplier. The number of engineering conformities conducted on the B787 program was substantially greater than that of previous airplane certification programs. Propulsion Previous experience with similar, less complex designs led to analysis assumptions that were not validated. FAA ODA policy does not provide adequate guidance to ensure risk-based conformity inspection plans. None. Inadequate verification that the design met requirements. Requirements clarity and verification must be emphasized throughout the supply chain; however, there did not appear to be any significant flaws in the development and verification of airplane, system, and component requirements themselves. See Recommendation No. 2. Boeing 787–8 Critical Systems Review Team Report Page A–29 Area Topic Observation Recommendation No. (see chapter 2 of this report) Occasional lack of rigor in adhering to the established design review process. Structures The formal design review process is well-defined; therefore, peers and SMEs must follow the established design review process. See Recommendation No. 2. Discrepant shimming on the horizontal stabilizer and aft fuselage sections. No systemic engineering or design/certification issues were identified during the examination of the horizontal stabilizer and the aft fuselage sections. N/A At the start of large, complex airplane development programs, production certificate holders and their suppliers should ensure controls are in place for closed-loop, objective evidence of critical process completion based on product and/or process risk at every level of the supply chain. A closed-loop system will minimize misinterpretation and gaps at the first tier creating a reliable subsequent flow of information to lower tier suppliers. Examples are— 1. Boeing should establish a means to ensure suppliers identify realistic program risks and complementary mitigation plans through a closed-loop flowdown validation of requirements. (Also see Recommendation No. 2 regarding allocation of sufficient resources.) Manufacturing/ Implementation of a Quality nontraditional business model at Boeing for the B787’s design and production has created unanticipated challenges in requirements flowdown, supply chain management, and FAA oversight. • • Ensuring appropriate engineering change approvals, and • Boeing 787–8 Critical Systems Review Team Report Validation of First Article Inspection report in compliance with SAE AS9102, Implementation of critical manufacturing and inspection plans. Page A–30 Area Topic Observation Recommendation No. (see chapter 2 of this report) High amount of travelled work, part shortages, and nonconformances in the B787’s initial build phase. Boeing has invested considerable resources to align supplier performance with necessary expectations to support the B787 business model. See Recommendation No. 1. Boeing should continue to provide adequate funding and resources to support new development programs and continue to implement and mature the Enterprise Standard Gated Process. Inspection delegation programs are not standardized and vary widely in approach throughout the industry. Industry standards should be required for inspection delegation programs that include certification and recertification processes. Variation exists in the use of Boeing must ensure work instructions and work instructions and appropriate standards are used in compliance engineering specifications with its QMS. for B787 fuel pylon coupling installations. FOD controls have improved but opportunities for improvement exist. Boeing 787–8 Critical Systems Review Team Report 4. Boeing should require its suppliers to follow industry standards for the training, qualification, and certification of supplier personnel performing Boeing-required (non-FAA) inspections. See Recommendation No. 2. FOD controls have improved and generally tend N/A to be generic and not differentiated by criticality of process or product. Page A–31 Area Topic Observation Recommendation No. (see chapter 2 of this report) The Boeing CMO has taken a proactive approach by significantly increasing its exposure to new and novel technologies, manufacturing processes and QMS procedures. The FAA developed a clear and detailed preproduction certificate B787 validation plan that included QMS compliance review, facility review, and targeted conformity inspections. However, its effort was heavily weighted toward structures. The FAA policy regarding production approval procedures does not recognize the differing levels of complexity of manufacturing systems and technologies between small, relatively simple aircraft manufacturers and large-scale, complex aircraft manufacturers with extended supply chains. None. FAA manufacturing certificate management policy does not align with the current B787 supply chain environment nor will it adequately accommodate surveillance of future aircraft manufacturing under similar business models. The FAA policy regarding certificate management of production approval holders does not recognize the importance of subtier suppliers in modern complex manufacturing business models. Instead, it addresses only risk at the manufacturer’s top system level. None. Boeing 787–8 Critical Systems Review Team Report Page A–32 APPENDIX B—ACRONYMS 14 CFR Title 14, Code of Federal Regulations ACO FAA aircraft certification office AD airworthiness directive ASIAS Aviation Safety Information Analysis and Sharing B777 Boeing 777 B787 Boeing 787 BCA Boeing Commercial Airplanes BSC Boeing South Carolina CMO certificate management office COS continued operational safety CSRT Critical Systems Review Team DAS Designated Alteration Station DOA delegation option authorizations ECE Equipment et Construction Electrique EE–1 ETOPS in-service event EIS entry into service EMCU electronic motor control unit EPGSS electric power generation and start system ETOPS Extended Operations FAA Federal Aviation Administration FMEA failure modes and effects analysis FOD foreign object debris GCU generator control unit ODA organization designation authorization QMS quality management system SEMA spoiler electromechanical actuator SFAR Special Federal Aviation Regulation SME subject matter expert UTAS United Technologies Aerospace Systems VFSG variable frequency starter generators Boeing 787‒8 Critical Systems Review Team Report Page B–1