Steve Swift Steve Swift Pty Ltd 2012

Transcription

1 Nil Findings Three nasty but neglected illusions of structural inspection programs Steve Swift Steve Swift Pty Ltd 2012 FAA Symposium on Human Factors for Maintenance and Ramp Safety, September 1-3, 2009, San Diego, CA, USA 1 This presentation is for all involved in maintenance: - mechanics - Directors of Maintenance and Chief Inspectors for airlines - Product Support departments of aircraft manufacturers - ATA, IATA and other industry bodies that publish standards or advice on maintenance - airworthiness authorities, like the FAA - MRBs and STGs - human factors experts It is about three nasty but neglected illusions of structural inspection programs, when there are "nil findings".

2 Wing Spar Cracks Like illusions for pilots, illusions for maintainers have caused crashes and close calls. Here are just four. 2

3 From FAA MHFPS Illusions are a human error we don t often talk about for maintainers. But, their cause is something we do talk about. It is one of Transport Canada s dirty dozen. It is lack of knowledge. 3

4 From FAA MHFPS 4 But, it is more than lack of knowledge. As former US Defense Secretary Donald Rumsfeld famously told a press briefing: There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we now know we don t know. But there are also unknown unknowns. These are things we do not know we don t know.

5 Known unknown Unknown unknown 5 Like Mr Rumsfeld, we all know the problem of not knowing something when we should. But, it is a known unknown. So, hopefully, we will ask someone who does know, for advice. (If the human factor of pride does not stop us.) But what if it is an unknown unknown? So, we don t know we don't know. Worse, sometimes we think we do know. Sometimes, like Mr Rumsfeld, we too can be happy and confident, but wrong.

6 il lu sion [i-loo-zhuhn] noun: 1. a false or unreal perception or belief. Sometimes, we could be experiencing an illusion. Illusions are not just the optical variety or the stuff of magicians. An illusion is any false or unreal perception or belief. 6

7 For example, people once thought the world was flat, like this. They were confident about it. It worked for them most of the time. It was an illusion. 7

8 Sydney Los Angeles Tokyo London 8 Australians think it is like this! So, now who is down under? Even today, whatever our perspective of a flat earth, it works for us most of the time. If we want to fly from Los Angeles to Sydney, it will show us roughly the right route. But, what if we want to fly from Los Angeles to London? Won t we think we have to fly a long route via somewhere like Tokyo? Now, our flat earth map isn t so good, is it? If we did not know better, we could try to plan our travel based on this illusion. As travel agents sometimes try to do! Of all people, you would think they would know better, wouldn't you? Note: McArthur s Universal Corrective Map of the World is available from: ODT Maps, Inc. PO Box 134, Amherst, MA, USA Tel Fax Int'l

9 London Los Angeles 9 Of course, we wouldn t do that, would we? We know the earth is round, not flat. So, we know the real route from Los Angeles to London is more like this. But, before we get too smug, are there things we don't know about maintenance, but we don't know we don't know, so we also experience illusions?

10 The knowledge divide Did you know that over the last few decades, we ve been learning new and sometimes radical things about structural inspection programs? Or, some have. On the left, the academics at the research establishments, and the engineers in the design offices, have been studying the science of fatigue and corrosion, and the theory and statistics of managing them. They present complicated papers to each other at high-powered conferences. Unfortunately, the new knowledge is not getting out quickly enough. It has not yet gone far enough. On the right, maintenance managers and practitioners are still often in the dark. There is a knowledge divide that is a human factors story in itself. There is more about it in my paper, A Collective Approach to Aircraft Structural Maintenance Programs, which I presented at the International Air Safety Seminar in Honolulu, USA, in See my web site, The authorities even put the new knowledge about inspection in the rules and advisories for design, not maintenance. But, what is the problem? Why do we need new knowledge when the old knowledge works? The problem, like a flat earth, is it only works most of the time. There are times when, without the new knowledge, we see illusions. 10

11 Neglected Mechanic training AMM/MPD/MRBR/SID/CPCP... ATA MSG-3 IATA Maintenance Programme Optimisation Guidance IMRBPB IP44 Evolution/Optimization Guidelines FAA Order FSIMS FAA MHFPS How well known are these illusions? You will not find anything about them in any of these. Some may even unwittingly encourage them. So, I would like to warn you about them. They can occur in the very practical maintenance situation of you trying to interpret what a history of nil findings means for a structural inspection program. As you know, there is always pressure to stretch the program. Or, even to stop inspecting altogether. The pressure is greater than ever, right now, in the global financial crisis. 11

12 The three illusions Immortality reluctance to do the program Individuality reluctance to improve the program Intervality overconfidence to escalate the program 12 Here are the three illusions: First, Immortality - the belief that a history of nil findings proves that an aircraft will last forever. It leads to a reluctance to do the program. Second, Individuality - the belief that a history of nil findings means the first finding must be an individuality that won t affect the rest of the fleet. It leads to a reluctance to improve the program. And third, Intervality, my own word for the belief that a history of nil findings tells us something about the inspection interval. It leads to overconfidence to escalate the program. Some may seem obvious. Others may leave you thinking.

13 Immortality First, the illusion of Immortality. Have you ever done the same inspection many times and not found anything? Nil findings again, boss!. Or, the same goes for life-limited parts. Have you ever replaced the umpteenth main rotor blade (if you maintain helicopters), or wing spar (if you maintain little Part 23 airliners) and said, another perfectly good part for the scrap bin, boss! Have you ever felt like you are wasting your time? If I were to borrow Dr Bill Johnson's human factors spectacles (see the FAA s Maintenance Human Factors Presentation System) and look at your thoughts, would I see any of these? These aircraft are as strong as a mallee bull, if you know Aussie slang. This inspection (or life limit) is just those greedy parts sellers or overprotective lawyers at it again. Or, have you thought that these aircraft are like a good Californian wine: the older they get, the better they get. So, they need less inspection, less replacing of parts, not more. 13

14 You think you are here Nil findings Failure rate Infant mortality Age Or, perhaps you know a bit of reliability theory. Perhaps you have worked with systems and their reliability programs. You think nil findings mean you are past infant mortality. So, the failure rate will now be very low. And then, from your airframe knowledge, you know a bit about endurance limits. So, won t that last forever? It is plausible, isn t it? So you don't think you need to inspect any more. So, you don't. Or, you don't inspect thoroughly. Or, you re happy to stretch the life limit. 14

15 But, you could be here Nil findings Wear out Failure rate Age But, it is an illusion. The new knowledge tells us that the reliability curve for aluminum aircraft structure actually looks like this. Unlike steel, aluminium does not have an endurance limit, especially in joints, which fret and corrode. So, your nil findings could also mean you are near the start of wear out, where the failure rate will rise. One illusion might not only affect one person s decision about one aircraft, it might also affect a whole organisation s decision about a whole aircraft type. Nil findings have even deterred authorities from enforcing inspections or life limits. It is an issue for Australia, which often has the world fleet leaders. 15

16 Used with permission For example, two years ago, Cessna published a life limit for the 441 Conquest, of 22,500 hours. Their tests predicted that, past 22,500 hours, there could be so much fatigue and corrosion happening, in so many places, known and unknown, there is no way anyone is going to be able to find it all in time. Some 441s in Australia, some in airline service, were already over 22,500 hours. They had been safe so far. The FAA was taking no action. The illusion was that they would stay safe forever. But, the engineering predicted they would not. 16

17 Used with permission So, sadly for operators, Australia s Civil Aviation Safety Authority (CASA) had no choice but to ground them. 17

18 Crack Close up Wing attachment fitting CASA trusted Cessna s engineering. A few years before, when it had enforced Cessna's SID for the 402C, operators found cracks, like this, in wing attachment fittings, where no one would normally look, at least not with eddy currents. Two Cessna 402Cs could have lost wings, as happened in the USA, were it not for doing the SID. There is often cynicism about a company s motives for their inspection program or life limit. But, rarely is it justified. Let us examine ourselves, too, for what the psychologists call confirmation bias. Sometimes we will believe an illusion because we want to - to save money, or work we do not want to do. 18

19 Illusion Reality Immortality Nil findings for a long time means more chance of findings next time Summarising, the first illusion is 'Immortality'. The reality is that nil findings for a long time means more chance of findings next time. 19

20 Individuality The next illusion is Individuality. It can occur when you find something by luck, not by the inspection program. Perhaps you were repairing an area, and looking more closely than you normally would. Or, the repair opened up an area that is usually inaccessible. You do the right thing and submit a Service Difficulty Report, but the manufacturer and the authority dismiss it because of a long history of nil findings for the area. Perhaps they are thinking: this aircraft must have been built or flown or maintained differently from others of the same type. Backing their thinking is the illusion that aircraft really are like one of those cutaway drawings. So, if the problem were common, wouldn't everyone be finding it and reporting it? Now, there may be genuine individuality. But, far more often, it s an illusion. 20

21 First finding Age Failure rate X Or, is it systems-style reliability thinking again, where the failure rate is often constant with age? So, a first finding, in only one old aircraft, simply proves a low failure rate (or a long MTBF). Isn t that good, and nothing to worry about? Let us just monitor the trend, they say. 21

22 First finding Failure rate Age X But again, the failure rate for structure is not constant, it increases with age. The first finding could be showing us where wear out starts, where the failure rate is starting to rise. And, failure is less likely to be redundant for structure, than it is for systems, despite noble ideals like fail-safety and damage tolerance. 22

23 Once upon a time, one authority wrote to another authority about several examples of corrosion as bad as this (and worse) in the wing spars of one aircraft type. It was impossible to see in the wing, without ultrasonics. For six years, the second authority dismissed the concern because they had nil findings from their country. When they finally issued an AD, nearly half their operators, now inspecting with the special method, found corrosion just like this, just as the first authority had warned. Sadly, before the AD, aircraft lost wings. 23

24 Could it happen again, with corrosion like this deep inside the wings of another popular range of aircraft? It is inevitable, because of the design, but you cannot see it because there are no access panels. Again, there is confirmation bias supporting the illusion. Often, the consequences of acknowledging that something could be fleet-wide, not individual, are too huge to face. If you work for a manufacturer or an authority, you know the feeling. 24

25 Illusion Reality Individuality Nil findings nil looking/reporting Absence of evidence is not evidence of absence Summarising, the second illusion is Individuality. The reality is, if you ve got one finding, and no obvious individuality, nil findings elsewhere usually means no one is looking in the right way. Or, if they are, they are not reporting their findings. As the lawyers say, "absence of evidence is not evidence of absence". 25

26 Intervality MPD 8 YR 26 The third illusion is Intervality. The name is as corny as the illusion is common. It is interpreting nil findings as evidence to escalate the inspection s repeat interval. It is most common for corrosion programs. We think nil findings must be good for something. But, if we don't know how the inspection program was developed in the first place (even if we have played the MSG-3 numbers game on an MRB), the illusion is we can extend the interval. So, here, for example, we might be tempted to extend this 6-year interval to 8 years.

27 Damage size Permissible Interval Missable Time From the diamond papers, International Committee on Aeronautical Fatigue (ICAF 2005, 2007 and 2009) So let us see how the inspection interval was developed. This is how it is done, even for CPCPs, even though it is not always so clear in MSG-3. Some call this damage tolerance ; others safety-byinspection ; others the P-F interval from the potential to the functional failure. Whatever we call it, unlike most systems, for structure we have to find damage early, before failure. In the confusing old MSG-2 jargon, it is on condition, not condition monitoring. These are the variables involved. Now. what does nil findings tell us about any of these? The answer is nothing. For more explanation, see the previously-mentioned paper, A Collective Approach to Aircraft Structural Maintenance Programs. Nil findings is not evidence to escalate an interval for a structural inspection program. It may be for the threshold, but not for the interval. Our preoccupation with escalating the interval - the illusion I call intervality - is again driven by confirmation bias. Once our aircraft has passed the threshold - which is why we have been inspecting and, so far, had nil findings - the interval is all we have to play with. So, we do. Unfortunately, it is common. 27

28 Once upon a time, the maintenance manager for an airline told their local airworthiness authority that they had inspected ten propeller blades, to the propeller manufacturer s service bulletin, with nil findings. So, could they extend the service bulletin s repeat interval? If you had not seen the graph and the variables from the previous slide, it would sound plausible, wouldn't it? So, the authority approved the request. Only a few months later, a blade failed. Luckily, it flew off outboard (as shown), not inboard. It could easily have killed all on board. This is not a criticism of the people involved. They were good people - smart people! They were just innocent victims of the illusion of "intervality". They lacked the new knowledge. But, they didn t know they didn t know. 28

29 Illusion Reality Intervality Nil findings may escalate the threshold, but not the interval Summarising, the third illusion is intervality - that nil findings are an argument to extend the repeat interval. The reality is that "nil findings" may escalate the threshold, but not the interval. You need to really watch out for this one. 29

30 Conclusions Illusions for maintenance (like flying) Three for nil findings : immortality individuality intervality Illusions neglected (unlike flying) 30 This slide is self-explanatory.

31 Recommendations Beware illusions esp. after nil findings Share maintenance esp. doers with designers 31 Finally, here are some simple recommendations. First, beware illusions, especially after nil findings. After this presentation, I hope they are no longer unknown unknowns, but are now known unknowns. Second, to make them known knowns, we should share maintenance, especially doers with designers. Designers of structural inspection programs should share their new knowledge*. And, doers should open their maintenance world up to them. Maintenance is no longer just for mechanics. Could we even end the untidy split between Aircraft Certification and MRB for structural maintenance programs? Could we analyse the interdependent processes of fatigue and corrosion together, not separately? In authorities, professional engineers and mechanics have long competed for the regulation of structural maintenance programs. I think I am seeing more cooperation. I hope I am not seeing an illusion!