It took a less than 70 hours of chafing for a hole to be breached in the fuel line, but only about 13 minutes for the computer systems to lead the pilots up the garden path of a fuel imbalance to total fuel exhaustion.
The case involves Air Transat Flight 236, in which both engines shut down from fuel starvation on the A330 twinjet high over the Atlantic Ocean on Aug. 24, 2001, forcing the pilots to glide to a deadstick landing at Lajes in the Azores. Given its landing on Portuguese soil, the event was investigated by that country’s safety board, the Gabinete de Prevencao e Investigacao de Acidentes com Aeronaves (GPIAA, or Aviation Accidents Prevention and Investigation Department). The Portuguese report, anticipated for some months, was released Oct. 18.
Its publication raises significant questions about extended range operations (ETOPS), as well as maintenance, thresholds for alerting displays, pilot-computer interactions, training of both pilots and mechanics, and the adequacy of flight data recorders to perform their vital function when the electrical supply degrades. The period covering the initiation of events that culminate in an incident or accident is a paramount time of data collection for investigators, yet it is classically the time when the vital cockpit voice and flight data recorder (CVR/FDR) function seems most prone to failing utterly.
ETOPS flights at ever-distant routes from emergency airfields are predicated on the prognostication of an infinitely small possibility of system failure. The contingencies are restricted to the loss of a single engine, with the remaining engine unaffected, or a belly hold fire that can be suppressed until a diversion is completed. The Air Transat case points to the potential for human factors on the ground and in the air to confound the finely wrought ETOPS risk calculations.
This case has parallels to two other events. One is the fumbling and confusion regarding the maintenance documentation that resulted in the miswired sidestick on a Lufthansa A320 twinjet. The mistakenly reversed control was discovered on takeoff, and disaster was only averted by the first officer’s quick action in applying an opposite roll control to the captain’s. Had the left wingtip contacted the runway, a spinning fireball would likely have resulted (see ASW, June 4, 2001).
The other relevant case involved the fatal 1998 crash of a Swissair MD-11. This tragedy began with just a wisp of smoke entering the cockpit. The pilots did not realize until too late that the tentative wisp was the telltale signof a raging fire in the attic space over the forward cabin and cockpit (see ASW, April 7, 2003).
In the Air Transat case, the first indication to the pilots was a low oil quantity reading for the right engine. By that point literally tons of fuel had been pumped overboard through a ruptured fuel line.
The parts mismatch
Engine work performed the previous weekend set the stage for what occurred four days later on Flight 236. Metal particles were found Aug. 15 and Aug. 17 on the metal chip detector that is part of the oil system on the right engine, a Rolls Royce RB211 Trent. Since the source of the contamination was unknown, Air Transat officials decided to replace the engine.
The company did not have a spare engine on hand. However, Rolls Royce had positioned a spare “loaner” engine at Air Transat’s Mirabel, Quebec, facility in case any of the engine manufacturer’s North American customers needed a spare.
Air Transat arranged to use this engine. The work began on Friday night, Aug. 17. The work continued over the weekend, and it was conducted by multiple shifts on Saturday and Sunday under the supervision of a lead technician who had been called in specifically to head the work. About 60 items would have to be installed on the loaned engine to make it ready for service, no trivial task. Some of these items were part of service bulletin (SB) changes.
The pressure was on to complete the work in order to maintain the airplane’s flight schedule and to clear the hangar for other activity.
Now follow some seminal highlights of the work that followed:
The engine had last been worked on by Hong Kong Aero Engine Services, Ltd. (HKAESL). Due to a parts shortage, HKAESL had not incorporated an SB involving a fuel line. The engine had been test-run without the hydraulic pumps installed, an accepted practice that did not uncover the chafing hazard later caused by the parts mismatch.
Thus, Air Transat technicians were faced with the challenge of replacing an engine in a post-SB condition with a “loaner” in a pre-SB state. With all other company engines in a post-SB configuration, Air Transat personnel had not been previously exposed to this situation.
A false hydraulic pump part number on the “Carry-Forward” items list that arrived with the replacement engine complicated the situation further. The erroneous part number led Air Transat technicians to wrongly believe that all the parts required for installation were available. They weren’t.
The overall appearance of a pre- and post-SB engine is similar, not likely to be revealed in a cursory visual inspection.
The work was running late. On Sunday morning, technicians noted difficulty fitting the post-SB hydraulic pump. They took the post-SB fuel line from the replacement engine, the different routing of which overcame the earlier difficulty installing the pump.
However, clearance between the post-SB fuel line and the pre-SB hydraulic line from the pump was inadequate. Scratch marks on the fuel line (probably from a screwdriver or other tool) indicated that the clearance between the two lines was forced.
Due to a computer network problem, Air Transat technicians were unable to access the database to check and verify what they suspected might by an incompatible SB problem (i.e., a fuel line SB having been accomplished but not a related hydraulic line SB).
The Rolls Royce representative was contacted by telephone, but he was not aware that the loaned engine was in a pre-SB configuration. Nor was he told of the difficulty accessing the SB database. His offer to visit the work site was not taken up, but he had cautioned as to the importance of proper clearance between the fuel and hydraulic lines.
Because the pre-modification configuration of the replacement engine had not been identified, no work sheets were issued to address applicable SBs needed to bring it into the same configuration as the engine it was replacing.
The pre-SB hydraulic line was a flex tube. Even if the clearance was forced, such clearance would vanish once the line was pressurized under the 3,000 psi of an operating hydraulic system. According to the Portuguese report, technicians in Canada were not trained on the unique considerations involving hydraulic lines incorporating a middle, flexible segment. The curriculum did not alert technicians to the risk of using connector torque as a means of establishing clearances for such lines during installation, as was the case in the Air Transat engine change.
There was no quality control (QC) person on site, as QC staff worked Mondays through Fridays. The QC task was accomplished by a review of the paperwork the following Monday.
There was no engine logbook entry regarding the unexpected replacement of the fuel line, as the technician responsible forgot to make the entry.
Post-installation inspections were limited to ensuring that the engine controls were properly connected, and that all remaining work was completed and within tolerances. Such inspections would not necessarily have revealed the installation of incompatible components (the fuel line and hydraulic line).
Under the pulsing of the hydraulic pump and the vibration attendant to flight, Portuguese investigators determined that the fuel tube fractured “in high cycle fatigue at multiple initiation sites … due to a combination of vibratory stresses being superimposed on the tube deformation.”
Note the defenses against such installation error that were breached:
- The SB database could not be accessed.
- Quality control was out of the loop.
- All work was not documented in the logbook.
- The clearance between the two lines was forced.
It took some four days and 67.5 hours of flying to reach the point of failure, but the fuel line rupture occurred some five hours into Flight 236’s overnight trans-Atlantic crossing from Toronto to Lisbon.
Fuel was pouring out of the irregular 3-inch long by 1/8-inch crack at the rate of about one gallon per second.
The masking messages
The first indication of anything amiss on the flight was during a routine check, when Capt. Robert Pich� and First Officer Dirk De Jager noticed that the right engine’s oil parameters were significantly different than those for the left engine – the oil quantity was lower, the oil temperature was higher, the oil pressure was lower (the unusual indications were caused by the high fuel flow rate through the fuel/oil heat exchanger, a manifestation of the leak). They selected the Engine Status page on their ECAM (Electronic Centralized Aircraft Monitoring) system to probe further. That action de-selected the Fuel page on the ECAM display.
Meanwhile, the A330’s automated fuel management system, responding to an imbalance between the left and right wing tanks, began pumping fuel from the horizontal stabilizer’s trim tank to the “lighter” side, the right wing tanks. A green advisory message (green indicating the system is operating normally) would have been displayed on the ECAM. No cause for concern, here, as the pumping of fuel from the tail tank takes place routinely during the latter stage of the flight (although occurring earlier than usual in this instance).
Unbeknownst to the pilots, the fuel being pumped forward out of the aft trim tank was feeding the leak. Focused on the anomalous oil conditions, the crew was engaged in radio conversations with Air Transat’s maintenance control center (MCC) about the oil problem.
Three minutes after the fuel was pumped out of the aft trim tank, the pilots received a fuel imbalance advisory message. At this point, they deselected the Engine page and the Fuel page came up on the ECAM.
It was flashing a message that there was more than a 6,000 lb. difference in fuel remaining between the left and right wing tanks. In other words, after the system could no longer deal with the fuel leak through pre-programmed fuel balancing, it shed the problem to the pilots with an advisory message (analogous to the way the autopilot maintains flight in icing conditions until it can no longer do so, snapping off and handing the unstable flight control situation to the pilots).
The fuel imbalance procedure calls for the crew to open the crossfeed valve in order to feed fuel from the heavier to the lighter side. Think of the A330 fuel system as garden hose with a “T” fitting, enabling fuel to be transferred between the left and right tanks, and with the upright of the T representing the fuel line from the aft trim tank.
By opening the crossfeed valve, fuel on the left side passed to the right side, and out the ruptured fuel line. The fuel imbalance procedure on the ECAM page contains a cautionary note that if a fuel leak is suspected, the crossfeed valve should not be opened.
But Capt. Pich� didn’t suspect a leak and he responded to the fuel imbalance message by performing the requisite checklist procedures from memory – to include opening the crossfeed valve but forgetting at the time the cautionary note at the top of the checklist. Fuel imbalance procedures were a familiar part of Air Transat’s simulator sessions. In this respect, Pich�’s action was consistent with his training. Significantly, simulator training had not dealt with fuel leak scenarios.
Shortly after opening the crossfeed valve, Pich� and De Jager were faced with the chilling realization that the fuel on board was a good seven tons lower than predicted for that stage of the flight.
They asked the director of cabin services to look out a cabin window for signs of a possible fuel leak. The report was negative – although it would have been difficult to detect a trailing cloud of fuel in the darkness. As the Portuguese report indicated, the pilots suspected erroneous messages on their ECAM display, which had begun with the contradictory engine oil readings for the right engine. Instead of a massive fuel leak, the crew thought they were dealing with a computer malfunction.
Nonetheless, with enough fuel remaining in the left tanks, certainly enough to continue flight on one engine, the crew elected to divert to Lajes.
Some 140 miles away from Lajes, the right engine flamed out. Attempting to ensure that all usable fuel remaining was available for the left engine, the crew attempted to pump fuel forward from the trim tank. Since the automated fuel management system had already done so, the pilots received a low pump pressure message, indicating that the trim tank was empty.
At 65 miles from Lajes, the left engine, starved of fuel, failed. The airplane was now an unpowered glider. The ram air turbine (RAT) deployed, providing sufficient electrical power to operate some flight instruments. Under these conditions, Portuguese investigators described the pilot’s achievement at bringing the airplane to a deadstick landing at Lajes as “remarkable.”
Much has happened since as a result of this very close call:
Air Transat was the first Canadian carrier to implement a safety management system, or SMS (see ASW, Dec. 8, 2003). In addition, the company modified its initial, recurrent and simulator crew training to include fuel leak scenarios.
Airbus issued SB A330-28-3080 on May 21, 2002, modifying the computer system and associated wiring to provide aircrews with a warning of significant loss of fuel in the event of a leak. Portuguese investigators in their report of the Air Transat event urged that this heightened warning capability should be required, not left to the whims of voluntary compliance with an SB. Moreover, Portuguese investigators recommended that a fuel loss alert should be required for other aircraft with similar fuel system designs.
The Azores glider – observations and implications
When parts weren’t available for the engine maintenance, the technicians went in another direction, unaware that the hydraulic and fuel line SBs represented a “matched pair” that critically needed to be incorp-orated at the same time. Whenever maintenance shortcuts are taken, serious problems are much more likely to result.
A supervisor’s sign off on maintenance work once again proved to be a greater responsibility than it is usually accorded.
It would seem that the naturally occurring vibration of the hydraulic line could be sudden death to nearby electrical wires or fuel lines. Since the chafing occurred close to the hydraulic pump, no vibration damping devices were interposed.
When the leak sprung, fuel gushed out, soaking the oil pipes, which led to the seemingly anomalous oil temperature and pressure alerts. The leak was downstream of the oil cooler, so the increased fuel flow through the fuel pipe overcooled the oil, making it more viscous, resulting in the increased oil pressure reading. How many pilots would react to a low oil temperature reading by checking for a fuel leak? One would have to be Dick Tracy with a degree from Sherlock Holmes in Abstract Intricacies.
The airplane’s automated fuel management system dealt with effect, not cause, pumping trim tank fuel to correct an out-of-balance situation, as a consequence feeding more fuel to the leak.
The ECAM advised the pilots of the fuel imbalance, but they mistakenly opened the crossfeed valve, causing the loss of fuel that would have enabled them to make a much safer single-engine divert to Lajes. Undue emphasis in training on fuel imbalance has tended to avoid the questions, “What could possibly create such an imbalance?” and “What history of gross imbalance exists for this model?” As always, questions unasked remain questions unanswered. The deeper question emerges: Have simulator sessions become drudge treadmills, or should they be opportunities for learning and probing inquiry?
Checklist design is extremely important. Recall that working from memory Pich� skipped right over the cautionary note at the top of the checklist about not opening the crossfeed valve. Pilots in emergencies tend to start their checklists at item one and skip over the preambles. If something is a paramount consideration, step one of the checklist might ask, Are you certain that no fuel leak exists? Have you checked fuel used against fuel remaining?
When the engines flamed out, and the attached engine generators and hydraulic pumps shut down, back up electrical and hydraulic power was dependent on deployment of the RAT. Thus, ETOPS calculations of system reliability, with probabilities of occurrence ranging upwards of one in tens of thousands, were reduced to one in two – whether the RAT deployed or not. If it had failed, all hydraulic and electrical power would have been lost, save for the limited 30 minutes of battery power.
As it turned out, the CVR/FDR were not connected to the emergency power, so they shut down when the airplane was some 65 miles from Lajes. The last 19 minutes of the engine-out approach and landing were lost.
Not all the emergency oxygen devices – the little yellow cups – deployed to supply cabin occupants when pressurization was lost as a consequence of engine shutdown. Fortunately, enough extra were on hand to supply everybody.
The touchdown was hard, blowing eight of 10 tires. The blowouts are likely related to one or both of the following:
- The electrically operated anti-skid braking system was unavailable (one of the systems not served by emergency electrical power).
- Due to degradations in the flight control laws, the pilot may have understandably underestimated just how much round-out input force would be necessary and had little control over the energy on finals required (i.e., knots indicated airspeed) and available for an unpowered flare. From his point of view, he was test flying and coping with a different vehicle (and without electric trim). With no thrust, no flaps, and only partial slats deployed, speed could dissipate very rapidly toward the higher stall speed. In consequence, the aircraft initially bounced/ballooned and then “fell through” to the final heavy landing – with brakes already applied.
Not all the emergency exit doors could be opened. Door L3 remained stuck partially opened, and the emergency escape slide did not fully deploy. The problem was traced to improper pin installation, about which manufacturer B.F. Goodrich had issued an SB. Air Transat had not incorporated this SB at the time of the incident. These details about balky exit doors and oxygen equipment have come up before, attesting to continuing functional failures of emergency equipment.
Anyone who reads aviation accident reports quickly appreciates that most crashes evolve from a fateful sequence of sometimes seemingly obscure circumstances leading the participants to their fate. What distinguishes the two Air Transat pilots is that they broke the chain of bad luck mistakes, oversights, errors, or whatever it was and, with considerable skill and breathtaking coolness, conducted a long glide to their hot landing on the runway at Lajes. One of Pich�’s vivid memories is of pieces of the landing gear continuing to roll down the runway just after the airplane came to a stop.
Ultimately, Pich� and De Jager did what pilots really are supposed to do: stay cool and focused even when things have gotten way out of the proverbial box.
Meanwhile, some questions of design come to mind:
- Might a “using reserve fuel” alert raise pilots’ attention level if inroads are being made into fuel remaining? The eventual alerting was due solely to a fuel imbalance. One wonders whether a leak in a central interconnect line (from the aft trim tank, or in cross-transfer lines or in the dump-valve) that did not create an imbalance would have been noticed much, much later.
- Should a fuel imbalance alert come at a lower threshold, say, about half the 3-ton mismatch between the left and right tanks on the A330 system design?
- Are the ECAM messages sufficiently explicit? Consider, there is only a single letter “D” difference between trim tank transfer underway (TRIM TANK XFR) and trim tank transfer completed (TRIM TANK XFRD). This was not a high-salience change that would have drawn the crew’s attention. Perhaps the “D” should flash until cancelled.
- Should fuel-flow metering occur at the beginning (tank end) of the system, instead of downstream at the high pressure end? Such an arrangement might make more plausible the detection of an abnormally high rate of fuel flow. If the pumps are designed to maintain pressure in the line, in a heavy leak situation they may happily turn up the volume until the tank goes dry. After all, fuel leaks in ETOPS flights are potential killers. Why? Well, who would have predicted that a 3-inch by 1/8-inch crack could lead to a loss rate of 13 tons of fuel per hour?
There is no guarantee, as amply demonstrated in this case, that pilots will pick up a fuel loss-rate leak early enough. No one can assure that the fire hazard would not be significant? No one can guarantee that another five tons of fuel will be tankered over and above ETOPS-related fuel reserves, as was the case here to save on the cost of fuel at Lisbon and concomitantly on hand to save one’s proverbial bacon.
- Is there a case here for having a gravity-fed fuel-header tank for the auxiliary power unit (APU)? At the very least, a functioning APU would keep the CVR and FDR operating. Moreover, the crew would have been in dire straits in poor weather (not the case here) without the additional APU electrical power provided to cockpit displays and aircraft systems.
- Are FDR parameters adequate to unravel the subtleties of such cases as this? Although the FDR on the incident aircraft recorded 450 items, oil pressure and temperature were not among them. Nor did the installed FDR record fuel pump status or the position of the crossfeed valve.
The FDR was capturing fuel flow but was keeping that abnormally high rate to itself.
The absence of an independent backup electrical supply meant the loss of 19 minutes of precious data. Following the six-minute gap in recorder data that complicated the Swissair accident investigation, the Transportation Safety Board (TSB) of Canada called for a 10-minute independent backup power supply. This recommendation was seconded by the U.S. National Transportation Safety Board (NTSB). Both agencies called for the installation of this improved capability by January 2005 (see ASW, March 15, 1999). That date is fast approaching and the improved capability seems unlikely to be met.
- Is a review now in order of the pre-programmed hierarchy of ECAM messages? The Portuguese report says that the FUEL ADV (advisory) message is only likely to occur because of a significant fuel leak. Since a fuel leak is a high-risk situation, this would suggest a change in color of the message to indicate a high risk (e.g., red, a color used now to indicate a “serious parameter exceedance” requiring “immediate crew action”).
Finally, the Portuguese report identified a significant computer-related risk quite different from the fuel imbalance. Namely, when the pilots realized that the ECAM was showing a seven-ton discrepancy in the fuel on board, they misdiagnosed this as a computer automation anomaly, because it seemed so unreal, rather than for the reality that it was. The crew stated that they persisted in this mistaken belief until the right engine flamed out, some 40 minutes later. That was a dangerously, almost fatally long time in which to believe in a computer malfunction, underscoring once again the human factors liturgy of mistaken mental models, framing bias, confirmation bias, high workload and such.
In its totality, the Air Transat case demonstrates that exalted ETOPS maintenance standards can turn out to be papier-mach� in practice, that simulator training can be too prescriptive and unimaginative and, ultimately, that it’s possible for a computerized system to mask the fact that all the fuel is exiting the airplane, stage right. In this day and age, any system that does not continuously integrate fuel on board with fuel used and fuel remaining (with ongoing discrepancy comparisons between calculated fuel remaining and actual fuel remaining) may be seriously undershooting what is required and what is possible.
(The full Portuguese report of investigation may be viewed at http://www.GPIAA-portugal-report.com)