‘To make an intractable problem go away – redefine it.’ – Anonymous
The plan to provide a stable vision of progress toward a higher level of safety is itself unstable, with some measures of safety performance dropped and others likely to be modified significantly to put the best face on an expected increase in near mid-air collisions. The document in question is the Federal Aviation Administration’s (FAA) Flight Plan 2005-2009. The plan, now in its second year of publication, lays out the agency’s goals for the next five years.
The latest edition drops the prototype safety index that was such a prominent feature of last year’s plan and, with an anticipated increase in air traffic control operational errors, FAA officials said statistics will be revised to feature rates instead of absolute numbers.
“We’d like to use rates more,” said FAA Administrator Marion Blakey. The drive behind this shift is evident in the latest statistics for the most serious types of air traffic control “operational errors” or, in lay terms, near misses. The actual numbers of such errors exceeded FAA targets for each of the three months in the last fiscal year ending September 2004.
Nevertheless, the second edition, Flight Plan 2005-2009, lays out the overall focus of FAA efforts over the next five years. Russ Chew, chief operating officer of the new Air Traffic Organization (ATO), said, “The flight plan ensures continuity beyond changes in leadership.”
The plan features 30 performance targets divided into four groups: safety, increasing capacity, international leadership and organizational excellence. Blakey pointed out that the agency hit 24 of its 30 targets for Fiscal Year 2004, which ended in September, for an 80 percent success rate. She added that performance bonuses for FAA officials are based on achieving 90 percent of the yearly safety goal for which they are responsible, thereby providing a built-in incentive to achieve the target.
In the safety arena, the box score is better than 80 percent, with the FAA claiming it has met targets in eight of its nine goals, for an 88 percent success rate. Two of those goals deal with injuries and damage to civilians and property in FAA-licensed commercial space launches. The 13 such launches in fiscal 2004 could be likened to a grain of sand relative to the entire beachfront of commercial and private airplane activity. Nevertheless, even if the two goals (personal injury/property damage) relative to space launches are deleted from the target list, leaving strictly airplane-related activity, the fiscal 2004 score still comes in at success in six out of seven safety metrics, for a rate of 86 percent.
The view of the National Transportation Safety Board (NTSB) is quite different. Last week the NTSB downgraded its assessment of FAA progress in three “Most Wanted” aviation safety improvements from “Open – Acceptable Response” to “Open – Unacceptable Response.” Those items involve aircraft fuel tank explosions, airframe icing and runway safety. If these items are matched to the FAA’s seven aviation-specific performance measures, only five of the items could reasonably be characterized as meeting goal, for a success rate of about 70 percent.
Indeed, the question comes to mind about the goals the FAA has set. Some targets are clearly more important than others in terms of their potential for loss of life and aircraft.
Accidents in Alaska and wayward space launches are clearly niche aspects of the aviation system at large. On the other hand, runway safety, the risk of mid-air collisions, and the safety record in general aviation are big issues meriting attention in the Flight Plan. In addition, the NTSB’s “Most Wanted” safety improvements deal with chronic, system- wide issues. The highest-visibility items on the NTSB’s list are subsumed under the FAA’s general goal to reduce fatal airliner accidents. A case could be made that the NTSB’s “Most Wanted” items could be elevated to comparable safety goals in the FAA, replacing some of the less important goals. A conceptual taxonomy might focus more attention on those safety shortcomings posing the greatest risk of injury or death.
In the meantime, the status of current FAA safety targets is as follows:
Target: Reduce the commercial airline fatal accident rate. The goal for fiscal 2004 (ended Sept. 30) was a rate not to exceed 0.028 fatal accidents per 100,000 departures, which equates to about one accident every 3.5 million flights. The 0.028 target worked out to not more than three fatal accidents in fiscal 2004. Two fatal accidents occurred, prompting FAA officials to hail 2004 as the safest year ever for airline travel.
Significantly, the FAA is measuring only fatal accidents, while the National Transportation Safety Board (NTSB) measures safety in terms of both fatal and nonfatal accidents. By this measure, the safety board shows a rate of fatal and nonfatal accidents that is roughly 80 percent higher than the FAA’s bookkeeping: roughly an accident for every 1.9 million flights. The latest NTSB statistics show airliner accidents occurring at an average rate of one per week for calendar 2003, which overlaps fiscal 2004 by nine months. The NTSB’s accident rate for 2003, based on 52 accidents in the year, was the highest since it began keeping score in 1984 (see ASW, March 29). Thus, the FAA and the NTSB see two completely different pictures. From the FAA’s perspective, fatal accidents are declining by number and rate. From the NTSB’s perspective, accidents overall are on the increase.
Basically, the FAA is ignoring all the warning shots from non-fatal accidents and counting only those that kill. The counting of people after they are dead may represent the ultimate in the oft-criticized tombstone approach to safety. Addressing more attention to events before fatalities occur might prompt a more proactive and preventive approach to safety.
Moreover, the death rate is not considered a measure of health in the medical system, because deaths do not indicate the general level of morbidity or disability in the population. Rather, in developed countries with adequate statistical systems, composite measures are used to reflect the expectancy of living free of disability.
Neither the FAA nor the NTSB measures include incidents, those events lower on the scale of damage and injury often differentiated by circumstance from accidents – and which often are harbingers of future disaster.
In a recent study of the contribution of maintenance error to accidents and incidents, Prof. Gary Eiff of Purdue University noted that while incidents are trending downward, the total number of accidents and incidents taken together is holding relatively constant. In other words, more accidents are occurring within that total. Thus the criticality of outcome “is becoming more severe.” (See ASW, May 3)
Clearly, the method by which system performance is measured can determine the results – which in turn shape perceptions.
Target: Reduce the general aviation (GA) fatal accident rate. Again, the FAA’s measure is of fatal accidents only. The FAA had set a fiscal 2004 target of no more than 349 fatal GA accidents, and came in at 340 actual accidents. The goal is to reduce this number to not more than 319 fatal GA accidents by fiscal 2009. However, the FAA is tracking only a fraction of the GA accidents. The NTSB database includes both fatal and non-fatal GA accidents, and its latest data, for 2003, show some 1,700 GA accidents for the year. In its Sept. 3 press release on the 2003 results, the NTSB said, “Aviation is the only transportation mode that showed no decrease in any subcategory. The total number of people killed in aviation accidents climbed from 625 in 2002 to 707 in 2003. General aviation accidents showed the largest increase in the aviation area, increasing from 581 to 626.” Thus, even though the FAA is following only fatal GA accidents, its goal of an 8 percent reduction is in that area of aviation activity where the overwhelming majority of deaths occur.
Target: Reduce GA and air taxi accidents in Alaska. When it comes to bush pilot and related operations in Alaska, the FAA includes both fatal and nonfatal accidents in its statistics. The FAA had a fiscal 2004 target of not more than 125 such accidents in Alaska, and came in at 100. Its Flight Plan 2004-2005 shows a target of not more than 99 accidents. Given that this record virtually has been achieved, future editions of the plan might reflect a more ambitious goal
Target: Reduce the risk of runway collisions at towered airports. Technically known as “runway incursions,” the most serious categories of these events involve the risk of aircraft colliding with one another. The FAA goal for fiscal 2004 was not more than 40 such events, and this performance target was bested by a considerable margin, with 28 such events. As above, the target for fiscal 2009 is 27 high-risk incursions, a goal that effectively has been achieved, at least by FAA record keeping.
The NTSB perception of the risk is quite different, as was evident in the board’s reclassification of its “Most Wanted” recommendation regarding runway safety. The FAA scorecard does not include the Aug. 19 near collision of an Asiana Airlines B747 and the Southwest Airlines [LUV] B737 at Los Angles International Airport (see ASW, Aug. 23). The Asiana jet roared over the Southwest jet, which had been cleared for takeoff on the same runway the Asian jet had been cleared to land.
“This highly visible incident was not reported,” said NTSB Member Debbie Hersman. “It may suggest a deeper reporting problem.”
The case also illustrates the relative ineffectiveness of the centerpiece system the FAA is deploying to reduce such hair-raising events – the AMASS, or airport movement area safety system. The AMASS at Los Angeles did not alert the tower controller until the Asian jet was within four tenths of a mile from the runway and the distracted controller, who was issuing another clearance, did not warn the Southwest pilots to cancel takeoff until 10 seconds later. The NTSB has called for a system that provides an alert directly to the pilots. Indeed, while intended to prevent accidents, AMASS recently caused an accident.
Target: Reduce cabin injuries from turbulence. The FAA had a target of not more than 18 such injuries for fiscal 2004, and the actual number was nine – with eight deemed serious. An FAA background paper said, “Only eight serious injuries have resulted from turbulence this year. We believe one of the factors for this below-average trends is that aircrews have been providing announcements to passengers related to turbulence that indicated an enhanced awareness of the risk.” Accordingly, the FAA paper said, “The FAA is removing this performance target from the Flight Plan, although [turbulence injuries] will continue to be tracked.”
However, the FAA database may significantly undercount the number of turbulence-related injuries. For the 40-week period from last February to the present, at least 33 cases of passenger and flight attendant injuries have been reported in the accident and incident table of this publication. At least 18 of these incidents involved serious injuries.
Target: Measure the safety of the U.S. civil aviation industry with a single, comprehensive index. The FAA has dropped back from the graphical prototype index featured prominently in last year’s Flight Plan. Now, the FAA hopes to implement such a measure by fiscal 2006 – a one-year delay. The prototype, which used relative risk per flight hour, generated a range of criticism (see ASW, Sept. 15, 2003). On the negative side, the safety index was criticized for including too much in the bucket – cargo, international operations, general aviation, and so forth. Rather, the index would be more useful if “the system” were defined as Part 121 scheduled passenger operations in the national airspace system – U.S. airspace to U.S. certified airports, by U.S. airlines.
At the other end of the range of commentary, it was suggested that the prototype safety index was analogous to the Dow Jones index of stock performance. The Dow Jones embodies sub-indices of various industry sectors, and the same approach could be taken to measuring the safety of the aviation system, according to this school of thought. This kind of tabulation would provide a more informed insight into the overall system’s performance. For example, in addition to the FAA’s goal of a single, comprehensive index of safety, it would be a straightforward exercise to disaggregate the data into the four major areas of operations: (1) scheduled passenger and cargo service (Part 121), (2) scheduled regional operations with planes of 10 or more seats (Part 121), (3) scheduled flights in aircraft with nine or fewer passenger seats and unscheduled aviation operations (Part 135), and (4) general aviation (Part 91).
Target: ensure the safety of commercial space launches. More specifically, the target is twofold: no fatalities or serious injuries, and no significant property damage to the uninvolved public. The Flight Plan indicates that risk analysis tools are being refined. Of 13 FAA-licensed launches in fiscal 2004, the Flight Plan said, “Four involved Scaled Composites’ SpaceShipOne. These flights were the first licensed launches of a reusable launch vehicle, and the first manned commercial rocket launches. As part of the license evaluation process, [the FAA] reviewed the planned launch vehicle trajectories to ensure that the risk to buildings or other facilities would be minimized.”
Target: enhance the safety of the air traffic system. Reduced to its essentials, this target involves mitigating the risk of midair collisions by lowering the number of air traffic control operational errors to 563 by fiscal 2009 (a 10 percent reduction). The FAA did not meet its fiscal 2004 target of 629 close calls, logging 637. That yearly figure masks the fact that each of the monthly targets for the last quarter of the year were not met. The September results included two separation losses as a result of a massive radio outage at the FAA’s Air Route Traffic Control Center (ARTCC) in Palmdale, Calif. (see ASW, Sept. 20).
Chew said, “We handled a lot more traffic than expected when the [original] plan was built.” The volume has surged as operations return to their pre-9/11 level. Chew pointed out that the rate of operational errors decreased in fiscal 2004, and therefore the FAA is evaluating the use of rates as the measure of performance.
The FAA believes there are several reasons why the absolute number of high-threat operational errors is on the cusp of an unfortunate increase:
- Changes to the mix of aircraft in the fleet.
- The advent on Jan. 20, 2005, of domestic reduced vertical separation minima (DRVSM). This program will halve vertical separation to 1,000 feet at flight altitudes between 29,000 and 41,000 feet, from the 2,000-ft. vertical separation that applies presently (see ASW, Nov. 3, 2003). Two of the aircraft involved in the ARTCC radio outage of last September passed other aircraft at a vertical separation of 1,000 feet or less, triggering on-board alerts from traffic alerting and collision avoidance (TCAS) systems – which are not required for operations in DRVSM flight space.
- Implementation of an aviation safety action program (ASAP) for air traffic controllers, by which more controllers may be acknowledging errors through this confidential reporting system.
- The aging controller workforce and the entry of new hires. In other words, a combination of burnout and inexperience may soon contribute to an increase in controller errors.
The FAA background paper said the increase in expected operational errors “and the level of risk in the system will need to be further defined.”
“The Office of Safety Services is currently reviewing operational error and performance data to formulate new performance targets that take into account changes in the NAS [national airspace system] since the original targets were established,” the FAA paper said.
The validity of the data serving as the basis for this review has been challenged. In a Sept. 23, 2003, report to Rep. James Oberstar (D-Minn.), ranking member of the Transportation and Infrastructure Committee, the General Accounting Office (GAO) said, “The possibility exists for underreporting by air traffic controllers.”
“The DOT IG [Department of Transportation Inspector General] found that in one instance [the] FAA rated an operational error as moderate that was less than 12 seconds from becoming a midair collision,” the GAO said.
Indeed, the Jan. 21, 2003, DOT IG report to which the GAO referred, called for a reexamination of the new severity rating system implemented by the FAA. “Some operational errors still pose a significant safety risk, with an average of three operational errors per day and one serious error every three days (in which a collision was barely averted),” the DOT IG report noted.
Given the dubious completeness of the raw numbers and the severity scores, two views might be taken of the development to use an error rate rather than the number of errors. One is that, having established a modest goal to reduce operational errors by 10 percent over five years, a target that was not met in fiscal 2004, and with the looming implementation of DRVSM, the FAA plans to finesse the statistics, using rates to blunt the safety implications of a rising number of operational errors.
The other view is that precision comes from variety, and that the use of error rates provides an additional perspective on system performance.
By this argument, to harken back to some of the other safety metrics, non-fatal accidents ought to be included in Part 121 and GA measures of safety performance. (For full text of Flight Plan 2005-2009, see http://www.faa.gov/aboutfaa/FP-final.pdf. A compilation of turbulence-related injuries may be viewed at http://www.iasa.com.au/turb.htm )
Zero For Three
NTSB actions Nov. 10 on “Most Wanted” aviation safety recommendations:
Fuel/Air Vapors in Fuel Tanks: preclude operations of transport category aircraft with explosive vapors in fuel tanks.
- Placed on “Most Wanted” list: May 1997.
- Previous status: Open – Acceptable Response
- Revised status: Open – Unacceptable Response
- Rationale: Lack of FAA initiatives on interim measures. Overall implementation progressing too slowly.
Airframe structural icing: reduce the dangers of flying in icing conditions.
- Placed on “Most Wanted” list: May 1997
- Previous status: Open – Acceptable Response
- Revised status: Open – Unacceptable Response
- Rationale: The oldest icing recommendations on the list date back eight years. FAA lack of progress cited.
Runway incursions: install a system that provides warning of probable collision directly to cockpit crews.
- Placed on “Most Wanted” list: Sept. 1990
- Previous status: Open – Acceptable Response
- Revised status: Open – Unacceptable Response
- Rationale: Aug. 19 near disaster at Los Angeles between a B747 and a B737 was not reported to the system, and reflects inadequate AMASS warning to the tower, not to the pilots. Chairman Ellen Engleman-Connors said, “The fact that such incidents are not being reported casts doubt on the FAA’s claims that the runway incursion rate is declining.” Source: NTSB
AMASS Causes Accident
From the NTSB preliminary report (extracts, edited for brevity):
On Aug. 7, 2004 United Airlines Flight 1223, a B727, aborted takeoff at Denver International Airport. According to the FAA, takeoff clearance was cancelled due to a target detected by the Airport Movement Area Surveillance System (AMASS) radar. The AMASS had been experiencing problems the day of the accident and the target that was detected did not exist. Damage to the airplane included several blown tired and melted brake fuse plugs.
Source: NTSB. Case No. DEN04IA124
Serious Turbulence Related Injuries
A sampling from the past 40 weeks of ASW accident & incident tables. Our definition of serious: the word “serious” is contained in the original report, or references to broken bones, stitches, medical diversions and/or visits to medical facilities
| Date/Operator | Summary |
|---|---|
| Nov. 7 ATA Airlines B737 | Encountered turbulence in cruise. Numerous neck, back and leg problems. Five injured flight attendants taken to hospital. |
| Nov. 5 American Airlines A300-600 | Encountered severe turbulence at approx. 21,000-23,000 ft.; five flight attendants injured, one seriously. |
| Nov. 4 Atlantic Southeast ATR-72 | Encountered moderate to severe turbulence at 11,000-12,000 ft., injured flight attendant taken to hospital. |
| Oct. 23 United A320 | Three flight attendants and one pilot hospitalized after hitting clear air turbulence. |
| Sept. 29 Delta B767 | Encountered turbulence on descent, flight attendant standing in aft galley sustained serious injury. |
| Sept. 13 Southwest B737 | Wake turbulence on approach, flight attendant suffered broken ribs. |
| July 17 America West A319 | Flight attendant suffered broken pelvis in severe turbulence encounter during descent. |
| May 26 American MD-82 | Encountered turbulence at FL170 and flight attendant suffered broken pelvis, ribs, wrist. |
| May 25 Delta B757 | Flight attendant struck by beverage cart in moderate turbulence, medical diversion. |
| May 10 Southwest B737 | Flight attendant seriously injured when thrown against a bulkhead in turbulence. |
| April 20 Pacific Wings C208 | Belted passenger sustained severe scalp lacerations. |
| March 1 United B757 | Moderate turbulence on descent, one flight attendant sustained broken leg. |
| Feb. 23 United B777 | Hawaii-Japan flight hit severe turbulence about 20 minutes from landing, three crew members hospitalized. |
| Feb. 18 Continental B737 | Experienced sever turbulence, a number of flight attendants were taken to the hospital. |
| For additional details of these cases and the complete compilation of turbulence-related injuries, see http://www.iasa.com.au/turb.htm | |
An Alternative Taxonomy
(Note: this is not a one-to-one match up)
FAA Safety Performance Targets
- Reduce airline fatal accident rate.
- Reduce GA fatal accident rate.
- Reduce accidents in Alaska.
- Reduce runway incursions.
- Produce a composite index of system-wide safety.
- Reduce risk to uninvolved persons and property from commercial space launches.
- Reduce risk of midair collisions.
Suggested Alternative Performance Targets*
(Note: robust safety programs repeatedly have demonstrated that they save more money than they cost. As examples, see ASW, Dec. 8, 2003 for Air Transat, and ASW April 15, 2002, for Delta Air Lines.)
- Reduce overall airline and GA accident and incident rate by focusing on non-fatal as well as fatal events.
- Install system(s) to reduce or eliminate risk of fuel tank explosions, progress to be measured as a percentage of fleet installation, Part 121.
- Reduce the risk of in-flight icing, and the number of accidents and incidents therefrom, for all categories of aviation by strengthening pilot education and improving ice protection system(s) performance.
- Reduce the accident and incident rate in air cargo operations, to include scheduled carriers and those operating under supplemental rules. Introduce cockpit take-off performance monitors as a supporting initiative to achieve this goal (see ASW Nov 1).
- Reduce the number of maintenance-related accidents, incidents and turnbacks, all categories, through strengthened oversight and by resolving the ongoing Service Difficulty Report (SDR) system review (see ASW, Jan. 20, 2003).
- Reduce the number and rate of the most serious runway incursions, at both towered and non-towered airports by deploying a Dreadle-type system that provides a direct warning to pilots (see ASW, July 16, 2001).
- Reduce the risk, by number and rate, of separation losses posing the greatest risk of midair collisions.
- Reduce the number of unscheduled landings for smoke and fire, especially those involving electrically related high temperature events by implementing a comprehensive detection and protection strategy, as called for by the Transportation Safety Board (TSB) of Canada (see ASW, March 8 and, more particularly, ASW, Nov. 1, 1999).
- Reduce the problem of post-accident data deficiencies by installing dual flight data and cockpit voice recorders with independent power backup on all Part 121 aircraft, and install cockpit video recorders on all aircraft carrying passengers for hire. Progress to be measured as a percentage of fleet installation.
- Expand air transportation oversight system (ATOS) to cover all Part 121 certificate holders, to include regional and cargo operators. Use percent of certificate holders under ATOS as performance target.
- Require Part 121 carriers to implement safety management systems (SMS), flight operations quality assurance (FOQA) and related aviation safety action program (ASAP), moving beyond voluntary implementation to achieve more robust “data-driven safety.”
- Reduce injuries from in-flight turbulence through improved data collection, crew education and procedures.
- Reduce the incidence of aircraft damage from bird strikes through strengthened airport wildlife management programs (see ASW, Sept. 21).
- Improve workplace safety by reducing the incidence of lost workdays per 100 employees (8.4/100) to achieve parity with average for all industries (2.1/100), with initial focus on flight attendant cabin injuries and injuries suffered by maintenance personnel falling off work stands, platforms, ladders, etc. (see ASW, May 21, 2001, and ASW Feb. 17, 2003).
*This program may support another major goal in the FAA’s Flight Plan 2005-2009, which is to strengthen the agency’s international leadership role. Source: FAA Flight Plan 2005-2009, ASW