Survey pits NASA administrators against researchers

Psychologists fear that key data they were gathering on airline accidents will end up being quashed by a NASA administration effort to cover up that information, ostensibly because it could tarnish the image of some pilots and airlines and frighten the public.

In 1998, NASA created the National Aviation Operational Monitoring Service, or NAOMS, to collect data on flight-safety concerns, with the goal of reducing the number of fatal airline crashes by 80 percent in 10 years.

Unlike the Federal Aviation Administration's (FAA) and other safety-related databases, which rely on anonymous pilot reports and cockpit "black boxes" to gather information, the $11.3 million NAOMS project was designed to routinely survey pilots, air-traffic controllers, flight attendants and mechanics on safety issues, such as pilot fatigue, runway congestion, engine problems and birds hitting the windshield.

In 2004, NAOMS researchers had finished collecting data on their first cohort of pilots-about 24,000 interviews-when trouble began to brew. They reported to the FAA preliminary findings suggesting that some safety-related problems seemed to be occurring at surprisingly high rates-in some cases four times those reported by the FAA. The FAA was "very unhappy" with that information and called for the project to be closed down, says social psychologist and Stanford University Professor Jon Krosnick, PhD, who helped to create the initial survey design and participated in that meeting.

Two years later, Krosnick learned that NASA had indeed shut the project down and planned to keep the data secret.

Krosnick wrote an op-ed piece on the situation that appeared in the Aug. 30, 2006 New York Times. At that time, Associated Press reporter Rita Beamish put in a Freedom of Information Act request for the squelched NAOMS data. After multiple denials, NASA said that the agency was suppressing the data because it could "materially affect the public confidence in, and the commercial welfare of the air carriers."

Beamish wrote an article revealing these events, and it was printed in hundreds of newspapers here and abroad. NASA Administrator Michael D. Griffin testified before the House Committee on Science and Technology, saying the project had been poorly conceived and executed, the methodology had not been peer reviewed, and the resulting data were unreliable.

At the same hearing, Krosnick rebutted Griffin's points in detail, adding that Griffin had terminated the project prematurely. (Krosnick noted that the preliminary findings showing a far higher rate of problems than expected could be partly the result of an incorrect statistical analysis that neglected to divide the number of reported problems by the number of pilots per aircraft who saw each one.)

Congress demanded that Griffin release the data by the end of 2007. On Dec. 31, he released it in a PDF format "that made it difficult if not impossible for outsiders to analyze it in search of trends," according to a Jan. 1 editorial in The New York Times. In response, Griffin re-released the data Jan. 14 in an easier-to-analyze Excel format. But members of Congress complained that NASA continued to withhold vital information and the file remained difficult to decipher.

In January, Congress was considering having the Government Accountability Office analyze the data, and Griffin had called for a National Research Council panel to evaluate the survey methodology.

"When the data are properly analyzed, I think we'll learn a ton," Krosnick says. "The survey is designed to collect a great deal of information that's not being gathered in any other way. Accident precursors that are witnessed by people but aren't tracked by black boxes happen all of the time."

                                                                                                                            —T. DeAngelis