Memorandum from the Director of Committee On Human Factors, National Research Council

December 21, 2001

MEMORANDUM

To: Mr. Jack Donovan
  Federal Air Marshall Program, Federal Aviation Administration

From: Anne Mavor
  Director, Committee On Human Factors, National Research Council

Re: Selection and Deployment of Federal Air Marshals

The following summary provides a list of the key issues and points raised in your discussion with the Committee on Human Factors during its open meeting on Thursday, December 5, 2001. The discussion focused on your request for research relevant to the selection and deployment of Federal Air Marshals (FAMs) on domestic and international flights on U. S. carriers. A number of key human factors considerations were explored including the specific question you posed on the relationship between age and job performance of FAMs. The summary begins with the points raised on aging and performance and then addresses other relevant topics. The points made should not be interpreted as committee consensus. We would be happy to work with you on developing any of these lines in inquiry or in suggesting individuals or groups with expertise in these areas.

Aging and FAM Performance

The effects of age on the mental (cognitive) and physical functions relevant to FAM performance can be examined by surveying the research literature in experimental psychology and human factors. A consensus finding among researchers is that many functions show a decline in efficiency with adult age, whereas some functions remain stable or may even improve. The range across which such age-related changes can be observed is broad, from 30 to 75 years. These changes are typically observed by comparing groups of individuals, and cannot be used to set reliable age cutoffs for selection of individual workers.

Furthermore, physical and cognitive skills are comprised of many inter-related, sub-component functions that may erode at different rates. For example, reaction time to decide between one of two actions (shoot or not to shoot) may show a steady decline from age 35 onwards, whereas situation assessment may not show much change until post-retirement age. In addition, there are large individual differences in aging effects (individuals age at different rates) as a result of not only genetic factors but differences in physical conditioning, expertise, job responsibilities, general health, and so forth.

These considerations indicate that the existing research does not provide evidence for the use of a single age cutoff for any given cognitive function related to FAM performance, let alone the entire range of FAM-relevant functions. Advanced age is clearly a correlate of reduced performance in many tasks, but it is not a precise, initial screening criterion. Although age cutoffs for mandatory retirement do exist for other positions in the aviation industry, for example 60 years for commercial airline pilots and 55 years for air traffic controls, these are not based on empirical research on the effects of age on cognitive and job performance. It is also unlikely that differences of plus or minus a few years around 40 will have strong behavioral consequences for the FAM program.

Instead of a simple age cutoff, it may be useful to supplement any age criterion with ongoing, repeated behavioral tests of fitness for duty. To do this, the component functions of FAM performance must be dissected. The essential functions of the FAM job identified in our discussion with you included:

The physical strength and dexterity to operate special weapons, move efficiently in closed quarters, and combat or subdue passengers as necessary

The endurance to stay mentally alert for long periods of time in order to respond to an extremely rare critical event (vigilance)

The capacity for rapid movement after sustained periods of physical inactivity

Other mental skills such as the ability to divide attention, maintain situational awareness, perform risk assessment, and engage in rapid decision-making

Task analysis, a standard tool in the human factors community (consisting of a number of validated analytic methods), is a useful approach to identifying the important component functions. Once the important component functions of FAM performance have been identified using task analysis, then those functions that are most sensitive to aging could be examined and valid fitness for duty tests developed. Such tests are also relevant because there is likely to be considerable day-to-day variability in FAM performance, especially for older air marshals. This variability may be important to measure in addition to average levels of performance.

Other Human Factor Considerations

Also discussed at our December 5^th meeting were the enormous challenges faced by the FAM program with respect to desired staffing levels and the need for rapid program development. There are several human factors considerations that could effect decisions made in this important area. There are other disciplines such as organizational psychology that specifically study optimal methods for hiring new employees. It appeared from your presentation that many of the pre-selection variables being considered qualify more as screening devices – tools to disqualify some candidates on some criteria, with the goal of conducting more extensive assessments of the remaining candidates. However, these other assessments were not described. Again, a task analysis of the job functions could be used as a basis for designing a selection process that optimizes the selection of competent air marshals. There are standard practices for doing this and these can be used to accumulate data that can be used to validate the selection process. It is likely that FAMs will be with us for some time and, thus, there is an opportunity to set up the system in a way that could generate data valuable for evaluating the selection process. These data would be useful the adapting the selection process when and if there was a need for change.

Training

Training is a second area in which existing systems may be modified to improve air marshal performance. One important topic is simulation practice. Training research has shown that different practice conditions promote acquiring vs. generalizing skilled behaviors. Thus, early in training, it is useful to repeat conditions (e.g., location and actions of terrorists) to assist initial skill acquisition, while later in training it is useful to make conditions vary as much as possible across simulations to help trainees learn to apply learned skills in as broad an array of situations as possible. There is also a rich literature on team training and inter-team training that may apply to improving coordination among air marshals and between air marshals and other individuals on a flight responsible for safety.

Organizational issues were not fully discussed while you were in attendance but we revisited this area later in the day. This discussion focused on the notion that not only is the Air Marshal Program being asked to hire, train, and deploy a highly reliable set of individuals, but it is also being asked to build what is essentially a new, large organization. This requires decisions to be made about hierarchical reporting relationships; mechanisms for performance evaluation and quality control, and systems for long term organizational maintenance (e.g., ensuring in-house successors for leadership positions are properly trained). There are human factors and organizational psychologists who have methods that could assist in building and maintaining the program and in ensuring that the organization as a whole is structured to provide maximum control over air marshal performance and to ensure ongoing growth and maintenance of the system.

Another important area raised in the discussion was the task of inserting the FAM unit into an existing system that has been operating for some time. With such tasks, there is a tendency to focus internally on the unit for which you are responsible and overlook the interfacing issues. Attention to the interfacing issues will be important.

Public Perceptions

Another important issue discussed at the meeting concerns public perception of the FAM program. The public is unlikely to tolerate very many accidental shootings of innocent individuals, even if the system is effective in shooting actual terrorists. One example is the public's evaluation of airbags. The estimated ratio of lives saves to lives lost by the use of airbags was highly favorable (on the order of 100 to 1). However, public discontent over the lives lost was considerable, and threatened the continuation of airbag usage unless there were technological improvements. The major point is that innocent lives lost are weighted extremely heavily vs. lives saved in public opinion. This bias is important in designing the procedural safeguards used by air marshals, particularly if the system is going to survive over the long run.

System Integration.

Any procedures developed for the air marshals will need to be integrated carefully with other anti-terrorist procedures being considered by the FAA and other agencies. Given that this system of procedures is being constructed quickly, it would be beneficial to have a human factors specialist with experience in systems integration troubleshoot for unintended consequences of combining subsystems that seem fine when considered in isolation.

Back to Top^