Danger of automation: It makes us complacent

By Beth Azar
Monitor staff

In the summer of 1995, the Panamanian cruise ship Royal Majesty ran aground off the coast of Nantucket Island. The ship drifted off course because its satellite-based navigational system silently failed, and the crew did not monitor other sources of information that would have warned them they were in danger.

Luckily, no one was hurt, but the mistake cost Majesty Cruise Lines $7 million in damages and lost revenue.

It would be easy to blame the crew for failing to monitor their status more closely. But the automated navigation system may be as much at fault, lulling the crew into a false sense of security, according to human factors psychologists.

Automation changed the sailors? work from an active task to passive monitoring of the system?s status. Unfortunately, people tend to be poor monitors, finds psychologist Raja Parasuraman, PhD, of the Catholic University of America. The more they trust a device, the more complacent they become and the less likely they?ll notice when something goes wrong. In contrast, the less people trust an automated devise or warning system the more likely they are to disregard it or turn it off.

These all-too-human reactions to automation are examples of why engineers need to better consider the human operator when designing automatic systems, say human factors psychologists.

'The person?s view that automation simply replaces human functions is not quite correct,' says Parasuraman. 'Rather, automation fundamentally changes the nature of the cognitive work that humans do, often in ways that were unintended or unanticipated by the designers of automation.'

'When you listen to concepts of automation proposed by engineers, it makes intuitive sense that they will make people?s jobs easier,' adds Christopher Wickens, PhD, of the Institute of Aviation and department of psychology at the University of Illinois, Urbana?Champaign. 'But there are little insidious problems that aren?t easy to grasp. You have to accept the fact that on some rare occasions, nearly all automation fails or people fail to operate it properly?in other words, humans abuse it. And when that happens in high-risk systems, the consequences can be disastrous.'

Psychologists are trying to prevent disasters by infusing human factors into the design of automated systems. Their research is demonstrating where current automation fails and how it can be improved to maximize the benefits and minimize the risks.

Unforeseen risks

Automation is often added to a job to improve efficiency and safety, and lessen the cognitive strain on workers. But an unintended consequence is often understimulation. For example, automatic navigation and monitoring systems reduce cognitive load on pilots during high-demand portions of a flight, such as take-off and landing. But they leave pilots with little to do during the main part of the flight. For long-duration commercial pilots, the tedium can be significant, says Mark Scerbo, PhD, a psychology professor at Old Dominion University.

'The human mind didn?t evolve to be unstimulated and if it is, it provides its own stimulation?daydreaming,' says Scerbo.

As a consequence, when something goes wrong or an automated system fails, people are often slow to notice and then to react, find Parasuraman and his colleagues. Such poor monitoring can also occur when people are over-stimulated?they begin to over-rely on an automated system because they?re engaged with other tasks.

People who over-rely on automated systems tend to use them as shortcuts for decision making?like the crew of the Royal Majesty, they trust the computer to keep them out of danger, rather than double-check against other indicators such as a compass or visual cues, find psychologists Kathleen Mosier, PhD, of San Francisco State University and Linda Skitka, PhD, of the University of Illinois?Chicago. They?ve termed this phenomenon 'automation bias' and in several studies of college students and professional pilots they find that pilots tend to rely even more on automation than students, perhaps because they?ve grown to trust automation more, says Mosier.

Adaptive automation

Another risk of over-reliance in automated systems is a gradual decline in manual skills, says Parasuraman. Without practice, people lose their ability to react effectively when an emergency arises.

'You have to keep humans in the loop, honing their skills,' says Wickens.

That?s where 'adaptive automation' comes in. Researchers have begun testing the idea that an ideal automated system is one that changes its functioning in response to how the human operator is working. The system takes on tasks when the operator is overloaded, and gives them back when the operator is underloaded, keeping people at a stable level of engagement and allowing them to hone their skills on each task, says Scerbo. He and his colleagues at Old Dominion and NASA Langley Research Center are developing methods to monitor workload so systems can automatically take over tasks.

To test their ideas, they use a series of flight simulator activities that resemble work pilots might do, such as tracking a flight pattern, monitoring gauges and monitoring fuel in two different tanks. Participants have to do either one, two or three tasks at a time, and a computer monitors their cognitive load by measuring brain activity through electrodes attached to their scalps. When brain activity increases to an overload level, the computer takes over one of the tasks.

The researchers find that people perform the overall flight simulation task better during adaptive automation than when doing the task completely on their own. The research is currently in the feasibility stage?the electrode technology is still too cumbersome for use in real-life situations. However, Scerbo believes that within 10 years there will be a system available that would be small enough to be hooked into standard pilot headgear.

'We tell people to approach this from a team perspective,' says Scerbo. 'The computer is a team member who covers when you need it to and steps out of the way when it?s not needed.'

Keeping humans in the loop

Complacency and skill degre-dation are just two examples of many widespread effects of automation on human performance that psychologists have discovered.

Slowly, their human factors perspective is creeping into automation design, they say. The most inroads have been made on the flight deck?Boeing and NASA have human factors experts working on their design teams, says Wickens. And the National Research Council has published two books recommending ways to incorporate human factors into the design of the new, more automated air traffic control system being considered in the United States.

But human factors must be integrated from the start of a new technology, says Wickens. And engineers have to realize that just because they can automate something doesn?t mean they should.

People also need to remain actively involved in their work in order to stay alert and to remain psychologically involved in their jobs, says Wickens, speaking from a philosophical standpoint.

'What we?re trying to do is say ?don?t design the human out of the loop, keep him actively involved in making decisions,?' he says. 'What happens in a completely automated system if the power goes out? How well will humans be able to recover? Not well if they?re complacent and have lost their skills.'