You smile when you’re happy, frown when you’re sad, scowl when you’re angry—at least, some of the time. After decades of research on emotions, evidence suggests that these signals are a far-from-foolproof way to infer someone’s state of mind.
“Facial movements are not a language that you can read the way you read words on a page,” says Lisa Feldman Barrett, PhD, a distinguished professor of psychology at Northeastern University who studies emotion in the brain.
That realization has implications for researchers in affective computing who are working to design artificial intelligence (AI) systems that can identify a user’s emotional state. These emotion-recognition technologies could have a wide range of possible uses, such as cars that can detect a distracted driver, robots that can help autistic children learn social skills and technology to help law enforcement predict the behavior of suspected criminals.
“Human emotion is involved in almost every interaction people have—even with computers. However, computers have not traditionally been programmed with emotional intelligence,” says Rosalind Picard, ScD, founder and director of the Affective Computing Research Group at the MIT Media Lab. “Human-computer interactions can be less frustrating when technology is equipped to show respect for people’s feelings.”
Barrett and Picard, who are speakers at APA’s 2020 Technology, Mind & Society showcase (see sidebar below), agree that by asking the right questions, and harnessing new technologies, researchers can take the science of inferring our emotions to a new level—one that might even revolutionize technology as we know it.
Basic emotions?
In the 1960s, clinical psychologist Paul Ekman, PhD, now professor emeritus at the University of California, San Francisco, launched the modern study of emotional expression with his work on the six “basic” emotions: anger, disgust, fear, happiness, sadness and surprise.
Using posed photographs of stereotypical expressions of those emotion categories, Ekman and other scientists showed that people in cultures around the world could identify these emotions with a high degree of agreement. A generation later, a meta-analysis of more than 100 studies confirmed these emotions were universally recognized at levels better than chance (Elfenbein, H.A., & Ambady, N., Psychological Bulletinopens in new window, Vol. 128, No. 2, 2002).
That body of work is strong evidence that these emotional expressions are widespread, if not universal, says Jessica Tracy, PhD, a professor of psychology who studies emotion at the University of British Columbia. “They’ve been shown across many cultures, blind people produce these expressions, and in many cases there’s cross-species evidence for these expressions,” she says.
But Barrett suggests that the way in which we express emotions is—unsurprisingly—more complex than recognizing a posed smile or frown. In a review of more than 1,000 published studies of facial movements and emotions, she and her colleagues found that the way people express and perceive these six emotion categories varies considerably across cultures, across situations and even within a single person (Psychological Science in the Public Interestopens in new window [PSPI], Vol. 20, No. 1, 2019).
Across groups including babies, congenitally blind people and people from remote, small-scale, non-Western cultures, the evidence for consistent basic expressions of emotion was weak, Barrett concluded. “We found that in urban, large-scale cultures, people do scowl in anger more than expected by chance, about 30% of the time. [That] means that 70% of the time they are moving their facial muscles in other meaningful ways when angry,” she says. “They might scowl when they’re concentrating, when they’re confused, when you tell them a bad joke or when they have gas.”
And the findings are comparable for the other emotions, Barrett says. “The findings are similar for the other five emotion categories we examined, and to the best of our knowledge, hold for any emotion category ever studied.”
That’s not to say facial expressions are meaningless. But their predictive value is limited, Barrett says. “Scowling in anger more often than chance gets you a publication in a respected journal, but it doesn’t necessarily mean scowling is a useful piece of knowledge to infer someone is angry and make decisions about that person’s life. Would you want someone to decide whether or not you get a job, whether or not you’re charged with a crime or whether your child has access to educational opportunities based on a system that is accurate 30% of the time?”
Much of the research on emotional expression focuses on how people perceive others’ expressions, often using photos of posed stereotypes of happiness or anger, for example. Those studies can reveal something about an observer’s perception of emotion, but the method doesn’t reveal anything about the emotions of the person who is smiling or scowling, Barrett suggests.
“If you want to understand how people move their faces when they’re expressing emotion, shouldn’t you be measuring how they actually move their faces, rather than measuring inferences people make about posed expressions?” she asks. “This debate about whether there are universal expressions has gone on for over a century. And I think that as scientists, the first thing we need to be asking ourselves is whether we’re actually asking the right questions.”
Head-to-toe expressions
If a human can’t reliably infer what someone is feeling based on their facial movements alone, then a computer algorithm programmed to detect those same facial movements won’t be up to the task either, points out Stacy Marsella, PhD, a professor and director of the Centre for Social, Cognitive and Affective Neuroscience at the University of Glasgow and a co-author of Barrett’s 2019 PSPI paper.
“But it isn’t very surprising to most researchers in affective technologies that a computer would need more than facial actions to make such inferences,” he says. Facial movements are one of many elements he and other researchers are using to design systems that can express and infer affective states, which include emotions as well as other emotion-related experiences, such as boredom and frustration.
“The goal of affective computing isn’t just to map facial expressions,” agrees Alan Cowen, PhD, a postdoctoral researcher in computational neuroscience at the University of California, Berkeley. “It’s much more nuanced than that, and the field has moved past the idea that you can just match facial expressions to the basic six emotions.”
Even within one emotion type, there is a multitude of variation, he points out. Consider amusement, joy and pleasant surprise: “There isn’t just one category of happiness,” he says.
And while a photo of a smile might make you think “happy,” facial movements aren’t static. A smile isn’t just a fixed state, Marsella notes. “You would also want to know the dynamics: How fast is it happening? How long is it held? Is it symmetric? What other facial actions are involved? These elements, along with the larger context in which the behavior occurs, tell you a lot.”
In their PSPI paper, he and Barrett champion a more comprehensive study of emotion, which would sample individuals more deeply, with higher-dimension measurements, across many different situations. Other researchers, including Cowen and Tracy, have also called for a more “comprehensive atlas” of human emotions, which would look beyond the face to consider other clues to a person’s feelings, such as the sounds they’re making and the way they’re moving their bodies (Cowen, A., et al., Psychological Science in the Public Interestopens in new window, Vol. 20, No. 1, 2019). Tracy, for instance, has found that head movements play a significant role in communicating emotions, especially anger. In a review with her student Zachary Witkower, she described how various body movements can telegraph pride, joy, sadness, shame, disgust and other feelings (Emotion Reviewopens in new window, Vol. 11, No. 2, 2019). Picture a person hanging their head in shame, or puffing up their chest in pride. “A lot of emotions are expressed and recognized through those behaviors,” she says.
Context matters
Affective computing also considers the context of the situation, and what’s happening in a person’s environment, says Picard, who coined the term “affective computing.” With a background in computer science and engineering, she sees the field of emotion research a little differently than psychologists might.
“For a long time, emotion researchers have argued about whether emotions are just clean and ‘basic,’ like happy and sad and angry, or whether they are continually changing dimensions,” she says.
Years ago, when she first began reading up on emotion theory, she was reminded of the debate in physics about the duality of light. “For a long time, physicists argued about whether light was a particle or a wave. They finally decided it’s both, and it just depends on what you’re trying to explain,” she says. “Emotion researchers could benefit from thinking about emotion this way, and allowing it to be context dependent.”
That fluid way of thinking about emotion might be one factor in Picard’s success in creating sensors and systems that can perceive affective information. She was a co-founder of the startup Affectiva, which develops emotion-sensing AI for use in gaming, robotics, education, marketing and health care. While she is no longer affiliated with Affectiva, the company has developed sensors to detect driver drowsiness and distraction and a computer game that helps kids with special needs develop social-emotional skills by imitating the expressions of characters they see on screen.
After leaving Affectiva, Picard co-founded Empatica with the goal of creating sensors and algorithms to improve health. One such device is an FDA-cleared smartwatch that monitors for seizures, a device born out of work to understand emotional responses in the amygdala. In the process of stimulating the brain to study emotion, researchers found physiological patterns that could be used by AI to detect and alert to grand mal seizures.
In other work, she’s partnered with psychiatrists to study devices to monitor moods in people being treated for depression. “With their informed consent, we’re taking information from wearables and their smartphones and using AI to see if we can tell whether they’re getting better or worse,” she says.
In an initial study of 12 people, she and colleagues found that these machine-learning techniques could predict depression ratings measured by the Hamilton Depression Rating Scale (HDRS) (Ghandeharioun, A., et al., Seventh International Conference on Affective Computing and Intelligent Interactionopens in new window, 2017). Now she’s testing the technology in a larger group of participants.
The project is a good example of the importance of context, she adds. “We know the person has a depression diagnosis. We have context from their phones: how much they’re getting out and about, how often they’re calling or texting others, how much they’re moving, the local weather, their sleep rhythm and physiology,” she says. “This isn’t just about counting smiles or reading one bio signal. But with all of this information together, we’re getting highly accurate tracking of their scores on the HDRS.”
New tech for emotion research
These projects provide good reason to be hopeful about the future of emotion-recognition technology. But so far, Picard cautions, the systems are reliable only in limited contexts. These technologies may be able to make useful inferences about a person’s affective state, but only in well-defined situations where the computer has a rich sample of all the possible inputs. In new contexts and completely unexpected situations, with people the computer has never “met” before, “then it is likely to run into all kinds of confusion and misunderstanding,” she says.
There are also important ethical issues to consider as the technology moves forward, such as collecting affective data without a person’s direct consent. “I’m in favor of regulations. We want to find the right balance of benefiting people while protecting their privacy and protecting them from being exploited,” she says.
With the right ethical safeguards, emotion-sensing AI could provide benefits, particularly as a research tool that can help plumb the fundamental questions about human emotion, such as whether emotions are truly universal, and how they are represented in the brain.
“If [emotion researchers] want to make real advances that will improve people’s mental health, physical health and behavior change challenges, there are ways to work together to advance both the theory and the practice,” she says. “There’s a lot to learn by teaming up with engineers and working together across the field to address some of the big theoretical questions in emotion.”
With enough data points, Barrett agrees that such systems have promise, especially in research domains. “There is a whole universe of questions [about emotional expression] that haven’t been asked properly. Sometimes in science, progress isn’t about answering questions. It’s about learning to ask better ones.”
