Mapping the Social Space of the Face
By Alexander Todorov
Alexander Todorov received his PhD from New York University in 2002. Currently, he is an associate professor of psychology and public affairs at Princeton University with a joint appointment in the Department of Psychology and the Woodrow Wilson School of Public and International Affairs. He is also an affiliated faculty of the Princeton Neuroscience Institute. His research focuses on the cognitive and neural basis of social cognition. His work has been published in more than 20 journals, including Science, PNAS, Trends in Cognitive Sciences, Psychological Science, Journal of Cognitive Neuroscience, and Journal of Personality and Social Psychology. His work on evaluation of faces on social dimensions has been funded by the National Science Foundation. He recently received the SAGE Young Scholar Award from the Foundation for Personality and Social Psychology.
Johann Kaspar Lavater, the father of the “science” of physiognomy, described in detail how to read the true, inner nature of the person from facial features (e.g., “the nearer the eyebrows are to the eyes, the more earnest, deep, and firm the character,” (Lavater, 1772/1880, p. 59). Most people today find these ideas fanciful. Yet, research suggests that people do subscribe to a naïve physiognomy. Lavater was probably wrong about his specific hypotheses but he was right about one thing: “whether they are or are not sensible of it, all men are daily influenced by physiognomy” (p. 9).
First, research shows that people agree in their social judgments from faces (Hassin & Trope, 2000; Todorov, Said, Engell, & Oosterhof, 2008; Zebrowitz & Montepare, 2008), indicating that faces provide information that is interpreted consistently across perceivers. Second, research shows that social judgments from faces are made rapidly without much mental effort (Bar, Neta, & Linz, 2006; Willis & Todorov, 2006). For example, we have shown that as little as 33 milliseconds exposure to a face is sufficient for people to decide whether the face looks trustworthy or not (Todorov, Pakrashi, & Oosterhof, 2009). Third, regions in the brain seem to track the valence of novel faces even when participants are not engaged in explicit evaluation of the faces (Engell, Haxby, & Todorov, 2007; Todorov & Engell, 2008; Winston, Strange, O’Doherty, & Dolan, 2002). Thus, it appears that our brains are automatically categorizing faces. Finally, research shows that social judgments from faces predict important social outcomes ranging from sentencing decisions to electoral success (Blair, Judd, & Chapleau, 2004; Eberhardt, Davies, Purdie-Vaughns, & Johnson, 2006; Little, Burriss, Jones, & Roberts, 2007; Montepare & Zebrowitz, 1998; Olivola & Todorov, in press; Zebrowitz & McDonald, 1991), suggesting that appearance can affect one’s social outcomes. For example, we have shown that rapid judgments of competence based solely on facial appearance predict the election outcomes of both senatorial and gubernatorial races in the US (Ballew & Todorov, 2007; Todorov, Mandisodza, Goren, & Hall, 2005). This finding has been replicated in a number of countries (for a review see Olivola & Todorov, in press). In a recent, particularly dramatic demonstration of this effect, judgments of Swiss children predicted the outcomes of French parliamentary elections (Antonakis & Delgas, 2009).
Why do people routinely engage in personality inferences from faces if these inferences are not necessarily accurate? In an attempt to answer this question, we used data driven methods to uncover the underlying structure and perceptual basis of social judgments from faces (Todorov, Said, Engell, & Oosterhof, 2008). The objective of this approach was to address two of the fundamental questions of the study of social judgments from faces: what do these judgments really measure and what is their functional basis?
The structure of face evaluation
It is an empirical fact that social judgments from faces are highly correlated with each other. For example, it is almost impossible to find social judgments that are uncorrelated with judgments of trustworthiness (Oosterhof & Todorov, 2008). Moreover, as shown in Figure 1, these correlations are sizeable (.83 with judgments of emotional stability, .75 with judgments of attractiveness, -.76 with judgments of aggressiveness, .63 with judgments of intelligence). These high correlations suggest that there is a simple dimensional structure that accounts for most of the variance in social judgments.
Figure 1. Scatter plots of mean judgments of trustworthiness from emotionally neutral faces and A) mean judgments of attractiveness; B) mean judgments of intelligence; C) mean judgments of aggressiveness; and D) mean judgments of emotional stability. Each point is a face. Judgments were made on a 9-point scale. The lines represent the best linear fit.
To uncover the structure of social judgments from faces, our research strategy was to a) select a set of traits used to spontaneously characterize unfamiliar, emotionally neutral faces; b) have participants rate faces on these traits; and c) submit these ratings to statistical analyses that reduce the dimensionality of the data (Oosterhof & Todorov, 2008). Specifically, we first selected the most frequently used trait dimensions from unconstrained person descriptions of emotionally neutral faces. We then had separate groups of participants rate two sets of faces – one natural and one computer generated – on these dimensions. We finally submitted the mean face ratings on these dimensions to a principal components analysis (PCA).
For both sets of faces, the first two principal components accounted for about 80% of the variance of these face ratings. Moreover, as shown in Figure 2, the PCA solutions were very similar for both sets of faces. The first principal component, which accounted for about 60% of the variance, could be interpreted as valence of the face given its strong positive relationship with positive trait judgments (e.g., trustworthiness) and strong negative relationship with negative trait judgments (e.g., aggressiveness). The second principal component could be interpreted as power or dominance of the face given its strong positive relationship with judgments of dominance, confidence, and aggressiveness. This two-dimensional solution corresponds to other dimensional models of social perception (Fiske, Cuddy, & Glick, 2007; Vigil, in press; Wiggins, 1979; Wiggins, Philips, & Trapnell, 1989).
Figure 2: Plots of solutions of principal components analysis of trait judgments of a) 66 natural faces and b) 300 computer-generated faces. The first principal component (PC) could be interpreted as valence/trustworthiness evaluation, and the second PC component could be interpreted as power/dominance evaluation. The PCs are a weighted linear combination of trait judgments. The plots show the location of judgments of trustworthiness, dominance, threat, and attractiveness within the two-dimensional space of face evaluation. The smaller the angle between a trait dimension and a PC, the stronger is their relationship. The plots also show natural and computer generated faces with similar location in the 2-dimensional space. The length of each line represents 6 standard deviation units (+3/-3 SD relative to the origin).
These results suggest that social evaluation of faces can be sufficiently described using only two dimensions: the perceived valence of the face and the perceived power/dominance of the face. Specific trait dimensions can be represented within this two-dimensional space, radically simplifying the problem of understanding face evaluation. As shown in Figure 2, trustworthiness evaluation was extremely close to the valence component. In essence, the angle between two dimensions indicates the strength of the correlation between these two dimensions. In the specific case, the correlation between trustworthiness judgments and the valence component was greater than .90. These findings suggest that trustworthiness judgments are an excellent approximation of general valence evaluation of faces. In fact, using this two-dimensional model, we have shown that prior findings that the amygdala, a subcortical brain region critical for the affective evaluation of stimuli, tracks the perceived trustworthiness of faces (Winston et al., 2002; Engell et al., 2007) are more appropriately described in terms of tracking the general valence of faces (Todorov & Engell, 2008).
The perceptual basis of face evaluation
The next question was to uncover the variations in the structure of faces that lead to different face evaluations. To do this, we used a statistical model of face representation (Singular Inversions, 2005) originally developed at the Max Planck Institute in Germany by Blanz and Vetter (1999, 2003). In this model, faces are represented as points in a multidimensional space (see Fig. 3a and 3b). The underlying dimensions are empirically derived and independent of each other. The resulting statistical model of face representation can generate an unlimited number of faces that are each linear combinations of these dimensions.
Figure 3: Computer modeling of social judgments of faces. A) Illustration of how the face model represents faces. Left: A surface mesh with fixed topology superimposed on the average face. Right: an expanded view of a section of the mesh, along with direction vectors specifying the linear changes in the vertex positions for the surface for one of the m=50 shape dimensions. B) A set of n random faces can be obtained by linear combinations of the m shape components, and represented in an n by m matrix. These dimensions are extracted from a principal component analysis of shape variations of the vertex positions and do not necessarily have inherent psychological meaning. Each row of the matrix contains the set of m weighting coefficients corresponding to a particular face. C) Each of the n faces is rated by participants on a trait dimension and given an average score yj. Multiplication of the social judgments vector by the set of randomly generated faces yields a dimension that is optimal in changing faces on the trait dimension, which can be controlled with a tunable constant k. The figure shows the generation of one face along the trustworthiness dimension. D) A two-dimensional model of evaluation of faces. Examples of a face with exaggerated features on the two orthogonal dimensions – trustworthiness plotted on the x-axis and dominance plotted on the y-axis – of face evaluation. The changes in features were implemented in a computer model based on trustworthiness and dominance judgments of n = 300 emotionally neutral faces (Oosterhof and Todorov 2008). The extent of face exaggeration is presented in SD units. The faces on the diagonals were obtained by averaging the faces on the trustworthiness and dominance dimensions. The diagonal dimension passing from the 2nd to the 4th quadrant was nearly identical to a dimension based on threat judgments of faces. The other diagonal dimension passing from the 1st to the 3rd quadrant was similar to dimensions empirically obtained from judgments of likeability, extraversion, and competence.
For our purposes, what is important is that each face can be numerically represented by a set of values on these dimensions (Fig. 3b). Thus, it is possible to construct new dimensions in the multi-dimensional face space that are optimal in representing specific attributes (e.g., trait judgment). To conceptually understand this process, imagine a set of faces that are measured on a specific attribute (this could be a trait judgment or the wideness of the nose). Each face will have a measure on this attribute and the values on the dimensions that define the face space. The correlations between the attribute measure and the dimensions values determine the location of the attribute dimension in the face space. For example, if the attribute measure is highly correlated with one of the dimensions, then large changes along the attribute dimension will correspond to large changes along this dimension. The resulting attribute dimension is a linear combination of the dimensions defining the face space (see Fig. 3c). Another way to think about this process is in terms of a regression approach. The attribute values of the faces are regressed on the face dimensions values. The predicted regression line represents the dimension corresponding to the face attribute (e.g., perceptions of trustworthiness).
Given the similarity of trustworthiness and dominance judgments to the two underlying dimensions of face evaluation (Fig. 2), we decided to model variation of faces on perceived trustworthiness and dominance. We first randomly generated 300 emotionally neutral faces. We then had participants rate these faces on trustworthiness and dominance. Finally, we used the mean trait judgments to construct novel dimensions that were optimal in representing face variations along these social dimensions (Fig. 3d). Subsequent validation studies confirmed that judgments of trustworthiness and dominance tracked the trustworthiness and dominance, respectively, of faces generated by these dimensions (Oosterhof & Todorov, 2008).
There are two specific applications of these models of social judgments of faces. First, in principle, it is possible to generate an unlimited number of faces and parametrically manipulate these faces on the respective social dimensions (see Fig. 3d). Second, these models can be used to reveal the facial cues used by people to make specific social judgments. For example, by exaggerating the features that contribute to social judgments of emotionally neutral faces, it is possible to reveal the underlying variations that account for these judgments. One way to think about this process is in terms of creating a caricature of a face on the dimension of interest or in terms of amplifying the diagnostic signal in the face that is used for the specific judgment. In the case of valence/trustworthiness, although faces were perceived as emotionally neutral within a 3 standard deviations range, they were perceived as emotionally expressive outside of this range (see x-axis of Fig. 3d and trustworthiness movie). Specifically, as confirmed by subsequent studies (Oosterhof & Todorov, 2008), whereas faces at the extreme negative end of the dimension appeared to express anger, faces at the extreme positive end appeared to express happiness (Fig. 4). In the case of power/dominance, whereas extremely submissive faces were perceived as feminine and baby-faced, extremely dominant faces were perceived as masculine and mature faced (see y-axis of Fig. 3d and dominance movie).
Figure 4: Sensitivity of trustworthiness and dominance dimensions to cues resembling emotional expressions. A) Intensity color plot showing the categorization of faces as neutral or as expressing one of the six basic emotions as a function of their trustworthiness and dominance. B) Mean judgments of expressions of anger and happiness. The judgments were made on a 9-point scale, ranging from 1 (angry) to 5 (neutral) to 9 (happy). Error bars show standard error of the mean. The line represents the best linear fit. The x-axis in the figures represents the extent of exaggeration of facial features in SD units.
Movie 1: trustworthiness. Changes of facial features on the trustworthiness dimension. The running trustworthiness score is presented in standard deviation units in the lower right corner of the movie.
Movie 2: dominance. Changes of facial features on the dominance dimension. The running dominance score is presented in standard deviation units in the lower right corner of the movie.
The functional basis of face evaluation
The findings described above are consistent with the overgeneralization hypothesis (Zebrowitz & Collins, 1997; Zebrowitz & Montepare, 2008) or the idea that attributes that are accurately revealed by face qualities such as emotion and age could be erroneously perceived in people who merely resemble one of those categories. For example, one interpretation of our findings is that emotionally neutral faces that resemble happy faces are perceived as trustworthy. In contrast, neutral faces that resemble angry faces are perceived as untrustworthy (for convergent evidence see Montepare & Dobish, 2003; Oosterhof & Todorov, 2009; Said, Sebe, & Todorov, 2009). That is, resemblance to a specific emotional state may be misattributed to a personality disposition that is associated with this state (Said et al., 2009). Importantly, the overgeneralization hypothesis can account for rapid and efficient but not necessarily accurate trait judgments from faces. For example, to the extent that these judgments reflect misattribution of facial cues to stable personality dispositions, they need not be accurate.
In terms of our two-dimensional model of face evaluation, the behavioral and computer modeling findings suggest that face evaluation is an overgeneralization of adaptive mechanisms for inferring mental states. Specifically, face variations along the valence/trustworthiness and power/dominance dimensions may give rise to inferences about behavioral intentions and power hierarchies, respectively. Subtle resemblance of neutral faces to expressions that signal whether a person should be avoided (anger) or approached (happiness) serves as the basis of valence evaluation. Cues for physical strength such as masculinity serve as the basis of dominance evaluation. Another way of thinking about these types of cues – approach/avoidance and strength – is that they serve as a basis of rapid, initial inferences about intentions to cause harm and the ability to implement harm (cf. Fiske et al., 2007).
Potential applications of this research
Our research was motivated by general questions about the nature of social judgments from faces. However, our findings, and particularly the computer models, have specific applications. First, any researcher who has used faces as research stimuli is aware of the difficulties of finding sets of stimuli that are well standardized and meet the specific requirements of the research. Imagine a study that requires several hundred trials with different face identities that need to vary on a specific social dimension. As noted above, the models of social judgments can be used to generate an unlimited number of faces and to parametrically manipulate these faces. Although we described modeling judgments of trustworthiness and dominance, the same methods can be used for any social judgment (see competence movie). As a first step, we have made freely available a number of face databases for academic, non-profit research. Second, these models can be used to construct culture-specific models of social judgments from the ground up and, hence, address important questions about the universal or culture-specific nature of social judgments from faces. The methods for constructing such models are completely data-driven without any a priori assumptions about the importance of specific facial parts (e.g., nose, eye brows). All that is needed is social judgments of faces by samples representative for specific cultural groups. Third, the parametrically manipulated faces can be useful in comparing the performance of different clinical populations in social cognition tasks. For example, as I described above, the valence/trustworthiness dimension maps onto social approach/avoidance responses. These responses may vary in a meaningful way across different clinical populations. Finally, the computer models could be useful in research and design of human computer interfaces. The models could be used for creating avatars that would be most appropriate for specific communications.
Movie 3: competence. Changes of facial features on the competence dimension. The running competence score is presented in standard deviation units in the lower right corner of the movie.
Antonakis, J., & Dalgas, O. (2009). Predicting elections: Child's play! Science, 323, 1183.
Ballew, C. C., & Todorov, A. (2007). Predicting political elections from rapid and unreflective face judgments. Proceedings of the National Academy of Sciences of the USA, 104, 17948-17953.
Bar, M., Neta, M., & Linz, H. (2006). Very first impressions. Emotion, 6, 269-278.
Blair, I. V., Judd, C. M., & Chapleau, K. M. (2004). The influence of Afrocentric facial features in criminal sentencing. Psychological Science, 15, 674-679.
Blanz, V., & Vetter, T. (1999). A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 187–194.
Blanz, V., & Vetter, T. (2003). Face recognition based on fitting a 3D morphable model. IEEE Transactions on pattern analysis and machine intelligence, 25, 1063-1074.
Eberhardt, J. L., Davies, P. G., Purdie-Vaughns, V. J., & Johnson, S. L. (2006). Looking deathworthy: Perceived stereotypicality of Black defendants predicts capital-sentencing outcomes. Psychological Science, 17, 383-386.
Engell, A. D., Haxby, J. V., & Todorov, A. (2007). Implicit trustworthiness decisions: Automatic coding of face properties in human amygdala. Journal of Cognitive Neuroscience, 19, 1508-1519.
Fiske, S. T., Cuddy, A. J. C., & Glick, P. (2007). Universal dimensions of social cognition: warmth and competence. Trends in Cognitive Sciences, 11, 77-83.
Hassin, R., & Trope, Y. (2000). Facing faces: Studies on the cognitive aspects of physiognomy. Journal of Personality and Social Psychology, 78, 837-852.
Lavater, J. C. (1772/1880). Essays on physiognomy; for the promotion of the knowledge and the love of mankind. Abridged from Mr. Holcrofts translation. London. Eighteenth Century Collections Online. Gale Group.
Little, A. C., Burriss, R. P., Jones, B. C., & Roberts, S. C. (2007). Facial appearance affects voting decisions. Evolution and Human Behavior, 28, 18-27.
Montepare, J. M., & Dobish, H. (2003). The contribution of emotion perceptions and their overgeneralizations to trait impressions. Journal of Nonverbal Behavior, 27, 237-254.
Montepare, J. M., & Zebrowitz, L. A. (1998). Person perception comes of age: The salience and significance of age in social judgments. Advances in Experimental Social Psychology, 30, 93-161.
Olivola, C. Y., & Todorov, A. (In press). Elected in 100 milliseconds: Appearance-based trait inferences and voting. Journal of Nonverbal Behavior.
Oosterhof, N. N., & Todorov, A. (2008). The functional basis of face evaluation. Proceedings of the National Academy of Sciences of the USA, 105, 11087-11092.
Oosterhof, N. N., & Todorov, A. (2009). Shared perceptual basis of emotional expressions and trustworthiness impressions from faces. Emotion, 9, 128-133.
Said, C., Sebe, N., & Todorov, A. (2009). Structural resemblance to emotional expressions predicts evaluation of emotionally neutral faces. Emotion, 9, 260-264.
Todorov, A., & Engell, A. (2008). The role of the amygdala in implicit evaluation of emotionally neutral faces. Social, Cognitive, & Affective Neuroscience, 3, 303-312.
Todorov, A., Mandisodza, A. N., Goren, A., & Hall, C. C. (2005). Inferences of competence from faces predict election outcomes. Science, 308, 1623-1626.
Todorov, A., Pakrashi, M., & Oosterhof, N. N. (2009). Evaluating faces on trustworthiness after minimal time exposure. Social Cognition, 27, 813-833.
Todorov, A., Said, C. P., Engell, A. D., & Oosterhof, N. N. (2008). Understanding evaluation of faces on social dimensions. Trends in Cognitive Sciences, 12, 455-460.
Vigil, J. M. (In press). A sociorelational framework of sex differences in the expression of emotion. Behavioral and Brain Sciences.
Wiggins, J. S. (1979). A psychological taxonomy of trait descriptive terms: The interpersonal domain. Journal of Personality and Social Psychology, 37, 395-412
Wiggins, J. S., Philips, N., & Trapnell, P. (1989). Circular reasoning about interpersonal behavior: Evidence concerning some untested assumptions underlying diagnostic classification. Journal of Personality and Social Psychology, 56, 296-305.
Willis, J., & Todorov, A. (2006). First impressions: Making up your mind after 100 ms exposure to a face. Psychological Science, 17, 592-598.
Winston, J., Strange, B., O’Doherty, J., & Dolan, R. (2002). Automatic and intentional brain responses during evaluation of trustworthiness of faces. Nature Neuroscience, 5, 277-283.
Zebrowitz, L. A. & Collins, M. A. (1997). Accurate social perception at zero acquaintance: The affordances of a Gibsonian approach. Personality and Social Psychology Review, 1, 204-223.
Zebrowitz, L. A., & McDonald, S. M. (1991). The impact of litigants’ babyfaceness and attractiveness on adjudications in small claims courts. Law and Behavior, 15, 603-623.
Zebrowitz, L. A. & Montepare, J. M. (2008). Social psychological face perception: Why appearance matters. Social and Personality Psychology Compass, 2, 1497-1517.