Speaking of Psychology: Can a personality test determine if you’re a good fit for a job? With Fred Oswald, PhD

These days, many companies use assessments such as personality tests as part of the hiring process or in career development programs. Fred Oswald, PhD, director of the Organization and Workforce Laboratory at Rice University, discusses why companies use these tests, what employers and workers can learn from them, and how new technologies, including artificial intelligence, are changing workplace assessments.

Fred Oswald, PhD, is a professor of industrial/organizational psychology and director of the Organization and Workforce Laboratory at Rice University in Houston, Texas. He studies the factors that contribute to workplace success, including how to understand, define and measure the individual differences that affect employees’ performance. He's also interested in school-to-work transitions and in the use of big data and machine learning in workforce analysis. His 2019 book, “Workforce Readiness and the Future of Work,” examines how technology, education and policy will shape the future of work. He is a former president of the Society for Industrial and Organizational Psychology, which is Division 14 of the American Psychological Association.

Kim Mills: Have you applied for a job lately? If so, you may have been surprised to find that a resume and cover letter weren't enough to get you in the door. In addition to these traditional methods of screening job applicants, many companies now also use some form of pre-employment testing, including personality assessments, to help determine whether a candidate will be a good fit for the job. In fact, one 2018 survey of human resources professionals found that 79% of them use some kind of testing when making external hiring decisions, and 72% use testing for internal hires. So even if you've been in your job for a while, you may find yourself taking an assessment at work. The same survey found that 79% of respondents used assessments in their companies’ career development programs.

So why do employers use these tests? Are the assessments that they're using reliable and valid? Can a person's personality or other characteristics help predict whether they'll succeed at work? And how are new technologies, including artificial intelligence, changing workplace assessments? What can employers, employees and job applicants expect to see next on this front?

Welcome to Speaking of Psychology, the flagship podcast of the American Psychological Association that examines the links between psychological science and everyday life. I'm Kim Mills. Our guest today is Dr. Fred Oswald, a professor of industrial organizational psychology and director of the Organization and Workforce Laboratory at Rice University.

Dr. Oswald studies the factors that contribute to workplace success, including how to understand, define and measure the individual differences that affect employees’ performance. He's also interested more broadly in the future of work. His 2019 book, "Workforce Readiness and the Future of Work," examines how technology, education and policy will shape the future of work. He is a past president of the Society for Industrial and Organizational Psychology, which is Division 14 of the American Psychological Association. 

Thank you for joining us, Dr. Oswald.

Fred Oswald, PhD: Kim, thank you for having me. This is a real pleasure and I look forward to the conversation today.

Mills: Great. Well, let's start with a broad question. Why do employers use pre-employment testing? What do they hope to learn about job applicants that they can't get from a resume, a cover letter or a job interview?

Oswald: Well, testing has a long history and a wide range of purposes. Whether we're talking about employment, or we all have taken plenty of tests in school. There are tests for certification, for example, airline pilots take tests to ultimately guard public safety. And employers are hoping to understand, who is coming into the workplace? What are their background characteristics? What are the knowledge, skills and abilities they have coming in? Or what are their qualities? I realize we'll talk about personality today. And so testing is not perfect, but perfect should not be the enemy of the good in terms of using tools that can help make good decisions, both for the organization and for the job applicant who obviously seeks to find employment that's fitting for them, not merely get an invitation to join the organization, but actually succeed in it.

Mills: So a moment ago, I mentioned that employers are using testing to make hiring decisions and companies use them in leadership and career development programs. What kinds of tests are they using? What are the questions that prospective employees are being asked?

Oswald: So my own research and experience deals more with personnel selection on the point of initial hire. So think about characteristics like personality, or job knowledge or sometimes logical reasoning skills. If they don't have prior background indicated in their resume, that they have prior technical skills they there might be a test for some general reasoning. Also interests and motivation. Those obviously differ between people and people have different profiles of interests and types of motivation and goals. And so tests attempt to get some of those characteristics of job applicants, hopefully in a reliable and valid way. And we can talk about that further.

Mills: Yeah, I did want to talk about that. What's the difference between a good workplace assessment tool and a bad one?

Oswald: A useful framework to go off of is think of a three legged stool of reliability, validity and fairness. Reliability deals with whether what you're measuring is stable over time. In the case of job applicants, in other words, you don't want an applicant to be taking a test that is essentially a roll of the dice or is measuring something like mood that fluctuates. Instead, what you want are characteristics that are likely to appear on the job upon the point of hire, and that's where the employee and the organization starts out with the employee to move them forward through training and development and, as you mentioned, leadership and career progression as well.

And so reliability is as a cornerstone of measurement, to make sure that a test that claims to measure personality, for instance, actually is doing so. We could put labels on any test and make the claim that it's measuring what we say. But how do we know? And we need data to inform that. And so there are data based approaches to ensure that scores are measuring what they shed personality, job knowledge, motivation, etc.

And also turning to the second leg of the stool of validity. So do these tests predict outcomes in organizations that we think they should? So we know that, for example, conscientiousness is a factor of personality known to predict job performance pretty consistently, across jobs over time, across cultures. And, of course, that relationship makes sense to some degree, you need to show up on time for your job, you need to follow the rules, you need to set goals and achieve them and so on.

But, like most things in psychology, the devil is in the details. Can you support that common sense with well-developed measures, with evidence and with good measures behind the data that you're collecting? So not just on the personality side, in the case of this example, but how are you measuring performance in the organization? What does it mean to be a successful worker? It's as much philosophical as it is a measurement and a statistical issue, to try and figure out what problem we're trying to solve through testing.

And then the third leg of the stool is fairness, to make sure that anyone who takes the test has the same opportunities to reflect who they are, what they know. And job irrelevant characteristics are suppressed. So you wouldn't want a test that had too high of a vocabulary level, if you're not measuring vocabulary, that would be that would be unfair, that kind of thing. Or accommodating people who have a vision impairment, for example, if visual acuity is not required for the job, then you shouldn't place visual demands on the test for people that need that accommodation. So being sensitive to the people who are taking tests is just as important as what you do with the scores that come out.

Mills: Is there any independent body that validates these tests? I mean, how would an employer choose one and know that it's a good one?

Oswald: So there are various professional documents that are used by folks who develop tests. So in my world, for example, as you mentioned, I'm a member of the Society for Industrial and Organizational Psychology, which is Division 14 within APA. We have a document called "The Principles for the Validation and Use of Personnel Selection Procedures." It's in its fifth edition. It's actually free to download through the APA website. And it contains some of the principles that I've reflected on in terms of reliability, validity and fairness because, those are the principles that go into any test no matter what you're measuring, how you're measuring it as we move forward into the world of AI and new technologies for measurement, these principles still hold to make sure we're measuring job-relevant characteristics and administering tests in a reliable, fair, valid, secure, ethical, legal manner.

And so that's a major resource in the area of employment testing. But there's also the standards for educational and psychological assessment, which APA is also involved in as a co-editor, with two other organizations, and similar principles apply. But it's broadened into the domains, as the title suggests, into education, whether that's K-12, or placement issues, into different educational programs, and it's more about that context. So hopefully, those who are listening understand that we've been thinking about testing for a long time, the SIOP document I mentioned, it's in its fifth edition, it's maybe as many decades old or close to it. And so serious attention has been paid to help ensure that tests are measuring what they should, and anyone seeking to purchase a test should be thinking about some of these issues, or at least they would benefit from doing so and not merely trusting what sounds good. But actually seeing whether there's evidence behind the claims of a test being reliable, valid and fair.

Mills: One of the reasons that I wanted to talk to you today was because of a kerfuffle that ensued among some psychologists a couple of months ago, when HBO Max aired a documentary film called "Persona, The Dark Truth Behind Personality Tests." The film focused mostly on the Myers Briggs Personality Test, which I'm sure a lot of our listeners have heard of, and maybe have even taken. What's your view of this test in particular? Is it a valid tool for employers? And why is it so popular?

Oswald: I'm hesitating here, because I don't want to put any particular company test on the mat, or as a target. But we all understand that the Myers Briggs is a popular test. It's used widely in organizations for a variety of purposes. And what I'll say about that test really applies more generally to tests that are like it. The early versions of the Myers Briggs, and I realized the test has been revised, perhaps in ways I'm not aware of. But at least the fundamentals of the test, in its early development had to do with profiles. So a person would say they are an INTJ and I honestly forget what that stands for, but it's something like introverted and thinking and judgmental, things like that. And so, the profile is used to... Intuitive, sorry, I'm going back to the end there. Introverted, intuitive thinking and judging.

But the point of that is to say that, profiles are a complex way of describing a person. And what you could ask before turning to profiles is whether the single variables that are part of that profile, where do you stand on those characteristics, rather than jumping forward to that complex INTJ combination. Because science prefers simplicity before you jump into complexity. So in other words, a researcher would look at INTJ and break those variables apart and see how those variables intuitive, introverted, thinking, judging how those correlate with each other, and how they correlate with organizational outcomes. See what those relationships are, and then see if a profile or something complicated like that predicts above and beyond the more straightforward relationships.

So I think that's where the general arguments lie around the test is that, it's not clear whether the profiles add anything above and beyond a traditional test that is more transparent and straightforward in terms of how it's measuring personality traits, rather than combining those traits into profiles. I will say that, the profiles do lead to interesting conversations and perhaps yield interesting and useful insights. But whether those conversations and insights actually had an impact in terms of if you took this test for leader development, would it help you in terms of your job success? I think the evidence is not as strong as it is for more tests in the more traditional vein that I mentioned, measuring one trait at a time.

Mills: So many of the validated tests that are being used in the workplace are based on what's called the Big Five personality traits. Some of our listeners may know what they are, but could you explain what the big five are and the history of that model of personality?

Oswald: Sure. So briefly, and this reflects on the point of measuring one trait at a time, the big five measures, as the name implies, five traits. So conscientiousness is one that I had mentioned already. There's extraversion, agreeableness, openness, and neuroticism, which is an unfortunate term because all the traits are intended to measure normal personality, it doesn't mean you're neurotic if you score highly, but it does mean that there's a distribution that people have in terms of their anxiety and worry, things like that, again, within the normal range. Same thing for all the other traits, if you're low on conscientiousness, that doesn't mean you're abysmally low, it just means you're low relative to other people. And same for the other traits that are being measured. These are all in the normal range, these are not clinical measures of personality.

And so how those factors were derived is actually from the dictionary. So some folks, early in the history of personality psychology, scoured the dictionary for these descriptive terms for people, and essentially gave those terms as a test asking people, how much do these terms apply to you. And there were various rules for which words got included and excluded, and so on. But the point is they gave this test and then they looked at the themes, they used statistics to find these factors, these underlying themes that basically cluster the types of responses people were making and there were five clusters. Hence, the big five came out of that.

And so psychologists will talk about the lexical hypothesis, and that refers to this dictionary, the lexicon dictionary based approach to deriving personality. There are variations on the Big Five, for example, the HEXACO model, there are nuances that make it more different than the Big Five than simply adding a sixth factor. But, I will say that one big distinction is a sixth factor called honesty and humility, the HH factor, honesty-humility. And that factor proves to be useful in measuring personality and understanding employees. There are organizations that will give so called integrity tests that they find from test vendors. And those tests are asking questions about the typical behaviors, someone exhibits that are, again, these are in the normal range, but what are described as your workplace behaviors in terms of honesty and integrity.

Mills: So it sounds like those five or six terms that we were talking about, that those can be fairly reliably predicted, then based on testing.

Oswald: Yeah, that people can write, the folks who are developing personality tests can write items under each of the Big Five that reflect reliability and validity and can write them in a way that is accessible to people and fair in that sense. They do vary in terms of their validity, so conscientiousness is probably the most valid of those five traits, just focusing within personality testing. But there's continued research on facets of personality. So, facets are the kind of sub factors underlie the Big Five. They focus on narrower aspects of personality. So for example, under the factor of conscientiousness, you might think about whether somebody is being more kind of achievement striving, like, it's kind of a moving forward motion of setting your goals and following through. Versus a rule following notion being dutiful, or maintaining order or paying attention to detail, that's more of a kind of a tediousness to it. Right? Is more kind of inward focus as opposed to outward focus. So there are these facets that can be measured reliably, and that perhaps have different relationships with organizational outcomes than those broader factors.

Mills: Is it possible for applicants to game these tests? I mean, I imagine that some of these questions are not so obvious, I'm not going to ask you are you an honest person? But can applicants figure out a way to give the employer an answer that would get them in the door for the interview, but it may not even be true?

Oswald: Yeah, there's no doubt that people can lie on personality tests. And so if everybody did so then we would throw the tests out the window, then there's nothing that will help in terms of being reliable or valid. But it turns out while some people do lie, as I mentioned, the perfect shouldn't be the enemy of the good because there's enough variation in responses that is reliable, that can be used for understanding the employee. And again these questions are trying to get at normal personality, the framing of the test is to indicate who you are and that it doesn't necessarily serve you well to lie on the test. Some tests actually attempt forms of lie detection to see if you're just kind of responding on the fly without thinking carefully through the items. There are various methods for that, those aren't perfect either.

But again, the goal is to try and find a good fit between the job applicant and the employer. And so the straightforward answer to your question is people certainly can lie. But in the context of real world personality testing, we don't see that to the extent that we would throw away personality test as part of the employment kind of battery of tests we would administer. I think you mentioned integrity tests items, an item might be something like would I turn in a fellow worker I saw stealing money, or an employee should be fired if the employer finds out the employee lied on their job application. And those items may have right or wrong answers, but there are degrees of how about you might endorse it so that the format might be on say, a one to five scale rather than true or false. And that provides enough variation to be able to get some reliable differences between people, and their standing on a personality trait.

Mills: So one of your research interests is new ways of using technology to measure personality, including big data and artificial intelligence. Can you talk about that? I think some of our listeners may have read news stories about companies using AI to interpret applicants’ facial expressions during job interviews, for example. So what are the potential advantages of these new technologies? And what might be potential pitfalls?

Oswald: Right. So we're talking about the Wild West here, that these AI technologies for selection are developing at the same time that we're thinking about them and safeguarding applicant privacy, and judging whether these tests will provide benefit, in general and above and beyond traditional tests, perhaps. And I think it's important for our listeners to appreciate at some level this distinction, several distinctions that are part of AI testing. So, the technology is one part of testing so, how are you gathering the data? So it could be video, although state legislation has rightfully come up to express concerns about whether gathering video images of applicants is really a good way of understanding the applicant. And we could talk about that further, if you like. But there's a technology for gathering the job applicants’ characteristics, so those could be audio, video. The audio could be processing tone, which again, might be problematic, but the language trying to extract themes out of what people say and maybe supplement the human interviewer in terms of reminding the interview, the things that were said and the themes that emerged.

In other words, it could be a hybrid approach, rather than pure AI, potentially. Other technologies are game-based assessment. So we all have probably heard ads like, hey, get online, play this game, and you can see what jobs are right for you. And it sounds engaging and fun and data driven, and so on. But again, the principles of whether the test is reliable, valid and fair, applies there. But there are games they're fast to take, they might be fun, so we're operating in this new world is selection where the standard fill in the bubble tests are maybe increasingly being left behind in light of these new technologies. So how do we make those technologies better? Is where my work kind of stands as, I think there's some criticisms to be made, but at the same time balancing criticisms with opportunities. What can we do better? How can we move these technologies forward?

Another part of the distinctions that I mentioned to be made is the algorithms. So there're the technologies, and then there're the algorithms that are applied to the data that are collected. So how do algorithms provide scores for people that are taking these tests? So it's one thing for somebody to take a personality test then you add up their scores, simple addition. But it's quite another thing to use machine learning, these sophisticated algorithms on somebody's behavior as they play a game and try and figure out what themes are in there. So that distinction, I think, is really important. And related to this is the type of data where some data may appear directly more psychological. A game might have you literally answer questions about personality, perhaps, and it would look almost like the test, traditional test. Or you might solve a cognitive puzzle, and it looks exactly like a puzzle you might get on paper in a cognitive test or knowledge test, right? You could be asked about knowledge within a game.

So those are more straightforward, and I would say are more intentionally measuring the psychological characteristics of job applicants you're interested in. And that's being contrasted with more incidental data. So incidental means as you mentioned, the video capture or learning about, maybe there's something in a resume that an AI engine is scraping from the web, getting people's resumes and extracting things from them. And they're not intentionally expressing the psychological features that might be inferred but instead, it's incidental, it happens to be discovered. And those are the promises of AI where we're hoping to discover psychological characteristics from data. But again, this opportunity also has risks and in terms of issues of privacy and ethics, which we all care about, not to mention litigation, which organizations care about.

Mills: So some of these games that are intended to tell you whether your personality is suited for a particular job, are those really valid? And what happens, say if your employer gives you one of these to play when you're already on the job, and it turns out that you don't have the right makeup for the job that you have. Maybe it requires creativity and the game says, "You're a brick, man, you're not creative at all, what are you doing in this job?"

Oswald: Yeah, well, tests are one part of a larger picture of decision making, or at least they should be treated as such. And how much weight can you put into a game? Is a good question, or how much weight can you put into any test? Speaking to the job applicant situation, if somebody is new to the world of work, they just got out of high school, and they're applying for jobs and don't have much work experience, in that case, a test might be more useful because it's supplying information in a systematic way that doesn't show up in any resume. And it's sort of the idea that there are personality features and knowledge features, that suggest the person will be a promising young employee.

By contrast, if somebody has an extensive resume with expertise being reflected, they're a neurosurgeon, for example. Perhaps, again, attempting to measure job knowledge is not necessary. And then there are cases where the test isn't perfect, but it's trying to measure these characteristics that are honestly hard to get at, through tests or through any other way, like creativity. And so, maybe those tests are used for developmental purposes and not for selection necessarily, not for saying you will have a problem in this job, but rather to say, here's what this game is indicating and as a developmental conversation, how do you see that informing what you should be doing next to work on your creativity?

And so to the extent those scores have some reliability and validity, then that's a worthwhile conversation, because the test is actually diagnosing something about creativity. Creativity is a hard thing to measure by the way. It sometimes depends on your prior knowledge, so you, you need to... It's not necessarily open ended in other words. Some types of creativity are where you just let your mind go and think about whatever pops into mind. But in other cases, it's where luck favors the well-prepared, in a sense, that you have a lot of knowledge in your head, and then you're asked to create, and you have all these knowledge frameworks that assist in your creativity. So I will say it from the get go, that that's a tough construct to measure, with tests or otherwise.

Mills: That a topic that we're going to be exploring in a future podcast, so listeners, stay tuned, because we will come back and talk to some experts about creativity, which is really fascinating. So what would you say to a job applicant who might be uncomfortable with the idea of an algorithm instead of a person evaluating a job application?

Oswald: Yeah, that's a great question. I think AI is as much a cultural feature as it is technology. And I think we're trying to figure out as researchers, and as people who are subjected to these technologies, how is that going to change the world of work? Will it be better for employees? Will it be better for organizations? How will it work when AI is even more infused into organizations? And perhaps becomes a given in how selection happens. That will be a different world than what we're experiencing today, where we see the entry point, we see AI entering the area of selection and continuing to grow. And as this is happening, we do see those going through those experiences as applicants are clearly talking and probably tweeting and getting a sense of how it works. State legislation keeps arising in response to these technologies to try and ensure the tests are measuring reliable characteristics of applicants, job-relevant characteristics that are important at the point of selection, not just relevant for the job, but relevant at the point of selection, I think that's important to emphasize.

And not invasive, and legal and ethical and those things. But the devil is in the details there. Reading these state bills or regulations, the language is open ended, because the technology is still being developed and we need to figure out, what are appropriate uses and what are inappropriate uses. Video capture seems to be most problematic to try and infer employee characteristics from their—how they look or what the emotions on their face, and so on, that seems very problematic and it is. It's not like the algorithm did detect highly relevant job features, and we should be using it, there doesn't seem to be evidence for that. So the lack of evidence is also a concern in that area.

But I think we will wait and see in terms of how people will—it will evolve, I guess, would be my answer, just like self-driving cars, right? Have barely happened yet. There's some very isolated instances of self-driving cars being in the world. But once they become more commonplace, we'll have entirely different phenomenon on our hands. I think there's also the applicant perceptions of the hiring process that will evolve as much as the decisions that are made from AI will evolve. So in other words, as you get more used to AI as a tool for selection, maybe people will see that humans, supplementing selection becomes—the part that's valued, which would kind of turn the tables on the whole thing.

So it'll be interesting to see how humans merge with AI as AI permeates selection. An applicant might feel that the organization is not paying much attention to them by farming it out to an AI to do selection on a broad scale, that would be a negative. But a positive could be that AI is treating applicants fairly. I mean, imagine AI tools that are making every attempt to be fair and accurate in ways that humans try to be. That would be a positive aspect of AI in the future as we continue to develop tools that are effective.

Mills: So it won't do things like humans do, like prejudge people based on their names, if they're unusual or foreign sounding or maybe you make a judgment because, "Oh, this person went to my alma mater, I'm going to give her an interview." Those sorts of things.

Oswald: Yeah, those sorts of things. That right. We're supposed to be selecting on-job relevant characteristics and not irrelevancies, like your name or your background. And another aspect, I think to consider is, sometimes people think, well, tests are not any better because they're deterministic. And that's not true either, tests are a way of letting employers know what an applicant looks like, what are their strengths and areas in need of development. And that then gets considered with other pieces of information. And again, the tenants of fairness apply to all aspects of the application. So it's not like a test score determines your fate in these systems, as much as we might put weight on these scores as we have sometimes hear about in the news, whether it's personality, the Persona series that you mentioned, or standardized testing in college admissions. No admissions office is relying solely on test scores, I don't imagine.

Mills: Right. Less and less. Yeah. Some of them aren't relying on them at all anymore.

Oswald: Yeah, exactly. And that's another conversation to have, I guess.

Mills: So the last question is kind of a big one. What are the major research questions that you think still need to be answered? And what are you studying now?

Oswald: Wow, that is a big question. Currently, I've been researching what products are out there in the world of AI and personal selection? What are vendors providing? And what evidence is there for their effectiveness? And we are not pretending to engage in a fully comprehensive survey of what's out there, because it keeps changing even by the day. But, at least getting a sense of what is being offered, what types of things are being offered. And what evidence is out there to support the technologies that are being offered and sold—sometimes at high dollar—and what do organizations get in return for their investment? What do employers hope to get? And how does that align with what they might get from these technologies?

So our research is looking there. Also, I've been interested in school-to-work transition. So, the question there is whether we can use some forms of detailed data collection or perhaps machine learning to look at the transitions that people make from their education into the workplace. And it might be a four year college, it might be a community college, it may or may not be a degree, it may be a certification, or people are taking courses relevant to their interests, and they're getting jobs. So what do those transitions look like? And how do algorithms capture those regularities so we can understand them? But then also, how can we intervene on that to make sure that people are seeking meaningful employment and steady employment in the world of work in a way that fits their capabilities, and also their desires to grow in their field?

Easier said than done, that's a tall order. But, we're using data to inform those efforts. But then also connecting with educators and policymakers to see what they do with those findings. So it's one thing to reflect on the way things have been through the data. But it's another thing then to take that information and say, well, where are we going? What are the new directions that we should be headed and that the data alone, don't tell us? So that's been pretty interesting to me and hopefully impactful to a range of folks who perhaps like me, didn't know what they were going to do when they grew up, and then they kind of fumble their way through the world. I've always been an advocate of even giving a little information can go a long way to open people's eyes to the possibilities.

Mills: Well, this has been very interesting. Thank you for joining us, Dr. Oswald. I've enjoyed speaking with you today.

Oswald: Yeah, thank you very much Kim. This has been really great. And more importantly, I hope listeners find it interesting and we keep our gears turning on these issues.

Mills: You could find previous episodes of Speaking of Psychology at speakingofpsychology.org or on Apple, Stitcher, or wherever you get your podcasts. If you have comments or ideas for future podcasts, you can email us at speakingofpsychology@apa.org. That's speaking of psychology, all one word @apa.org. Speaking of Psychology is produced by Lea Winerman. Our sound editor is Chris Condayan. 

Thank you for listening. For the American Psychological Association, I'm Kim Mills.