For 35 years, McMaster University's Michael G. DeGroote School of Medicine used the traditional method of interviewing medical school applicants: Candidates sat in a room for half an hour with three or four interviewers, fielding a range of impromptu questions meant to determine their personal fitness as future physicians.
That changed in 2004 when the Canadian medical school instituted a new interviewing process developed by its faculty. Called the Multiple Mini Interview, or MMI, it uses a kind of "speed-dating" technique that allows applicants to visit some 10 different stations, each with a different interviewer who poses predetermined questions in such areas as ethical and moral reasoning, responsibility, teamwork and communication skills.
The method is fairer to applicants, says McMaster professor emeritus Jack M. Rosenfeld, PhD, who helped to develop the protocol. Because each station provides a fresh start, applicants are able to show their strengths and weaknesses more accurately than the single "hot-seat" approach allows, he says. It also gives applicants a chance to compensate for a lack of knowledge or experience in one area by shining in another, he explains.
More than 85 studies from around the world support the team's hypothesis that the method captures more of a candidate's actual strengths and weaknesses than traditional interviews, particularly in nonacademic but relevant "soft skills" areas such as the ability to communicate clearly, accurately and empathetically, Rosenfeld says. That includes a 2012 article in the Journal of the American Medical Associationopens in new window by Kevin W. Eva, PhD, Harold I. Reiter, MD, Jack Rosenfeld, PhD, Kien Trinh, MD, Timothy J. Wood, PhD, and Geoffrey R. Norman, PhD, which found that applicants who passed the McMaster MMI did better on portions of the licensing exam that assessed these kinds of skills four years later than those who didn't pass the MMI but were admitted to other medical schools.
Others agree: The MMI is now being used in medical schools, schools of pharmacy, and medical residency and physician-assistant programs throughout the world.
The MMI is one of myriad so-called noncognitive tests or assessment methods that are starting to be employed in academic settings, some more rigorously tested than others. Generally imported from the business world, these types of assessments — many of which are taken like regular tests, via paper and pencil or online — are now being considered for use by educators and college and university admissions officers.
Despite the common label "noncognitive," such tests actually assess a wide range of success-related noncognitive and cognitive factors, field experts say. These include skills such as critical thinking and creativity; behaviors such as conscientiousness, reliability and good study habits; and values and attitudes such as integrity and honesty. Because the term "noncognitive" does not adequately capture the breadth of what these tests cover, researchers are starting to use such terms as "nonacademic" to better describe their range. The key difference is that noncognitive (or nonacademic) tests measure personal qualities, skills or behaviors related to academic success rather than discrete academic knowledge or reasoning in areas like math or biology.
Major tests of this type include innovative interviewing techniques such as the MMI; structured references that ask reference providers to rate applicants on specific characteristics such as conscientiousness and the ability to collaborate; and the increasingly popular situational judgment test, where test-takers choose how they would respond to challenging real-life scenarios. In the academic context, for example, they might be asked which of five choices they'd make if they caught a peer texting answers to another student during a test.
Several factors help explain the burgeoning interest in these kinds of measures, says industrial-organizational psychologist Wayne Camara, PhD, senior vice president of research at ACT, the testing and assessment research company that publishes the ACT college placement test and a range of learning tools.
For one, they could help make the college admissions process more inclusive. On average, ethnic-minority students score lower than whites on the SAT and ACT, and some educators argue that assessing such nonacademic qualities as leadership, persistence and study habits — which have few or no subgroup differences but are correlated with positive academic outcomes including graduation rates — could help colleges gain a more holistic picture of each applicant. Research shows that if elite colleges gave equal weight to academic and noncognitive or nonacademic scores, "you would improve the diversity of the admitted class dramatically," Camara says.
Another reason to use these tests is their cost-cutting potential, particularly at universities that process thousands of student applications, says Neal Schmitt, PhD, an industrial-organizational psychologist and professor emeritus at Michigan State University who conducts assessment research. MSU, for instance, gets about 35,000 applicants a year, so using computer-scored noncognitive multiple-choice tests, rather than interviews or student essays that must be read individually, could save admissions committees time and money, he says. A 2013 article in Medical Educationopens in new window by researchers at the University of Sydney underscores the point: Administering the MMI via Skype saved 84 percent in costs over administering it in person, with no loss of content validity or reliability, the team found.
What's more, well-validated assessments and empirical data are superior to human judgment when it comes to assessing nonacademic qualities important for success, says Nathan R. Kuncel, PhD, an industrial-organizational psychologist who studies testing at the University of Minnesota. A 2013 meta-analysis in the Journal of Applied Psychologyopens in new window that Kuncel conducted with colleagues, for instance, found that using statistical methods to analyze information about employees or college students was 50 percent more accurate in predicting their workplace or academic success than "holistic" or clinical techniques that relied on human judgment alone.
Nonacademic or noncognitive measures also can add extra validity to cognitive or academic measures such as grades and SAT scores, these researchers say. In studies of a noncognitive test called ACT Engage — a multiple-choice assessment designed by ACT that measures academic motivation and skills such as goal striving, social engagement and academic self-regulation — good scores increased the likelihood of correctly predicting academic success by an additional 10 percent to 20 percent, according to the ACT websiteopens in new window. Examples of items on the college version of ACT Engage include statements asking students to rate themselves on general determination — "I am serious about fulfilling my obligations," for instance — or on their ability to strive for goals — "I bounce back after facing disappointment or failure," for example.
"I think we've hit a point where we're making a bunch of concerted efforts to capture nonacademic qualities in more rigorous ways," Kuncel says.
U.S. use still sparse
So far, though, noncognitive measures are still not widely used by U.S. universities and medical schools, though they are in European institutions, Kuncel says. Some exceptions are Oregon State University, which screens wait-listed undergraduate applicants using a noncognitive test called the Insight Résumé; the medical schools at Stanford University, the University of California, Los Angeles, and the University of Cincinnati, which employ the MMI; and Michigan State University, which for the first time this fall will use two kinds of noncognitive multiple-choice tests to help decide which wait-list applicants are admitted and which are not.
Developed and studied independently by Schmitt for The College Board, one test asks about student experiences related to qualities like leadership or open-mindedness — the number of clubs or organizations they headed in high school, for instance, or how often they'd eat a new type of food just to see what it was like. The other asks them to choose what they would do in an ambiguous ethical situation, a type of situational judgment task.
Other universities have tried their own versions of noncognitive tests on an experimental basis. Former APA President Robert Sternberg, PhD, for instance, developed noncognitive admissions tests for Yale and Tufts universities that were used for a short time. And now, the Association of American Medical Colleges is conducting research on the possibility of requiring medical school applicants to take a situational judgment test at the same time they take the Medical College Admission Test (MCAT).
Among the reasons that the tests are still sparsely used are competition, tradition and, to some extent, ongoing work to develop good instruments, testing researchers say.
For instance, while in theory elite schools support the addition of such tests, in practice they fear losing top-notch applicants if they require more admissions materials than other schools. Other schools worry about extra costs despite the potential long-term savings of using such tests. Finally, some are concerned that unscrupulous companies will quickly figure out the "codes" for some nonacademic tests and give students the ability to cheat — to provide answers schools would want to hear rather than information on their actual behavior.
That's a valid concern, says industrial-organizational psychologist Neil Christiansen, PhD, a professor and assessment researcher at Central Michigan University. His research finds that job applicants can and do fake noncognitive tests to get job offers, unless the tests deliberately conceal which answers earn higher scores. He also has found that people who read books on how to improve their scores on personality and other noncognitive tests were able to score much higher on them.
"Although noncognitive tests have the potential to predict different aspects of college success, people may not be honest when completing them," Christiansen says. "The tests have to be carefully developed so they cannot be easily distorted."
At the moment, therefore, test developers including ACT, ETS and The College Board are finding other educational applications for these measures. One is to help two-year colleges, community colleges, high schools and middle schools assess skills and behaviors related to academic success such as study skills, which in ACT terms means the extent to which students believe they know how to assess an academic problem, organize a solution, and successfully complete academic assignments.
The aim is partly to predict retention, and partly to help students improve their study habits so they are better equipped to graduate.
"Behavioral tests need not be used only as gatekeepers, but as diagnostic tools to help colleges and even high schools prepare students for success," Camara explains.
According to the SSATB, a nonprofit organization representing independent secondary schools, admissions officers at these schools are using a wide array of noncognitive tests to gauge applicant fit. When the college preparatory school Philips Exeter Academy interviews applicants, for instance, it assesses them on eight noncognitive criteria that correlate highly with college success, including having a positive self-concept, realistic self-appraisal and a track record of successful leadership.
In a similar vein, the Galloway School in Georgia is using the Grit Survey — an assessment developed by positive psychologist Angela Duckworth, PhD, and based on the work of educational psychologist Carol Dweck, PhD — to assess students' persistence and perseverance.
College-ready, career-ready
Developers are continuing to improve and refine these tests to address concerns about cheating, to make them more attractive to college admissions teams, and to ensure that they continue to offer important information above and beyond good grades alone — including to the students themselves.
Work by educational researcher David T. Conley, PhD, who directs the Center for Educational Policy Research at the University of Oregon, points to an additional diagnostic application of using such measures: determining young people's job and career readiness and developing their skills based on the findings.
Through a four-part model he calls the Four Keys to College and Career Readiness, Conley uses evidence-based strategies to help young people gather and use nonacademic tools not just for school success, but also to succeed on a career path. School-based skills include engaging with instructional material in effective ways; learning good study attitudes, habits and skills; and acquiring the knowledge to successfully navigate major transitions, for example from high school to college or college to a career. Similar critical-thinking tactics are important in the career context, he notes, along with professionally oriented strategies such as seeking out mentors and engaging in lifelong continuing education.
Conley's team has validated two instruments based on the model, one for use at the K–12 level called CampusReadyopens in new window, and the other for use at the postsecondary level called the Conley Readiness Indexopens in new window. The researchers are now studying the relationship between responses on the measures and a range of performance factors.
Developing a "readiness" orientation sets the stage for young people to take the steps they need to succeed in college or a career, whether that's law or welding, Conley says. That's increasingly important as the economy, technology and the workplace continue to undergo big shifts.
"Lack of success results less often from a lack of ability than from a lack of strategy, skills and tools," Conley says. "If you tell students what's important for college and career success and you encourage them to develop those skills," he adds, "more students will do so and more students will succeed."