Emerging technologies, such as computer-based simulations and personal digital assistant-administered surveys may change the way businesses provide training and job-candidate assessments, said presenters at the annual conference of APA Div. 14--the Society for Industrial and Organizational Psychology (SIOP)--held in Los Angeles, April 15-17. However, whether such technologies improve on older, pen-and-paper techniques depends on I/O psychologists' involvement and expertise, said SIOP President Fritz Drasgow, PhD, a psychology professor at the University of Illinois at Urbana-Champaign.

"The wild west of Internet testing needs to be tamed," he said during his presidential address.

For example, many new Internet tests do not adhere to the gold standards of reliability and validity, perhaps because sales professionals often develop the tests without consulting testing experts, Drasgow said. And even good tests can be rendered useless when translated poorly into another language--a common problem with Internet-based tests, he noted.

Psychologists are beginning to address such problems and explore the possibilities new technology brings, said conference participants. Some technologies they're investigating include:

  • Web-based tests. Companies are increasingly using Web-based tests that applicants take on their home computers rather than on-site, Drasgow noted. While many researchers are concerned that applicants might take that opportunity to cheat--perhaps by asking for help from a family member--researchers are finding evidence that people do not score much differently in proctored versus unproctored tests, said Drasgow.

  • Personal digital assistants (PDAs). Psychologists are beginning to use PDAs to administer surveys to people throughout the day to capture events and emotions soon after their occurrence, reported Joyce Bono, PhD, an assistant professor at the University of Minnesota. Bono and her colleagues used such technology to survey 54 health-care workers four times a day.

  • Computer simulations. Such simulations might help psychologists more accurately assess people's job skills, reported conference presenters.

For example, the National Board of Medical Examiners is already using an emergency room simulation to test physicians' diagnostic skill, and the American Institute of Certified Public Accountants (AICPA) started using a spreadsheet-based simulation for their licensure exam in April 2004, said Drasgow. Psychologists can help such organizations by ensuring that new, computerized tests effectively measure job skills, he said.

"There are many opportunities for people to do research and assessments, and really improve the quality of these testing programs," Drasgow noted.

Lifelike simulations

In fact, I/O psychology graduate student Krista Mattern and her colleagues at the University of Illinois at Urbana-Champaign are helping AICPA refine its simulation and better score the test. Unlike the old multiple-choice questionnaire, the new accountant-certification test requires would-be accountants to solve problems rather than just demonstrate factual knowledge.

"Once you get the job, you generally do not do multiple choice questions as a doctor, an architect or an accountant," Mattern noted.

That's why the new accountant exam asks test-takers to perform real-world tasks, such as calculating the value of a hypothetical company's new bonds, using a spreadsheet embedded in the test. To answer the question, the test-takers search a reference database similar to the one professional accountants use to look up the case law that justifies their answers.

Such simulations are more difficult to score than traditional tests, Mattern said. For example, statistical analysis revealed that many answers within each simulation were highly correlated. A test-taker might be asked to put the name of a bond in one box and the price of that bond in the adjacent cell--leading to two correct answers or two incorrect answers, but rarely one correct and one incorrect.

To avoid counting the same skill twice, the researchers suggested AICPA delete many of the highly correlated questions--about seven test items for each of the six simulations. In the future, Mattern hopes to devise more elegant solutions to scoring conundrums, perhaps by weighting answers differently, she said.

Testing anywhere

Instead of taking tests under the supervision of an employer or outside agency, people are increasingly taking tests on their home computers, accessing material online. Employers like using the Internet to disseminate tests because they don't have to schedule candidates for testing, said conference participants. However, psychologists are just beginning to explore how people score differently when they take tests at a place of employment versus anywhere the Internet is available.

In particular, Internet test-takers might be more likely to cheat by enlisting a smart friend or using outside resources, said Ben-Roy Do, a University of Illinois, Urbana-Champaign graduate student.

But if people do cheat on Internet-based tests, they aren't very effective, Do found in a study of 12,620 employees of a retail chain. In the study, the participants took a personality and aptitude test that the company used to select team leaders and distribution center supervisors. About one-quarter of the participants took the test at a central location with supervision, and the rest took it on the Internet wherever they chose.

Contrary to the researcher's expectations, they found that the people who took the test at home scored lower than those taking the proctored test.

Perhaps more cheating would have occurred had the test been higher-stakes, noted session participant Julie Olson-Buchanan, PhD, a psychology professor at California State University, Fresno. Future research could address such questions and further improve on Internet-test technology and practice, she noted.

Portable assessment

One of the advantages of Internet-based tests is they can be taken almost anywhere, but some researchers are making their tests even more portable with handheld computers, such as Palm Pilots. Using such devices, researchers can conduct on-the-spot surveys of participants' moods, thoughts and recent experiences, said Daniel Beal, PhD, assistant psychology professor at Houston's Rice University, during a session on using new technologies.

Moreover, PDAs allow researchers to get a sense of the temporal sequence of events--for example, a conversation with a supervisor immediately preceding a bad mood--which can aid in establishing cause and effect, he noted.

Researchers from the University of Minnesota did exactly that by providing 54 study participants--including nurses and administrative professionals--with Palm Pilots, which beeped at random intervals four times a day. When prompted by the PDA, the participants recalled the most recent interaction between themselves and a colleague, and reported their emotion as well as the emotion they outwardly portrayed.

"Irritation was the most common emotion experienced and suppressed at work," said Bono.

Additionally, employees tend to experience less optimism and happiness when talking with their supervisors than when they spoke with their colleagues, Bono reported. And when employees had to regulate their emotions, they tended to feel less satisfied with their job, she said.

Frequent polling also allowed Bono to look at variation within individual respondents, noting, for example, that a participant is rating herself as two points less happy than she usually does.

To aid researchers such as Bono, Beal has helped develop the Purdue Momentary Assessment Tool--free software that psychologists can use to administer surveys through PDAs. The program--which can be downloaded at www.mfri.purdue.edu/PMAT--allows researchers to enter in specific survey questions and determine how often the PDA will ask the participant to fill out the survey. Then, researchers can download the survey program onto the individual handheld computers.

While PDAs have a few disadvantages--they cost more than pen-and-paper surveys and they can annoy participants--the technology allows researchers to capture the changing nature of everyday experience, Beal says.