APA is hosting Technology, Mind & Society, an interdisciplinary conference exploring interactions between humans and technology, on Oct. 3–5 in Washington, D.C. Continuing education is offered. For more information, visit https://tms.apa.orgopens in new window.
Technological advances are increasingly moving psychological testing out of the clinic or lab and into people’s everyday lives, allowing clinicians and researchers to gather multiple data points rather than just a periodic snapshot when people come into their offices.
Of course, psychological and neuropsychological assessments have been going digital for decades now. But the new and evolving assessment technologies go far beyond traditional tests delivered via paper or computer screen. Psychologists are developing assessments embedded in smartphones, games, virtual environments, even watches. They’re using these mobile forms of digital monitoring to assess suicidality and other problems. And they’re using "big data" drawn from such assessments to explore the possibilities of just-in-time interventions that provide support to people when and wherever they need it (see "Trends Report: Technology Is Revolutionizing Practice," Monitor, November 2017).
These new kinds of mobile assessments have many advantages. "One is just the vast amount of data you can get compared to what you can get from someone with a paper-and-pencil test," says Thomas Parsons, PhD, who directs the University of North Texas’s Computational Neuropsychology and Simulation Lab.
Digital assessments can also be more accurate, he says—for example, they can measure people’s real-life functioning instead of how well they perform in a psychologist’s office or research lab and allow comparison of their functioning with their own past results rather than with those of a generalized population that shares their demographic characteristics.
Of course, there are still challenges as this quickly evolving field develops, says Justin B. Miller, PhD, ABPP/CN, who directs the neuropsychology program at the Cleveland Clinic Lou Ruvo Center for Brain Health in Las Vegas. Concerns include the pressing need to validate and create norms for these new instruments; data security and privacy issues; and the fast pace of technology itself, which can quickly render advances obsolete. Another problem is some psychologists’ reluctance to adopt new technology, says Miller, owing to both inertia and fears of being replaced by computers (Archives of Clinical Neuropsychologyopens in new window, Vol. 32, No. 5, 2017).
But these changes in assessment methods are coming, whether psychologists want them or not, says Miller. "Often concerns are raised that older people aren’t comfortable with technology," he says. "The challenge is that as the younger population ages, they’re going to be equally unfamiliar with paper-and-pencil testing."
Here’s a look at what’s coming:
Smartphone assessments. Smartphone-based apps continue to be a major trend. While commercially available products focus on providing stress relief and other instant interventions, researchers are now exploring ways to use smartphones as assessment tools.
These mobile tools can give psychologists a more accurate picture of the problems they’re trying to assess, says psychologist Martin Sliwinski, PhD, who directs the Center for Healthy Aging at Penn State University. He is researching and developing an app in partnership with a company called Sage Bionetworks that delivers brief tests of cognitive functioning that older people can take as they go about their everyday lives. He and his colleagues have found that averaging the results of these real-world brief assessments is just as psychometrically reliable as assessments made in more controlled environments (Assessmentopens in new window, Vol. 25, No. 1, 2018).
Assessing people in a lab or clinic can affect their performance just like "white coat syndrome" can increase people’s blood pressure at doctors’ offices, says Sliwinski. Plus, mobile assessment eliminates the problem of a research participant or a patient having a bad day. "A bad night’s sleep or stress can impact performance," says Sliwinski. "If we distribute assessment across multiple times, we can average those things out."
Mobile assessments can also provide different kinds of data than are possible to collect in a clinician’s office, says Christine Vinci, PhD, an assistant member at the Moffitt Cancer Center in Tampa, Florida (Addictive Behaviorsopens in new window, Vol. 83, 2018). In her research on smoking cessation, participants wear sensors that assess their stress levels through physiological readings of such variables as heart rate and breathing patterns and even record the distinctive hand-to-mouth gestures of smoking; the sensors then transmit the data through participants’ smartphones. "This eliminates the need to have participants come in person to the clinic for these types of assessments," says Vinci. There’s even technology that allows clinicians to monitor how much people are smoking, says Vinci, who notes that other researchers ask participants to blow into carbon monoxide monitors attached to their phones to verify that they’re not smoking, while cameras ensure that they’re not giving their phones to nonsmoking friends.
Wearables. The use of watches, clothing and other wearable assessment devices is continuing to grow and is now moving beyond tracking physical data—like steps taken or hours slept—to assessing psychological states (see "Can an App Change Your Mood?" Monitor, January 2018). Consumers can already track their emotional states with the Feel Wristband, and their heart rates, breathing, sleep and other physiological measures with the Spire Health Tag, which clips to clothes, for example.
Now psychologists are working to make wearables useful for clinical purposes. MHS Assessments, a company that publishes assessments for use in clinical, educational, research and other settings, has partnered with a company called Revibe Technologies to produce a watch-like focus tracker for use by individuals with difficulties maintaining focus and attention, such as those with attention-deficit hyperactivity disorder (ADHD). A vibration sent every seven to 10 minutes during a prescheduled period of time prompts wearers to tap the device to note whether they’re on or off task, and motion sensors track how much the person is fidgeting. The device learns the wearer’s individual patterns and changes the vibrations sent to reflect his or her attention span and prevent habituation. For psychologists assessing ADHD, the objective, real-life data provided by the device supplements traditional assessments, says MHS chief executive officer Hazel Wheldon, who adds that it can be part of a treatment plan as well as an assessment method.
Such devices are spreading into other assessment areas. Clothing with embedded sensors is allowing sport psychologists to gather data on heart rate and other physiological factors that can affect performance, for example. Wearables combining heart rate data with GPS data can help psychologists identify particular locations that can trigger panic attacks, adds psychologist Lindsay Ayearst, PhD, principal scientific adviser at MHS.
Games. Embedding assessments within games provides another way to gather data in more natural environments. One example is a game called Rumble’s Quest in which primary-school children help a character called Rumble find his way home, all while being assessed for self-regulation and other measures of social and emotional well-being. Players enter a mythical world where they meet Rumble, who asks questions that assess major domains of middle childhood well-being, such as self-regulation, attachment to school and social confidence. Rumble also guides children through a set of interactive tasks that assess inhibitory control and other elements of executive functioning.
"This narrative approach provides an authentic mechanism for children to respond in a way that feels natural and meaningful, while contextual relevance increases motivation and engagement," says Jamin Day, PhD, a postdoctoral research fellow at the University of Newcastle in Callaghan, Australia, who described the game in a review paper with his colleagues (Clinical Child and Family Psychology Reviewopens in new window, Vol. 22, No. 1, 2019).
Traditionally, schools remove children from their classrooms for monitored assessment, says Day. "That’s resource and time intensive," he says. With a game like Rumble’s Quest, test administrators can assess entire classes as part of students’ routine activities. The goal is to provide assessment data that can identify targets for prevention and early intervention. "Although you can drill down to individual students as well, what you’re getting is a sense of how a class or school overall is doing," says Day. "It’s really about aiding service planning."
Games with embedded assessments can also target individuals. Psychologist Rachel Flynn, PhD, of Northwestern University’s Feinberg School of Medicine and colleagues have looked at one game-based assessment’s potential for monitoring attention in children (Journal of Autism and Developmental Disordersopens in new window, Vol. 49, No. 5, 2019). A feasibility study of the digital game at a summer camp for children with special needs showed that it demonstrated good reliability, producing stable results over the eight sessions of monitoring. The game also showed promise when it came to validity, with results from the game-based assessment strongly associated with results from a validated measure of attention called the Test of Variables of Attention (T.O.V.A.). Plus, exit questionnaires revealed that the children thought the assessment was fun.
Virtual reality. Assessments can also be incorporated into familiar everyday environments via virtual reality. For example, Parsons developed an assessment embedded in a virtual reality grocery store to assess memory problems in older people. In a feasibility study of the tool, he and colleague Michael Barnett, PhD, of the University of Texas at Tyler, found that participants preferred it over paper-and-pencil assessments (Journal of Alzheimer’s Diseaseopens in new window, Vol. 59, No. 4, 2017). "This test is really assessing real-world functioning," says Parsons. For instance, users must remember to pick up certain items and to drop off a prescription, and then remember where they parked their car in the virtual parking lot.
These kinds of virtual environments allow psychologists to introduce elements that would be impossible to assess in a traditional clinic setting, says neuropsychologist Tyler Duffield, PhD, an assistant professor of family medicine at Oregon Health and Science University, who is collaborating with Parsons on the virtual grocery store as well as on a virtual school. Both environments allow them to see how real-world distractions, social interactions and stress influence attention, response time or memory. With a virtual classroom, for instance, assessors can see how students respond to avatars passing them in the hallway or react to being excluded from playground fun.
These virtual environments may also increase the sensitivity of some neuropsychological testing, especially when the problem being assessed is subtle, such as a mild traumatic brain injury, says Duffield. Assessment technologies now under development, such as eye trackers and actigraphy, which quantifies people’s movement, will eventually produce even more data from these virtual environments. "We will have a much deeper understanding of some of these neurological conditions," says Duffield.
First, however, there needs to be much more research comparing virtual reality assessments with traditional assessments, says Duffield (Child Neuropsychologyopens in new window, Vol. 24, No. 8, 2018). Companies creating new products don’t always make scientific rigor a priority or perform proper validation and norming work before commercializing their products, he says. Studies can be underpowered or not fully describe the hardware being used.
Digital phenotyping. This assessment approach consists of collecting data both passively via GPS, movement trackers and other smartphone sensors and actively via periodic self-reports, then using that information to create phenotypes, or different categorizations of thought and behavior patterns. With traditional assessments, there’s just one data point, explains Evan Kleiman, PhD, an assistant professor of psychology at Rutgers University. With digital phenotyping, a smartphone collects data throughout the day in real-world contexts. Take thoughts of suicide, for example. In a traditional assessment, psychologists ask people to reflect on the past month and report on their suicidal thoughts over that time period. "The problem is that suicidal thinking varies too much," says Kleiman. "One day might represent the most severe day or just an average one."
In a recent study, Kleiman and colleagues used data collected four times a day from adults with past-year suicide attempts and psychiatric inpatients with suicidal ideation or attempts. From that data, they were able to develop five phenotypes of suicidal thinking, with the most at-risk group being those who reported high levels of suicidal thinking and low levels of variation in such thoughts (Depression and Anxietyopens in new window, Vol. 35, No. 7, 2018).
Digital phenotyping has the potential to help identify people who need treatment, says Thomas R. Insel, MD, who directed the National Institute of Mental Health before co-founding a health-care innovation company called Mindstrong (World Psychiatryopens in new window, Vol. 17, No. 3, 2018). By using smartphones to monitor where individuals are, what they’re saying and doing and how they’re typing on their phones, psychologists could one day use digital phenotyping to identify those who need care or are in danger of relapse. Slowed activity and a preponderance of use of the first-person singular could be signs of depression, for instance. But before digital phenotyping can come into its own as a clinical tool, adds Insel, ethical challenges—such as the potential for harmful surveillance and the need for data protection—must be resolved.
APA’s Committee on Human Research is considering convening a multidisciplinary working group to explore such issues, says Sangeeta Panicker, PhD, director of research ethics at APA. The goal would be to consider whether—and how—the ubiquity of mobile and digital technology in everyday life as well as its increased use in research are affecting the ethical framework of research with human participants.
Big data. In the Couple Mobile Sensing Project, Adela Timmons, PhD, and Gayla Margolin, PhD, use smartphones and wearable biosensors to collect a huge amount of data on couples’ interactions, including their heart rates, physical activity, proximity to each other, location, vocal pitch, even the frequency with which they use emotion-laden words (Social Psychological and Personality Scienceopens in new window, Vol. 8, No. 5, 2017). By collecting these data, Timmons hopes to get a more accurate picture of couples’ conflicts, what triggers them and what helps resolve them. The only problem: the sheer quantity of data.
"We’re collecting a lot more data than we can actually make use of and sense of," says Timmons, an assistant professor of clinical and quantitative psychology at Florida International University’s Center for Children and Families in Miami. To overcome that challenge, Timmons is working on a machine learning algorithm to predict and detect conflict. The goal is to use the assessment data to guide the development and delivery of individualized, just-in-time interventions—for example, a meditation exercise delivered by smartphone—that can reach people when and wherever they need them.
Remaining challenges
To realize the full potential of these new digital assessment methods, many challenges must be addressed, says psychologist Laura Germine, PhD, technical director of the McLean Institute for Technology in Psychiatry at McLean Hospital in Boston and an assistant professor of psychiatry at Harvard Medical School (The Clinical Neuropsychologistopens in new window, Vol. 33, No. 2, 2019).
One major challenge posed by digital assessment is the variability introduced by differences in hardware and software. The age and model of various devices can affect how quickly they can capture input, for example. "These are factors that are going to affect scores in ways that look like cognitive function but are actually artifacts of device type," says Germine.
Another source of variability is people’s familiarity with the devices being used in assessment. Rapid technological evolution is another challenge, with assessments and their norms quickly becoming obsolete. "Most assessments—particularly those that rely on controlled stimuli and measurement of reaction time—cannot be built in a platform-independent form," says Germine. "The idea of a platform-independent format … is never 100% achievable."
To help overcome these and other challenges, Parsons and his colleagues have outlined parameters for developing and using the new technologies, described software and hardware factors that can affect computerized assessments and laid out best practices for minimizing errors (The Clinical Neuropsychologistopens in new window, Vol. 32, No. 1, 2018).
As for psychologists’ own fears about digital assessment, says Miller, clinicians should rest assured that such tests will supplement—rather than replace—their own expertise.
"A lot of people think technology will be the end of us," he says. But just as radiologists discovered when automated image reading was introduced, psychologists will find that digital assessment will increase demand for their services by identifying and allowing triage of the patients who need more intensive follow-up now, he predicts. Says Miller, "This technology will change the role of the clinician—not replace the clinician."
