The benefits of experience sampling for longitudinal research

You’ve come up with a compelling research question for a longitudinal study. You’ve received funding and begun recruiting participants. You’ve pushed out a series of prompts and questionnaires and hired a student to manage the project and crunch the data. All is good—except your participants don’t always complete all the surveys, or they miss important details because of recall bias or other factors affecting the accuracy of their responses.

Most researchers at one time or another have struggled with research compliance, or the extent to which participants adhere to study protocols, complete the required number of surveys or tasks, and provide accurate responses. The challenges are magnified in longitudinal studies, whether participants are asked to report over a period of days or decades. Participants might complete too few surveys, complete them at times when their evaluations are less reliable, or zip through them inattentively, essentially button-smashing to receive a payment.

Regardless of the reason, non-compliance can artificially increase variance in a study’s findings, lower the study’s power when participants drop out or don’t provide all the data points needed, extend its duration, and misrepresent the magnitude of treatment effects.

One powerful longitudinal research method that can support participant compliance is experience sampling, a structured diary approach for capturing participants’ thoughts, feelings, behaviors, and symptoms throughout the day. (Other terms for experience sampling include ecological momentary assessment, ambulatory assessment, real-time data capture, intensive longitudinal designs, and time series design.)

High-frequency sampling also facilitates capturing the dynamics of psychological experiences, minimizing recall bias, and addressing temporally fine-grained questions, such as the order in which anxiety and rumination levels change over time, for example, or noting reactions to events that occur often throughout the day, like a social interaction with co-workers or friends.

• Interval contingent: Participants report in the moment at pre-determined time points, such as every three hours during the day. Interval spacing is key, according to Thai. If your intervals are too big, participants might miss important intervening events; too small and they could create a higher participant burden, decreasing compliance and degrading data quality. However, the anticipation of making a report at a specific time may also cause participants to think about the study, which could influence their responses—a phenomenon called reactivity.
• Signal contingent: Participants describe their activity in the moment when a signal or notification is delivered to their smartphone, for example. This design is commonly used to track naturally fluctuating phenomena like mood and health-related phenomena. Signals can be sent randomly or on some combination of fixed or random schedule—for example, pinging participants at some point during every 2-hour block.
• Event contingent: Participants report whenever they experience an event matching a predetermined definition. This is especially useful for events that are infrequent or difficult to predict, such as interpersonal conflict. The key, Thai explained, is to provide an unambiguous definition of the event that is narrow enough to remove doubt about when to report.

Regardless of the exact sampling design, researchers must decide how often, when, and even where they will ask participants to report their experiences.

If context-specific effects are of interest (e.g., work vs. home), think about when your participants will experience the different contexts and how many data points to include in each. If you’re worried about recall bias, increase the sampling frequency. And when it comes to timing, incorporating participants’ schedules into your sampling protocol can ensure you don’t ping them when they’re sleeping or otherwise don’t want to be disturbed, all of which can vary throughout the week or month.

If optimizing the experience sampling protocol sounds like a lot of work, use one of the widely available apps that help organize all of these parameters. An example is Thai’s ExperienceSampleropens in new window, a free open-source tool with deep customization that can be used on many types of devices, unlike other apps that are written for either iOS or Android. It requires researchers to do a bit of coding but comes with a tutorial geared toward nonprogrammers so they can adapt the script themselves. “We try to use all open-source, low-cost, or no-cost software to help researchers build the infrastructure needed to run their own study,” Thai said.

Chanel Meyers, an assistant professor of psychology at the University of Oregon, first heard of ExperienceSampler at the Society for Personality Social Psychology conference in 2016. She used it to track Hawaiian participants’ exposure and interactions to people of other races. “The app perfectly fit my needs,” said Meyers, who is now using the app for a second study on microaggressions. “The data we obtained was really rich and layered. I don’t think I would have been able to capture that kind of data had I not used this methodology.”

Other apps that support experience sampling range in cost from free to thousands of dollars per year. They vary in terms of being open-source, offline-capable, encryption-ready, or able to capture geolocation, motion, and camera data, among other features. Two other free or low-cost options are SEMA3opens in new window and Murmurasopens in new window, which Thai says are simple to use and loaded with helpful features and backed by reliable tech support.

• Incentivize participants. Compliance rates typically go up if participants are compensated for their time as well as for each task they complete.
• Keep surveys and tasks short. Research suggests one-off questionnaires should have no more than 25 to 30 questions and require no more than 30 minutes to maintain participants’ interest and attention. Surveys administered multiple times daily should take only a few minutes.
• Keep the wording simple. Think about a meta perception study along the lines of ‘What do you think I think you think of me?’ Such mental gymnastics can be burdensome and tiring.
• Get buy-in. Explaining the importance of your research to participants, as well as underscoring how helpful timely responses are, can increase compliance.
• Train participants. If you can’t bring down the complexity of the task, spend time up front training participants about your construct so, for example, they’ll understand how to recognize events that match your definition or better understand what kinds of social comparisons you want them to report on.