Fascinated by the level of worry over Y2K, late last year Lisa Aspinwall, PhD, decided to conduct a last-minute study on people's millennial fears. Knowing she had little time to collect data, she decided to put a survey on the Internet. Her site was up and running by November, and by Jan. 1, she had more than 2,000 visitors, of whom 850 filled out usable surveys.
She's still analyzing her results, but she already considers the study a success. She got a very high response rate from a diverse sample of people in a short time period. And she saved money because she didn't have to print copies of the survey or pay someone to administer it and input the data.
But Web-based experiments present their own problems, as well. Critics point out that it's difficult to control the study environment because Web users use unlimited types of software, hardware and Internet connections. In addition, study participants are largely unmonitored: There's no one watching to see that they're being honest about their age or gender, that they aren't getting help from a friend or that they aren't cheating by using reference materials.
Even more critical, admit researchers, is the fact that people who participate in online experiments are self-selected and by no means random or representative of the general population.
These are real issues, admit proponents of Web-based research. But, they argue, the benefits still outweigh the disadvantages. And, besides, improvements in technology are already addressing many of the problems.
In addition, say proponents of Web-based research, many of the problems critics point to aren't unique to the Internet. Of course, Internet samples aren't random or representative, they say. Neither are the "convenience" samples used in the average psychology study.
"No one has ever gotten a random sample in the lab," says Hanover College psychologist John Krantz, PhD, who maintains the APS Web experiments list. "But Web users are certainly more diverse than college sophomores. And the interesting thing in terms of validity is that people get the same results with college and Web samples."
Krantz has research to back that up. For a chapter appearing in "Psychological Experiments on the Internet" (Academic Press, 2000), he reviewed all the studies to date that compared results from Web-based and laboratory-based samples. He found that the data match up beautifully--there's little difference between results gotten in the lab and those obtained online.
As for the diversity of study participants, "most studies on the representation of Web-study participants suggest that, if anything, those populations are more representative of the public than samples from more traditional lab experiments using college students," says Wesleyan University's Scott Plous, PhD, who's been tracking Web-based research as Webmaster of Social Psychology Network (www.socialpsychology.org/).
But as with college students, people on the Internet are skewed toward the high end of the socio-economic and educational spectrums, says Minnesota State University's Linda Jackson, PhD, who recently reviewed the demographics of Internet study participants.
"For some questions--such as basic cognitive processes--it probably doesn't matter," she admits. "But for issues of prejudice, attitudes and the like, we probably need to be more careful."
In the end, researchers who use the Web should recognize and report that their samples aren't globally, or even nationally, representative, adds Aspinwall.
If researchers want a controlled sample, they should limit who accesses their research site rather than open up a Web-based study to the world, adds University of Maryland psychologist Kent Norman, PhD, a pioneer of Internet research.
"You can set up studies on your university Web server and then limit access through a password system," he says.
Researchers can also gain access to special populations of people through the Internet. Michael Birnbaum, PhD, of California State University, Fullerton, for example, sent e-mail to members of statistics-related associations asking them to participate in a Web-based study examining how people with expertise in statistics fared on a decision-making task. The same study conducted via regular mail would be costly, time-consuming and result in a relatively small sample. But within a few days of sending out his e-mail notes, Birnbaum had several hundred responses.
Low cost, high speed
Along with obtaining larger sample sizes through the Web, online research saves money, a major advantage to researchers at small colleges and universities, says Hanover College's Krantz. Not only do researchers save on laboratory space, and printing of study materials, but labor costs are reduced because researchers don't need to hire lab assistants to man their labs or input data.
In addition, researchers can do more experiments in a shorter period of time--something the University of Pennsylvania's Jonathan Baron, PhD, and his students do. On average, they post a study a week devoted to research on judgment and decision-making. Baron monitors the site to see if people understand the study questions. If they don't, he pulls the study, discards the data and makes changes at little cost of money or time.
"On the Web, it's so easy to make changes," he says.
And he draws so many participants, it's easy to run a study over and over. In fact, now that his site is well-known, he reaches his goal of 50 participants per study within three hours.
Baron's high participation rate is due in part to the fee he pays to participants--between $1 and $6 per study. In fact, Baron secured a $15,000 per year grant from the National Science Foundation just for paying Internet participants.
Of course, if an incentive is too good, some people may come back many times. But the risk to studies of people submitting more than one set of data is probably smaller than most fear, according to data collected by Ulf-D. Reips, PhD, of the universities of Tübingen and Zürich. For one of his studies, he monitored the number of responses he received from the same computer--something researchers can track through a code that uniquely identifies all computers. If he omitted all but the first set of data from each computer, he found it made no difference to his results.
Web research poses other potential drawbacks besides repeat participation. Although most types of psychology research will translate well to the Internet, some may not, says Krantz.
For example, he's not yet convinced that he can control visual or auditory stimuli well enough over the Web to conduct a valid experiment in his main research area of sensation and perception. In particular, people on the Internet use different hardware and software and connect via systems with variable connection speeds. There's no way to ensure that everyone who participates in an experiment will receive exactly the same stimuli in terms of sound, color or timing, says Krantz.
Lack of control over the study environment is certainly a concern, says Kenneth McGraw, PhD, of the University of Mississippi, who runs the PsychExperiments Web site (www.olemiss.edu/psychexps/). But he believes the technology will eventually become sophisticated enough to solve many of the problems. He thinks he's already solved the timing issue--which is critical for researchers who measure reaction times or priming effects.
Part of the trick is that his experiments and the software needed to run them download directly onto a participant's computer, eliminating differences in connection speed. Using this technique and programming software called Authorware, McGraw and his colleagues tested whether they could measure reaction times over the Internet as accurately as in the laboratory. They found they could. On a standard reaction time task, their results were accurate to the millisecond. The results will be published in Psychological Science.
"We got data as good as people get in the lab," says McGraw.
And as Web-related research technology matures, opportunities for more creative and interactive experiments will only grow, say Web-savvy psychologists. Even now, the benefits of larger and more diverse study samples and less costly studies far outweigh the disadvantages for most types of psychology research, they say. Indeed, for most, the promise of big study samples--with enough statistical power to wipe out any anomalous data--is reason enough to move from lab to Web.
Says UPenn's Baron: "There are so many advantages to doing research on the Internet, any disadvantages get overshadowed. I'm so convinced, all of my research is now done on the Web."
Birnbaum, M.H. (Ed.). (2000). "Psychological Experiments on the Internet." San Diego: Academic Press.
Birnbaum, M.H. (2000). "Introduction to Behavioral Research on the Internet." Upper Saddle River, N.J.: Prentice-Hall.
Letters to the Editor
- Send us a letter