Advanced Methods for Conducting Online Behavioral Research
ISBN: 978-1-4338-0695-7The Internet is revolutionizing the way psychologists conduct behavioral research. Studies conducted online are not only less error-prone and labor-intensive but also rapidly reach large numbers of diverse participants at a fraction of the cost of traditional methods. In addition to improving the efficiency and accuracy of data collection, online studies provide automatic data storage and deliver immediate personalized feedback to research participants—a major incentive that can exponentially expand participant pools. Furthermore, behavioral researchers can also track data on online behavioral phenomena, including Instant Messaging (IM), social networking, and other social media.
This book goes beyond the basics to teach readers advanced methods for conducting behavioral research on the Internet. Short chapters offer practical advice by leading experts in key domains of Internet research. Readers are shown, step by step, how to conduct online experiments, surveys, and ability testing, use advanced graphic tools such as drag-and-drop objects, apply automatic text analysis tools, check the validity of protocols, automate the storage and analysis of data, record "field notes" on the behavior of online subjects and chatroom or blogging communities, and much more. Chapters also address critical issues such as data security, ethics, participant recruitment, and how to ensure the completion of tests or questionnaires. This volume also features supplemental resources, links, scripts, and instructions to further assist readers with their online research. See the supplemental materials tab for details.
This book is designed for researchers and advanced graduate students in the behavioral sciences seeking greater technical detail about emerging research methods. Readers will be well equipped to implement and integrate these exciting new methods into their own Internet-based behavioral research effectively, securely, and responsibly.
Contributors
I. Getting Started
- How to Use This Book
—John A. Johnson and Samuel D. Gosling - An Overview of Major Techniques of Web-Based Research
—Michael H. Birnbaum
II. Considerations When Designing Web Pages
- Design and Formatting in Internet-Based Research
—Ulf-Dietrich Reips - Using Graphics, Photographs, and Dynamic Media
—John H. Krantz and John E. Williams - Drag & Drop: A Flexible Method for Moving Objects, Implementing Rankings, and a Wide Range of Other Applications
—Wolfgang Neubarth
III. Studying Internet Behavior
- Collecting Data From Social Networking Web Sites and Blogs
—Elizabeth Mazur - Using Automated "Field Notes" to Observe the Behavior of Online Subjects
—Sonja Utz - Automatic Text Analysis
—Matthias R. Mehl and Alastair J. Gill
IV. Transporting Traditional Methodologies to the Web
- Internet-Based Ability Testing
—Ulrich Schroeders, Oliver Wilhelm, and Stefan Schipolowski - Web-Based Self-Report Personality Scales
—John A. Johnson - Online Collection of Informant Reports
—Simine Vazire - Conducting Online Surveys
—Tracy L. Tuten - Conducing True Experiments on the Web
—Ulf-Dietrich Reips and John H. Krantz
V. Cross-Cutting Issues
- Using Lotteries, Loyalty Points, and Other Incentives to Increase Participant Response and Completion
—Anja S. Göritz - Security and Data Protection: Collection, Storage, and Feedback in Internet Research
—Olaf Thiele and Lars Kaczmirek - Ethical Issues in Psychological Research on the Internet
—Tom Buchanan and John E. Williams
Index
About the Editors
Samuel D. Gosling, PhD, is a professor of psychology at the University of Texas at Austin. He has been using the Internet to collect data since the mid 1990s, when he created a questionnaire to collect personality ratings of pets by their owners. Since then, he has published numerous articles that make use of data collected on the Internet; these articles focus on such diverse topics as personality change over the life span, the links between music preferences and personality, geographic variation in psychological traits, and perceptions of others based on their Web sites and their online social networking profiles (e.g., Facebook). His 2004 American Psychologist article focused on evaluating the pros and cons of Internet methods.
Dr. Gosling's substantive research has focused on animal personality and on how human personality is manifested in everyday contexts like bedrooms, offices, clothing, Web pages, and music preferences. The latter topic was summarized in his book, Snoop: What Your Stuff Says About You (2008).
Dr. Gosling is a recipient of an American Psychological Association Distinguished Scientific Award for Early Career Contribution.
John A. Johnson, PhD, is a professor of psychology at the Pennsylvania State University, DuBois. He also serves as the consultant for the International Personality Item Pool, a Web-based repository for psychological measures in the public domain. He entered the field of computer-assisted psychological research in 1986, when he wrote microcomputer programs for scoring and interpreting the Hogan Personality Inventory. When the World Wide Web emerged in the 1990s, he transported concepts from these programs to the Web. He has published research on assessing the validity of data collected on the Internet and on sharing data through Web-based collaboratories.
This content was submitted by the authors as supplemental material for Advanced Methods for Conducting Online Behavioral Research.
The material is offered for personal use but the reader is expected to respect the intellectual property of the authors and the copyright holder. Content should not be reprinted or adapted for publication without permission from the authors and the copyright holder.
- Chapter 2: An Overview of Major Techniques of Web-Based Research
Michael Birnbaum - Chapter 3: Design and Formatting in Internet-Based Research
Ulf-Dietrich Reips - Chapter 4: Using Graphics, Photographs, and Dynamic Media
John Krantz and John E. Willams - Chapter 5: Drag and Drop: A Flexible Method for Moving Objects, Implementing Rankings, and a Wide Range of Other Applications
Wolfgang Neubarth - Chapter 6: Collecting Data from Social Networking Websites and Blogs
Elizabeth Mazur - Chapter 7: Using Automated "Field Notes" to Observe the Behavior of Online Subjects
Sonja Utz - Chapter 8: Automatic Text Analysis
Matthias Mehl and Alastair Gill - Chapter 9: Internet-Based Ability Testing
Ulrich Schroeders, Oliver Wilhelm, and Stefan Schipolowski - Chapter 10: Web-Based Self-Report Personality Scales
John Johnson - Chapter 11: Online Collection of Informant Reports
Simine Vazire - Chapter 12: Conducting Online Surveys
Tracy L. Tuten - Chapter 13: Conducting True Experiments on the Web
Ulf-Dietrich Reips and John Krantz - Chapter 14: Using Lotteries, Loyalty Points, and Other Incentives to Increase Participant Response and Completion
Anja Göritz - Chapter 15: Security and Data Protection: Collection, Storage, and Feedback in Internet Research
Olaf Thiele and Lars Kaczmirek
Chapter 2
An Overview of Major Techniques of Web-Based Research
Michael Birnbaum
Examples from the Text
- Exhibit 2.1 Webform for testing purposes
https://ati-birnbaum.netfirms.com/mbirnbaum/BareForm.htmopens in new window - Data.txt
https://ati-birnbaum.netfirms.com/data/data.txtopens in new window - Instructions for Installing a Generic Perl Script
https://ati-birnbaum.netfirms.com/Install_Perl_script.htmopens in new window - Bare Bones Website containing two webforms
https://ati-birnbaum.netfirms.com/opens in new window - SurveyWiz
https://ati-birnbaum.netfirms.com/surveywiz3.htmopens in new window - Example Website
https://ati-birnbaum.netfirms.com/opens in new window
Example Website hosted by Netfirms with many additional resources by Dr. Birnbaum. - Examples of JavaScript
https://psych.fullerton.edu/mbirnbaum/brmic/opens in new window
Drs. Birnbaum and Wakcher (2002) show how JavaScript can be used to control an experiment on probability learning and present a basic tutorial with a series of examples.
Links to Additional Resources Discussed in Text
- Advanced Training Institutes in Psychology Experiments via the WWW
https://psych.hanover.edu/NSFATI/opens in new window
Instructional materials by J. H. Krantz for the creation and editing of media - Psychological Research on the Net
https://psych.hanover.edu/Research/exponnet.htmlopens in new window
A site that lists known psychological experiments on the Internet that can be utilized as way to recruit volunteer participants. - How to Conduct Behavioral Research over the Internet
https://www.web-research-design.net/opens in new window
This is the supplementary site for R. Fraley’s How to Conduct Behavioral Research over the Internet: A Beginner's Guide to HTML and CGI/Perl (2004). Various resources, examples, and more are available on it. - WWW Survey Assistant
https://www.mohsho.com/s_ware/how.html (site no longer available)
Everything you need to get your survey or test up and running on the web. No programming experience required! - Tools for Internet-based Data Collection
https://www.psychology.uzh.ch/en.htmlopens in new window
A repository of tools for Web experimenters.
Chapter 3
Design and Formatting in Internet-Based Research
Ulf-Dietrich Reips
Examples from the Text
- Web Survey List
Intended to be the world's most comprehensive archive of current and past psychological Web experiments - WEXTOR
https://wextor.orgopens in new window
WEXTOR is a Web-based tool that lets you quickly design and visualize laboratory experiments and Web experiments in a guided step-by-step process. - VAS Generator
https://vasgenerator.netopens in new window
A free tool to create VAS for Internet-based research. - Surveymonkey
https://surveymonkey.comopens in new window
Intelligent survey software for primates of all species. SurveyMonkey has a single purpose: to enable anyone to create professional online surveys quickly and easily. - Unipark
https://unipark.comopens in new window
EFS Survey is the online research tool of choice for over 300 businesses worldwide. Its friendly web-based interface allows users to create high quality surveys with minimal effort. - iScience Server
https://www.iscience.euopens in new window
A portal with a number of useful services in Internet-based research.
Chapter 4
Using Graphics, Photographs, and Dynamic Media
John Krantz and John E. Willams
- Image J A Scientific-grade image editing-program used for medical, chemical, and even neuroscience images, for which the need for high-quality image processing is a must.
- Cognition Laboratory Experiments
https://psych.hanover.edu/JavaTest/CLE/Cognition/cognition.htmlopens in new window
A collection of web experiments, many of which utilize Authorware or Flash. - Psychological Research on the Net
https://psych.hanover.edu/Research/exponnet.htmlopens in new window
An expansive list of psychological experiments on the web.
Chapter 5
Drag and Drop: A Flexible Method for Moving Objects, Implementing Rankings, and a Wide Range of Other Applications
Wolfgang Neubarth
Measurement of moveable objects
Teaching Example
- Basic Example (ZIP file)opens in new window
- Restricted Range Example (ZIP file)opens in new window
- Reorder Example (ZIP file)opens in new window
- Application Exampleopens in new window
- Download Zip File of Chapter 5 Files (ZIP file)opens in new window
Chapter 6
Collecting Data from Social Networking Websites and Blogs
Elizabeth Mazur
Ancillary Links
- comScore
https://www.comscore.comopens in new window - Pew Internet & American Life Project
https://www.pewinternet.orgopens in new window - Nielson/NetRatings
https://www.nielsen.com/us/en/press-releases/2015/nielsen-and-sun-broadcast-group-sign-multi-year-agreement-for-radar-ratings/opens in new window - Technorati
https://technorati.comopens in new window - CyberJournalist
https://www.cyberjournalist.netopens in new window - BlogSearchEngine
https://www.blogsearchengine.comopens in new window - EatonWeb - The Blog Directory
https://portal.eatonweb.comopens in new window - The BOBs
https://www.thebobs.com (site no longer available) - Second Life
https://www.secondlife.comopens in new window - Habbo
https://www.habbo.comopens in new window
Chapter 7
Using Automated "Field Notes" to Observe the Behavior of Online Subjects
Sonja Utz
- Newsreaders
https://www.newsreaders.comopens in new window - HTTrack
https://www.httrack.com/opens in new window - PuTTY
https://www.chiark.greenend.org.uk/~sgtatham/putty/opens in new window
PuTTY–Telnet client with log option. - LogBot
https://freshmeat.net/projects/logbot/opens in new window - Chat Watch 5
https://chatwatch.net/opens in new window - WebSpider 2
https://xaldon.com/node/32opens in new window - Web2Map
https://web2map.software.informer.com/opens in new window - GNU Wget
https://www.gnu.org/software/wget/wget.htmlopens in new window - Linguistic Data Consortium tips for creating annotated data
https://www.ldc.upenn.edu/communications/ldc-instituteopens in new window - HyperTRANSCRIBE 1.5
https://www.researchware.com/opens in new window - atlas.ti
https://www.atlasti.com/opens in new window - HyperRESEARCH™ 2.8
https://www.researchware.com/opens in new window - MAXQDA
https://www.maxqda.com/opens in new window - NVivo
https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/homeopens in new window - Advene
https://liris.cnrs.fr/advene/index.htmlopens in new window - Qualitative research in information systems
https://www.qual.auckland.ac.nz/opens in new window
Chapter 8
Automatic Text Analysis
Matthias Mehl and Alastair Gill
- LIWC
https://www.liwc.netopens in new window
This website is a great resource for all things relating to Linguistic Inquiry and Word Count. - Wmatrix
https://ucrel.lancs.ac.uk/wmatrix/opens in new window
Can be accessed directly through this site, in addition to providing details about how Wmatrix works and how to use it. - CLAWS Input / Output Format Guidelines
https://ucrel.lancs.ac.uk/claws/format.htmlopens in new window
Provides useful guidelines about how to format data for use with Wmatrix. - Latent Semantic Analysis
https://lsa.colorado.eduopens in new window
Use this site to access the web version of LSA and to find related resources. - Coh-Metrix
https://cohmetrix.memphis.eduopens in new window
Is a useful site for information and publications about (and also access to) Coh-Metrix.
Chapter 9
Internet-Based Ability Testing
Ulrich Schroeders, Oliver Wilhelm, and Stefan Schipolowski
Programming
- XAMPP
- PHP Freaks
https://www.phpfreaks.com/opens in new window
This site offers a great collection of articles about PHP programming, a code library, tutorials, and forums where you can post questions. - MySQL
https://dev.mysql.com/doc/refman/6.0/en/index.htmlopens in new window
You can use the MySQL 6.0 reference manual (located here) to look up the MySQL syntax, for instance, on how to write different types of data to your database. - PHP: Documentation
https://www.php.net/docs.phpopens in new window
Here you can find the official PHP documentation. - phpMyAdmin
https://www.phpmyadmin.netopens in new window
On this site you can get additional information about the program phpMyAdmin that is an easy to use tool for creating, editing, and exporting MySQL databases. - Java SE Desktop Technologies
https://java.sun.com/products/javawebstartopens in new window
On Java’s official site you can inform yourself about the Java Web Start technology if you are interested in accurately recording reaction times.
Web Sites for Recruiting Participants
- Queendom.com: Tests, Tests, Tests and more Tests
https://www.queendom.comopens in new window - Psychtests.com
https://www.psychtests.comopens in new window - Psychological Research on the Net
https://psych.hanover.edu/Research/exponnet.htmlopens in new window
Chapter 10
Web-Based Self-Report Personality Scales
John Johnson
- Informed consent form from Dr. Revelle
https://www.sapa-project.org/opens in new window - Informed Consent form by Dr. Johnson
https://drj.virtualave.net/fall2006/opens in new window - A Front page directing visitors to different Scales
- Disclaimer Page for the 120-item IPIP measure
https://drj.virtualave.net/IPIP/ipipneo120.htmopens in new window - Perl code for shortipipneo1.cgi
https://drj.virtualave.net/IPIP/shortipipneo1.txtopens in new window - Perl code for shortipipneo2.cgi
https://drj.virtualave.net/IPIP/shortipipneo2.txtopens in new window - Perl code for shortipipneo3.cgi
https://drj.virtualave.net/IPIP/shortipipneo3.txtopens in new window - Perl code for computing consistency indices for the 300-item IPIP measure
https://drj.virtualave.net/IPIP/JacksonConsistency.txtopens in new window
https://drj.virtualave.net/IPIP/GoldbergConsistency.txtopens in new window - Sample report for the IPIP-NEO
https://drj.virtualave.net/IPIP/SampleReport.pdfopens in new window - International Personality Item Pool (IPIP) Web site
https://ipip.ori.orgopens in new window - searchengines.com
https://www.searchengines.com/opens in new window - Psychological Research on the Net
https://psych.hanover.edu/Research/exponnet.htmlopens in new window - www.outofservice.com
https://www.outofservice.comopens in new window
Chapter 11
Online Collection of Informant Reports
Simine Vazire
- Collecting Informat Reports
https://www.simine.com/informants.htmlopens in new window
A page developed by Dr. Vazire containing information for researchers interested in collecting informant reports for their research. - WorldMerge
https://www.coloradosoft.com/worldmrgopens in new window
A program that allows you to use a Microsoft Excel spreadsheet as input and send out hundreds of individualized e-mails from your own e-mail account with the push of a button.
Chapter 12
Conducting Online Surveys
Tracy L. Tuten
- Harris Interactive
https://harris-interactive.co.uk/opens in new window - Greenfield Online
https://www.greenfield.comopens in new window - Confirmit
https://www.confirmit.comopens in new window - Inquisite
https://www.inquisite.orgopens in new window - SurveyMonkey.com
https://www.surveymonkey.comopens in new window - Web Surveyor
https://www.vovici.comopens in new window - Zoomerang
https://www.zoomerang.comopens in new window - Perseus Survey Solutions
https://www.vovici.comopens in new window
Chapter 13
Conducting True Experiments on the Web
Ulf-Dietrich Reips and John Krantz
Metasite
- iScience Server
https://iscience.euopens in new window
This is a free portal to many of the services mentioned below. Hosted by Reips at the University of Deusto in Bilbao, Spain – after three years at the University of Zurich, Switzerland.
Study Generators and Editors
- WEXTOR
https://wextor.orgopens in new window
This free Web application with small fee for comfort features, creates laboratory experiments and Web experiments and will work with between-subjects, within-subjects, quasi-experimental, and mixed designs. Many techniques important to Web research are built in. Its user friendly step-by-step process is also great for teaching students. WEXTOR stores and hosts the experimental designs, so you can work on and run your experiments using any type of browser from any computer. - FactorWiz
https://psych.fullerton.edu/mbirnbaum/programs/factorWiz.htmopens in new window
This freeware generates HTML pages to conduct within-subject factorial experiments with random order of conditions. - VAS Generator
https://www.vasgenerator.net/opens in new window
This freeware easily creates visual analogue scales for Web use. - Generic HTML Form Processor
https://www.goeritz.net/brmic/opens in new window
This “Citeware” collects data from Web questionnaires using PHP. - Scientific LogAnalyzer
https://sclog.euopens in new window
This program is free for small log files in academic use and can analyze any type of log file. It also does dropout analysis.
Examples, Recruitment, and Archiving
- Decision Research Center
https://psych.fullerton.edu/mbirnbaum/decisions/thanks.htm (site no longer available)
Several student experiments that are research projects in judgment and decision making, supervised by Michael Birnbaum, one of the leading experts in Internet-based data collection. He writes on the Web page introducing the Decision Research Center: “As with any content you find on the Internet, after you link to a study, you should read the materials and decide for yourself if you want to participate.” - Psychological Research on the Net (“Exponnet site”)
https://psych.hanover.edu/research/exponnet.htmlopens in new window
One of the most comprehensive Web sites listing online psychology related studies. The Web site is maintained by the second author at Hanover College. - Socialpsychology Network
https://www.socialpsychology.org/expts.htmopens in new window
A Web site devoted to listing Internet-based data collection efforts specifically in the area of social psychology. - Web Experiment List
Together with the “Exponnet list” mentioned above this is the largest list of Internet-based research studies on the Web. Researchers can fill in their own studies to be listed. Studies can be searched by category, language, status (active or not), and type (experiment or correlational study). - Web Experimental Psychology Lab
https://wexlab.euopens in new window
This was the first virtual laboratory with real data collection via experiments on the World Wide Web, founded in 1995 by the first author. Then located at the University of Tübingen, Germany, it moved to the University of Zurich, Switzerland, in 1997, and since 2009 is located at the University of Deusto in Bilbao, Spain.
Chapter 14
Using Lotteries, Loyalty Points, and Other Incentives to Increase Participant Response and Completion
Anja Göritz
- Meta-analysis of incentive effects
https://www.goeritz.net/opens in new window - phpPanelAdmin List Management Software
https://www.goeritz.net/panelwareopens in new window
Chapter 15
Security and Data Protection: Collection, Storage, and Feedback in Internet Research
Olaf Thiele and Lars Kaczmirek
The following information is only for educational purposes. Readers are fully responsible on their own if they use these links. These products are designed to secure data and can therefore also cause loss of data and restrict access to data. The authors should not be held liable if such things happen.
Websites
- Verification institution Versign
https://www.verisign.com/opens in new window - Verification institution CAcert
https://www.cacert.org/opens in new window - Webservice Tinyurl
https://tinyurl.com/opens in new window
Software Links
- Truecrypt, data encryption
https://www.truecrypt.orgopens in new window - Locknote, File encryption
https://www.steganos.com/en/opens in new window - Eraser, Secure deletion of data
https://eraser.heidi.ie/opens in new window
Additional Resources
- Bruce Schneier Blog and Newsletter
https://www.schneier.comopens in new window
Bruce Schneieropens in new window has written a well known cryptography text book (Applied Cryptography) and offers an excellent blog and newsletter. The Crypto-Gram newsletter has more than 125.000 readers and regularly informs them about security related problems. - PHP and MySQL Documentation
https://www.php.net/manual/en/opens in new window
https://mysql.org/doc/opens in new window
Both PHP and MySQL offer important great documentation on available functions and commands. Each website has a section where it is possible to add comments to the official documentation, which usually gives good advice on security and other issues. The PHP websiteopens in new window offers a good section on overall PHP security. The MySQL documentationopens in new window presents a shorter section on security, but generally informs well on database subjects. - Mailing Lists on Current Vulnerabilities
https://www.kb.cert.org/vuls/opens in new window
https://www.cert.org/opens in new window
Local CERTs (Computer Emergency Readiness Teams) offer informative official mailing lists on vulnerabilities. The US-CERTopens in new window offers constantly updated technical information (approx. once a day). Other CERTs can be found on the CERT websiteopens in new window.

