skip to main content

Appendix: Guidelines for reading questions and interpreting data

There are multiple ways to ask questions and collect data when conducting survey research. It is important to think clearly about what the goal of each question is — what is the purpose of the question? — so that the best format can be selected. Similarly, once data are collected, reporting and interpreting the data accurately are equally important, and maintaining an understanding of the question structure will allow for correct interpretation.

We have provided a few examples here to help readers better understand the purpose of different question structures and the importance of using accurate language when interpreting the data. This does not cover every kind of question that can be asked in survey research; rather, we have included a few question structures that we have found are commonly misreported.

APA Style leaf logo Cite this in APA Style
American Psychological Association. (2014, December 21). 2014 Stress in America Appendix. https://www.apa.org/news/press/releases/stress/2014/appendix

expand all

 

Q625: Please indicate how significant a source of stress the following is in your life:

 

Very Significant

Somewhat Significant

Not Very Significant

Not at All Significant

Money

31%

33%

20%

16%

Work

22%

38%

23%

17%

Relationships (e.g., spouse, kids, girl/boyfriend)

17%

27%

26%

30%

Health problems affecting my family

17%

26%

26%

31%

Family responsibilities

17%

30%

27%

26%

Housing costs (e.g., mortgage or rent)

16%

24%

25%

35%

The economy

16%

34%

29%

22%

Job stability

16%

19%

19%

46%

Personal health concerns

15%

30%

29%

25%

Personal safety

6%

18%

33%

44%

BASE: All respondents (Adults n=3068)
What is a grid question?
  • Allows respondents to evaluate multiple attributes or characteristics using the same scale (e.g., attitudes, perceptions, behaviors, etc.).
  • Scales can vary and measure a wide variety of attitudes or behaviors (e.g., importance, agreement, likelihood, favorability, etc.).
What does a grid question measure?
  • Measures the level of endorsement for a particular attitude, perception or behavior (each of the attributes evaluated).
  • Attributes are evaluated independently.
  • Provides insight into the strength and depth of feeling for each attribute.
What does a grid question not measure?
  • Does not measure absolutes.
  • Does not ask respondents to “rank” attributes.
  • Reporting should reflect that respondents did not “rank” items against each other.
Reporting: 

When reporting on data collected in a grid type question, it is important to remember what these questions do and do not measure.

Type of Reporting

Correct

Incorrect

Rationale

… when reporting on individual attributes evaluated in the grid

Sixty-four percent of adults say money is a somewhat/very significant source of stress.

Sixty-four percent of adults say money is a significant source of stress, with 31 percent saying it is very significant and 33 percent saying it is a somewhat significant source of stress.

Money is the most significant source of stress (64 percent).

 

The question measures the degree of significance placed on each item.

In each of these examples, the incorrect statement is misleading because the finding is reported as if the question asked for a “ranking” rather than a “rating.”

With the grid type question format, respondents are not asked to make a direct comparison between the attributes that may have resulted in a different finding. As such, while we can discuss which attributes the sample is most likely to rate “significant,” we cannot state that one is more significant than another.

 


… when comparing multiple attributes

Many adults say that money (31 percent) and work (22 percent) are a very significant source of stress in their life.

Very significant sources of stress are money (31 percent) and work (22 percent), followed by relationships, health problems affecting family and family responsibilities equally, at 17 percent.

The most significant sources of stress are money (31 percent) and work (22 percent).

Americans rank money (31 percent) ahead of work (22 percent) as a source of stress.

Money (31 percent) is a more significant source of stress than work (22 percent).

… when comparing subgroups on individual attributes

Those that earn <$50,000 (74 percent) are more likely than those that earn >=$50,000 (57 percent) to say that money is a very/ somewhat significant source of stress in their life.

More lower income adults (<$50,000; 74 percent) say money is a very or somewhat significant source of stress than those who earn more (>=$50,000; 57 percent).

Those who earn <$50,000 say money (74 percent somewhat/ very significant) is a more significant source of stress than work (60 percent).

Grid type questions: Key takeaways
  • Grid type questions measure multiple attributes across the same scale.
  • They are “rating” questions, not “ranking” questions.
  • They measure attributes as they relate to each other, but not absolutes.
  • They indicate the level of endorsement (e.g., importance, agreement, etc.) for each attribute.
Q2250: In general, would you say your health is ...

Excellent

9%

Very good

29%

Good

42%

Fair

18%

Poor

2%

 BASE: All respondents (Adults n=3068)
What is a simple scaled question ?
  • Simple scaled questions ask about specific attitudes or behaviors.
  • Scales that can vary and measure a wide variety of attitudes or behaviors (e.g., importance, agreement, likelihood, favorability, etc.).
What does a simple scaled question measure?
  • Measures the level of endorsement for a specific attitude, perception or behavior.
What does a simple scaled question not measure?
  • Does not measure how the item being measured relates to other attitudes or behaviors.
Reporting: 

When reporting on simple scaled questions it is important to remember that these questions answer only the specific question asked. Errors in reporting are less common than with grid type questions described above.

Type of Reporting

Correct

Incorrect

Rationale

… when reporting on individual attributes evaluated in the grid

Most adults report their overall health as good (42 percent) or very good (29 percent).

Very few (nine percent) would say their overall health is excellent.

Thirty-eight percent of adults are in excellent or very good health.

Most adults are in good health (42 percent) and very few (two percent) are unhealthy.

 

The incorrect findings are not specific enough. The question specifically asked respondents to evaluate their own health; it does not represent objective measures of health or the opinion of a qualified health care professional, which may differ from the self-report.

… when comparing subgroups on individual attributes

Both men (39 percent) and women (36 percent) are likely to think their health is excellent or very good.

Men and women are equally healthy (39 percent and 36 percent, respectively).

Simple scaled questions: Key takeaways
  • Measure specific attitudes or behaviors.
  • Findings indicate the level of endorsement (e.g., importance, agreement, etc.) for specific attitudes or behaviors.
Q965: Do you do any of the following to help manage stress? Please select all that apply.
Top 10 most common responses:

Listen to music

44%

Exercise or walk

43%

Watch TV/movies for more than two hours per day

40%

Surf the Internet/go online

38%

Read

36%

Spend time with friends or family

35%

Pray

29%

Nap/sleep

27%

Spend time doing a hobby

24%

Eat

23%

BASE: All respondents (Adults n=3068)
What is a multiple response question?
  • Used to understand a range of attitudes, behaviors or perceptions.
  • Provides insight into the prevalence of different attitudes, behaviors or perceptions.
What does a multiple response question measure?
  • Prevalence of attitudes, behaviors or perceptions.
What does a multiple response question not measure?
  • Does not necessarily measure the frequency of a specific attitude, behavior or perception.
  • Does not necessarily measure the strength of the attitude or perception measured.
  • Does not specifically capture preference (i.e., “favorites”) or rank order among attitudes, behaviors or perceptions.
Reporting:

When reporting on data collected from a multiple response question, it is important to remember that they measure prevalence. They do not necessarily measure frequency, strength of endorsement or preference. Rather, these data are useful to understand the range of behaviors or attitudes on a given topic.

Type of Reporting

Correct

Incorrect

Rationale

… when reporting at the aggregate level

The most common ways people manage stress are listening to music, exercising/walking, and watching TV/movies for more than two hours a day.

Two out of 5 adults listen to music as a way to manage stress (44 percent).

Listening to music, exercising/walking, and watching TV/movies for more than two hours a day are the most popular ways to manage stress.

Listening to music is the most frequent stress management technique.

 

In reporting, use of the word “frequently” or “frequency” implies how often a behavior is done.

This question, as phrased, measures prevalence (i.e., how many people are doing these activities) rather than actual frequency (i.e., how many times per week or month they are doing each of these).

When reporting on subgroups we know that more people from a particular subsample (i.e., women) engage in a behavior as compared to men. The question does not address the question of whether those women engaging in the behavior actually do the behavior more often than men.

It is important to consider the whole question as it was asked. As such, results from this question cannot, for example, be used to measure the likelihood of listening to music overall — only the likelihood of listening to music for the specific purpose of managing stress.

… when comparing multiple attributes

Exercising (43 percent) and listening to music (44 percent) are equally likely to be used as stress management techniques.

Listening to music is more commonly mentioned as a stress management strategy than napping.

Listening to music (44 percent) is the stress management technique embraced by the highest percentage of adults, followed by exercising (43 percent).

Listening to music (44 percent) is done more frequently than exercising (43 percent) when it comes to stress management.

Adults exercise more than read to manage stress (43 percent versus 36 percent).

Adults prefer listening to music (44 percent) over watching TV (40 percent) as a way to manage stress.

… when comparing subgroups on individual attributes

Women (44 percent) are more likely than men (26 percent) to say they read to manage their stress.

More men (23 percent) than women (16 percent) play video games to manage stress.

 

 

Women read more frequently than men as a stress management technique.

Men play video games more often than women to manage stress.

Multiple response questions: Key takeaways
  • Measures prevalence of attitudes, behaviors, perceptions.
  • Provides insight into a range of behaviors or attitudes on the specific topic in question.
  • Do not necessarily measure frequency of behaviors.
  • Do not necessarily measure strength of an attitude or perception.
Date created: 2014