Guidelines for reading questions and interpreting data

There are multiple ways in which to ask questions and collect data when conducting survey research. It is important to think clearly about what the goal and purpose of each question is so that the best format can be selected. Once data are collected, reporting and interpreting the data accurately is as important as asking the questions. Maintaining an understanding of the question structure will allow for correct interpretation.

We have provided a few examples here to help Stress in America™ readers better understand the purpose of different question structures and the importance of using accurate language when interpreting the data. In this methodology, we do not cover every kind of question that can be asked in survey research; rather, we include a few question structures that we have found are commonly misreported.

Grid questions — evaluating multiple attributes or characteristics on the same scale

 

How important are each of the following to you?

 

Extremely Important

Very Important

Important

Somewhat Important

Not at All Important

Having good relationships with my family

51%

26%

17%

5%

2%

Getting enough sleep

30%

33%

28%

7%

2%

Doing well in my
career/studies/school

30%

30%

27%

9%

3%

Having good relationships with my friends

29%

33%

28%

9%

2%

Managing stress

28%

33%

27%

8%

4%

Eating healthy

24%

31%

31%

12%

2%

Being physically active or fit

24%

26%

33%

14%

3%

BASE: All respondents (Adults n=1950)
What is a grid question?
  • Allows respondents to evaluate multiple attributes or characteristics using the same scale.
    • Attitudes, perceptions, behaviors, etc.
  • Has scales that can vary and measure various attitudes or behaviors.
    • Importance, agreement, likelihood, favorability, etc.
What does a grid question measure?
  • Measures the level of endorsement for a particular attitude, perception or behavior.
  • Evaluates attributes independently.
  • Provides insight into the strength and depth of feeling for each attribute.
What does a grid question not measure?
  • Does not measure absolutes.
  • Does not ask respondents to “rank” attributes.
  • Reporting should reflect that respondents did not “rank” items against each other.

Reporting: When reporting on data collected in a grid question, it is important to remember what these questions do and do not measure.

Type of Reporting

Correct

Incorrect

Rationale

… when reporting on individual attributes evaluated in the grid

Seventy-seven percent of adults think having good relationships with family is extremely or very important.

Seventy-seven percent of adults think having good relationships with family is important, with 51 percent saying it is extremely important and 26 percent saying it is very important.

Having good relationships with family is the most important thing in people’s lives (51 percent).

The question measures the degree of importance placed on each item.

In each of these examples, the incorrect statement is misleading because the finding is reported as if the question asked for a “ranking,” rather than a “rating.”

With the grid question format, respondents are not asked to make a direct comparison between the attributes that may have resulted in a different finding. As such, while we can discuss which attributes the sample is most likely to rate “important,” we cannot state that one is more important than another.

… when comparing multiple attributes

Many adults believe that having good relationships with family (51 percent), doing well in my career/ studies/school (30 percent) and having good relationships with friends (29 percent) are extremely important to them.

The areas of life most commonly seen as important are having good relationships with family (51 percent), doing well in my career/studies/school (30 percent) and having good relationships with friends (29 percent).

The most important areas of life are having good relationships with family (51 percent), doing well in my career/studies/school (30 percent) and having good relationships with friends (29 percent).

Americans rank having good relationships with family (51 percent) ahead of doing well in my career/studies/school (30 percent).

Having good relationships with family (51 percent) is more important than having good relationships with friends (29 percent).

… when comparing subgroups on individual attributes

Women (80 percent) are more likely than men (73 percent) to think having good relationships with family is extremely or very important.

More women (80 percent) than men (73 percent) think having good relationships with family is extremely or very important.

 

Women think that having good relationships with family (80 percent extremely or very important) is more important than having good relationships with friends (65 percent).

Grid questions: Key takeaways
  • Grid questions measure multiple attributes across the same scale.
  • They are “rating” questions, not “ranking” questions.
  • They measure attributes as they relate to each other, but not absolutes.
  • They indicate the level of endorsement (e.g., importance, agreement, etc.) for each attribute.
Simple scaled questions — eliciting a specific attitude or behavior from a respondent
How would you rate your overall health?

Excellent

7%

Very Good

28%

Good

45%

Fair

18%

Poor

3%

BASE: All respondents (Adults n=1950)
What is a simple scaled question ?
  • Asks about specific attitudes or behaviors.
  • Has scales that can vary and measure various attitudes or behaviors.
    • Importance, agreement, likelihood, favorability, etc.
What does a simple scaled question measure?
  • Measures the level of endorsement for a specific attitude, perception or behavior.
What does a simple scaled question not measure?
  • Does not measure how the item being measured relates to other attitudes or behaviors.

Reporting: When reporting on simple scaled questions, it is important to remember that these questions answer only the specific question asked. Errors in reporting are less common than with grid questions, described previously.


Type of Reporting

Correct

Incorrect

Rationale

… when reporting on individual attributes

Most adults report their overall health as good (45 percent) or very good (28 percent).

Very few (7 percent) would say their overall health is excellent.

Thirty-five percent of adults are in excellent or very good health.

Most adults (45 percent) are in good health and very few (3 percent) are unhealthy.

The incorrect findings are not specific enough. The question specifically asked respondents to evaluate their own health; it does not represent objective measures of health or the opinion of a qualified health care professional, which may differ from the self-report.

… when comparing subgroups 

Both men (35 percent) and women (34 percent) are likely to think their health is excellent or very good.

Men and women are equally healthy (35 percent and 34 percent, respectively).

 

Simple scaled questions: Key takeaways
  • Simple scaled questions measure specific attitudes or behaviors.
  • Their findings indicate the level of endorsement (e.g., importance, agreement, etc.) for specific attitudes or behaviors.
Multiple response questions — asking respondents to report on a range of behaviors, attitudes or perceptions
Do you do any of the following to help manage stress? Please select all that apply.
Top ten most common responses:

Listen to music

48%

Exercise or walk

43%

Surf the Internet/Go online

42%

Watch TV or movies for more than two hours per day

40%

Read

39%

Spend time with friends or family

36%

Nap/Sleep

32%

Pray

30%

Spend time doing a hobby

28%

Eat

27%

BASE: All respondents (Adults n=1950)
What is a multiple response question?
  • Used to understand a range of attitudes, behaviors or perceptions.
  • Provides insight into the prevalence of different attitudes, behaviors or perceptions.
What does a multiple response question measure?
  • Measures the prevalence of attitudes, behaviors or perceptions.
What does a multiple response question not measure?
  • Does not necessarily measure the frequency of a specific attitude, behavior or perception.
  • Does not necessarily measure the strength of the attitude or perception measured.
  • Does not specifically capture preference (i.e., “favorites”) or rank order among attitudes, behaviors or perceptions.

Reporting: When reporting on data collected from a multiple response question, it is important to remember that these questions measure prevalence. They do not necessarily measure frequency, strength of endorsement or preference. Rather, these data are used to understand the range of behavior or attitudes on a given topic.


Type of Reporting

Correct

Incorrect

Rationale

… when reporting at the aggregate level

The most common ways people manage stress are listening to music, exercising or walking, and surfing the Internet.

Roughly half of adults listen to music as a way to manage stress (48 percent).

Listening to music, exercising or walking, and surfing the Internet are the most popular ways to manage stress.

Listening to music is the most frequent stress management technique.

In reporting, use of the word “frequently” or “frequency” implies how often a behavior is done.

This question, as phrased, measures prevalence (i.e., how many people are doing these activities) rather than actual frequency (i.e., how many times per week or month they are doing each of these).

When reporting on subgroups, we know that more people from a particular subsample (e.g., women) engage in a behavior as compared to another subsample (e.g., men). The question does not address the question of whether those women engaging in the behavior actually do so more often than men.

It is important to consider the whole question as it was asked. As such, results from this question cannot, for example, be used to measure the likelihood of listening to music overall — only the likelihood of listening to music for the specific purpose of managing stress.

… when comparing multiple attributes

Exercising (43 percent) and surfing the Internet (42 percent) are equally likely to be used as stress management techniques.

Listening to music is more commonly mentioned as a stress management strategy than napping.

Listening to music (48 percent) is the stress management technique embraced by the highest percentage of adults, followed by exercising (43 percent).

Listening to music (48 percent) is done more frequently than exercising (43 percent) when it comes to stress management.

Adults exercise more than read to manage stress (43 percent vs. 39 percent).

Adults prefer listening to music (48 percent) over watching TV (40 percent) as a way to manage stress.

… when comparing subgroups

Women (48 percent) are more likely than men (29 percent) to say they read to manage their stress.

More men (29 percent) than women (14 percent) play video games to manage stress.

 

Women read more frequently than men as a stress management technique.

Men play video games more often than women to manage stress.

Multiple response questions: Key takeaways
  • Multiple response questions measure the prevalence of attitudes, behaviors and perceptions.
  • They provide insight into a range of behaviors or attitudes on a specific topic.
  • They do not necessarily measure the frequency of behaviors.
  • They do not necessarily measure the strength of an attitude or perception.