skip to main content

Guidelines for Reading Questions and Interpreting Data

There are multiple ways to ask questions and collect data when conducting survey research. It is important to think clearly about the goal and purpose of each question so that the best format can be selected. Once data are collected, reporting and interpreting the data accurately is equally important. Maintaining an understanding of the question structure will allow for correct interpretation.

We have provided a few examples here to help readers better understand the purpose of different question structures and the importance of using accurate language when interpreting the data. This does not cover every kind of question that can be asked in survey research; rather, we have included a few question structures that we have found are commonly misreported.

The data contained in this section is for illustrative purposes only and does not relate to this year's report.

expand all
  For each one, please indicate how significant a source of stress it is in your life:
 
Very Significant
Somewhat Significant
Not Very Significant
Not Significant
Money
37%
29%
20%
13%

Work

31%
35%
22%
12%
Relatonships (e.g. spouse, kids, girl/boyfriend)
21%
26%
26%
27%
Health problems affecting my family
20%
30%
28%
22%
Family responsibilities
21%
32%
25%
22%
Housing costs (e.g. mortgage or rent)
21%
27%
23%
30%
The economy
20%
30%
31%
19%
Job stability
18%
20%
19%
44%
Personal health concerns
20%
31%
29%
20%
Personal safety
11%
19%
34%
36%
Discrimination
9%
12%
22%
57%
What is a grid type question?
  • Allows respondents to evaluate multiple attributes or characteristics using the same scale (e.g., attitudes, perceptions, behaviors, etc.).
  • Scales can vary and measure a wide variety of attitudes or behaviors (e.g., importance, agreement, likelihood, favorability, etc.).
What does a grid type question measure?
  • Measures the level of endorsement for a particular attitude, perception or behavior (each of the attributes evaluated).
  • Attributes are evaluated independently.
  • Provides insight into the strength and depth of feeling for each attribute.
What does a grid type question not measure?
  • Does not measure absolutes.
  • Does not ask respondents to “rank” attributes.
  • Reporting should reflect that respondents did not “rank” items against each other.
Reporting: 

When reporting on data collected in a grid type question, it is important to remember what these questions do and do not measure.

 Type of Reporting   Correct Incorrect Rationale
… when reporting on individual attributes evaluated in the grid
Sixty-seven percent of adults say money is a somewhat/very significant source of stress. Sixty-seven percent of adults say money is a significant source of stress, with 37% saying it is very significant and 29% saying it is a somewhat significant source of stress.
Money is the most significant source of stress (67%).
The question measures the degree of significance placed on each item. In each of these examples, the incorrect statement is misleading because the finding is reported as if the question asked for a “ranking” rather than a “rating.” With the grid type question format, respondents are not asked to make a direct comparison between the attributes that may have resulted in a different finding. As such, while we can discuss which attributes the sample is most likely to rate “significant,” we cannot state that one is more significant than another.
… when comparing multiple attributes
Many adults say that money (37%) and work (31%) are a very significant source of stress in their life. Very significant sources of stress are money (37%) and work (31%), followed by relationships, health problems affecting family and family responsibilities equally, at 21%.
The most significant sources of stress are money (37%) and work (31%). Americans rank money (37%) ahead of work (31%) as a source of stress. Money (37%) is a more significant source of stress than work (22%).

… when comparing subgroups on individual attributes

Those that earn <$50K (76%) are more likely than those that earn >=$50K (59%) to say that money is a very/ somewhat significant source of stress in their life. More lower income adults (<$50K; 76%) say money is a very or somewhat significant source of stress than those who earn more (>=$50K; 59%).
Those who earn <$50K say money (76% somewhat/ very significant) is a more significant source of stress than work (64%).
Grid type questions: Key takeaways
  • Grid type questions measure multiple attributes across the same scale.
  • They are “rating” questions, not “ranking” questions.
  • They measure attributes as they relate to each other, but not absolutes.
  • They indicate the level of endorsement (e.g., importance, agreement, etc.) for each attribute.
In general, would you say your health is ...
Excellent
6%
Very good
29%
Good
41%
Fair
21%
Poor
2%
What is a simple scaled question?
  • Simple scaled questions ask about specific attitudes or behaviors.
  • Scales that can vary and measure a wide variety of attitudes or behaviors (e.g., importance, agreement, likelihood, favorability, etc.).
What does a simple scaled question measure?
  • Measures the level of endorsement for a specific attitude, perception or behavior.
What does a simple scaled question not measure?
  • Does not measure how the item being measured relates to other attitudes or behaviors.
Reporting: 

When reporting on simple scaled questions it is important to remember that these questions answer only the specific question asked. Errors in reporting are less common than with grid type questions described previously.

Type of Reporting   Correct Incorrect Rationale
… when reporting on individual attributes evaluated in the grid

Most adults report their overall health as good (41%) or very good (29%).

Very few (6%) would say their overall health is excellent.

Thirty-eight percent of adults are in excellent or very good health.

Most adults are in good health (41%) and very few (2%) are unhealthy.

The incorrect findings are not specific enough. The question specifically asked respondents to evaluate their own health; it does not represent objective measures of health or the opinion of a qualified health care professional, which may differ from the self-report.
… when comparing subgroups on individual attributes
Both men (38%) and women (33%) are likely to think their health is excellent or very good.
Men and women are equally healthy (38% and 33%, respectively)
Simple scaled questions: Key takeaways
  • Measure specific attitudes or behaviors.
  • Findings indicate the level of endorsement (e.g., importance, agreement, etc.) for specific attitudes or behaviors.
Do you do any of the following to help manage stress? Please select all that apply.

Top 10 most common responses:
Listen to music
46%
Exercise or walk
43%
Surf the Internet/go online
40%
Watch TV/movies for more than two hours per day
39%
Spend time with friends or family
35%
Read
35%
Pray
30%
Nap
29%
Eat
27%
Spend time doing a hobby
23%
What is a multiple response question?
  • Used to understand a range of attitudes, behaviors or perceptions.
  • Provides insight into the prevalence of different attitudes, behaviors or perceptions.
What does a multiple response question measure?
  • Prevalence of attitudes, behaviors or perceptions.
What does a multiple response question not measure?
  • Does not necessarily measure the frequency of a specific attitude, behavior or perception.
  • Does not necessarily measure the strength of the attitude or perception measured.
  • Does not specifically capture preference (i.e., "favorites") among attitudes, behaviors or perceptions or rank order.
Reporting:

When reporting on data collected from a multiple response question, it is important to remember that they measure prevalence. They do not necessarily measure frequency, strength of endorsement or preference. Rather, these data are useful to understand the range of behaviors or attitudes on a given topic.

Type of Reporting   Correct Incorrect Rationale
… when reporting at the aggregate level

The most common ways people manage stress are listening to music, exercising or walking, and surfing the Internet/going online.

Almost one in two adults listen to music as a way to manage stress (46%).

Listening to music, exercising or walking, and surfing the Internet/going online are the most popular ways to manage stress.

Listening to music is the most frequent stress management technique.

In reporting, use of the word “frequently” or “frequency” implies how often a behavior is done.

This question, as phrased, measures prevalence (i.e., how many people are doing these activities) rather than actual frequency (i.e., how many times per week or month they are doing each of these).

When reporting on subgroups we know that more people from a particular subsample (i.e., women) engage in a behavior as compared to men. The question does not address whether or not those women engaging in the behavior actually do the behavior more often than men.

It is important to consider the whole question as it was asked. As such, results from this question cannot, for example, be used to measure the likelihood of listening to music overall — only the likelihood of listening to music for the specific purpose of managing stress.

… when comparing multiple attributes

Surfing the Internet/going online (40%) and watching television or movies for more than two hours per day (39%) are equally likely to be used as stress management techniques.

Listening to music is more commonly mentioned as a stress management strategy than napping.

Listening to music (46%) is the stress management technique embraced by the highest percentage of adults, followed by exercising (43%).

Listening to music (46%) is done more frequently than exercising (43%) when it comes to stress management.

Adults exercise more than read to manage stress (43% versus 35%).

Adults prefer listening to music (46%) over watching TV (39%) as a way to manage stress.

… when comparing subgroups on individual attributes

Women (41%) are more likely than men (28%) to say they read to manage their stress.

More men (11%) than women (3%) play video games to manage stress.

Women read more frequently than men as a stress management technique.

Men play video games more often than women to manage stress.

Multiple response questions: Key takeaways
  • Measures prevalence of attitudes, behaviors, perceptions.
  • Provides insight into a range of behaviors or attitudes on the specific topic in question.
  • Do not necessarily measure frequency of behaviors.
  • Do not necessarily measure strength of an attitude or perception.
Last updated: 2020Date created: 2017