Question Order, Question Length, And Question Context #AAPOR


Moderator: Jessica L. Holzberg, U.S. Census Bureau
Satisfied or Dissatisfied? Does Order Matter?; Jolene D. Smyth, University of Nebraska-Lincoln Richard Hull, University of Nebraska-Lincoln

  • Best practice is to use a balanced question stem and keep response options in order
  • What order should it be in the question stem
  • Doesn’t seem to matter whether the scale is left to right or top to bottom
  • Visual Heurstic Theory – help make sense of questions, “left and top mean first” and “up means good”, people expect the positive answer to come first, maybe it’s harder to answer if good is a the bottom
  • Why should the question stem matter, we rarely look at this
  • “How satisfied or dissatisfied are you?  [I avoid this completely by saying what is your opinion about this and then use those words in the scale, why duplicate words and lengthen questions]
  •  Tested Sat first and Disat second in the stem, and then Sat top and Disat bottom in the answer list, and vice versa
  • What would the non repsonse look like in these four options – zero differences 
  • Order in question stem had practically no impact, zero if you think about random chance
  • Did find that you get more positive answers when positive answer is first
  • [i think we overthink this. If the question and answers are short and simple, people change no trouble and random chance takes its course. Also, as long as all your comparisons are within the test, it won’t affect your conclusions]
  • [She just presented negative results. No one would ever do that in a market research conference🙂 ]

Question Context Effects on Subjective Well-being Measures; Sunghee Lee, University of Michigan Colleen McClain, University of Michigan

  • External effects – weather, uncomfortable chair, noise in the room
  • Internal effects – survey topic, image, instructions, response sale, question order
  • People don’t view questions in isolation, it’s a flow of questions
  • Tested with life satisfaction and self-rated health, how are the two related, does it matter which one you ask first; how will thinking about my health satisfaction affect my rating of life satisfaction
  • People change their behaviors when they are asked to think about mortality issues, how is it different for people whose parents are alive or deceased
  • High correlations in direction as expected
  • When primed, people whose parents are deceased expected a lesser lifespan 
  • Primed respondents said they considered their parents death and age at death
  • Recommend keeping the questions apart to minimize effects [but this is often/rarely possible]
  • Sometimes priming could be a good thing, make people think about the topic before answering

Instructions in Self-administered Survey Questions: Do They Improve Data Quality or Just Make the Questionnaire Longer?

Cleo Redline
, National Center for Education Statistics Andrew Zukerberg, National Center for Education Statistics Chelsea Owens, National Center for Education Statistics Amy Ho, National Center for Education Statistics

  • For instance, if you say “how many shoes do you have not including sneakers”, and what if you have to define loafers
  • Instructions are burdensome and confusing, and they lengthen the questionnaire 
  • Does formatting of instructions matter
  • Put instructions in italics, put them in bullet points because there were several somewhat lengthy instructions
  • Created instructions that conflicted with natural interpretation of questions, eg assessment does not include quits or standardized tests
  • Tried using paragraph or list, before or after, with or without instructions
  • Adding instructions did not change mean responses 
  • Instructions intended to affect the results did actually do so, I.e., people read and interpreted the instructions
  • Instructions before the question are effective as a paragraph
  • Instructions after the question are more effective as lists
  • On average, instructions did not improve data question, problems are real bu they are small
  • Don’t spend a lot of time on it if there aren’t obvious gains
  • Consider not using instructions

Investigating Measurement Error through Survey Question Placement; Ashley R. Wilson, RTI International Jennifer Wine, RTI International Natasha Janson, RTI International John Conzelmann, RTI International Emilia Peytcheva, RTI International

  • Generally pool results from self administered and CATI results, but what about sensitive items, social desirability, open end questions, what is “truth”
  • Can evaluate error with fictitious issues – e.g., a policy that doesn’t exist [but keep in mind policy names sound the same and could be legitimately misconstrued ]
  • Test using reverse coded items, straight lining, check consistency of seeming contradictory items [of course, there are many cases where what SEEMS to contradict is actually correct, e.g., Yes, I have a dog, No I don’t buy dog food; this is one of the weakest data quality checks]
  • Can also check against administrative data
  • “AssistNow” loan program did not exist [I can see people saying they agree becuase they think any loan program is a good thing]
  • On the phone, there were more substantive answers on the phone, more people agreed with the fictitious program [but it’s  a very problematic questions to begin with]
  • Checked how much money they borrowed, $1000 average measurement error [that seems pretty small to me, borrow $9000 vs $10000 is a non-issue, even less important at $49000 and $50000]
  • Mode effects aren’t that big

Do Faster Respondents Give Better Answers? Analyzing Response Time in Various Question Scales; Daniel Goldstein, NYC Department of Housing Preservation and Development; Kristie Lucking, NYC Department of Housing Preservation and Development; Jack Jerome, NYC Department of Housing Preservation and Development; Madeleine Parker, NYC Department of Housing Preservation and Development; Anne Martin, National Center for Children and Families

  • 300 questions, complicated sections, administered by two interviewers, housing, finances, debt, health, safety, demographics; Variety of scales throughout
  • 96000 response times measured, left skewed with a really long tail
  • Less education take longer to answer questions, people who are employed take longer to answer, older people take longer to answer, and none glish speakers take the longest to answer
  • People answer more quickly as they go through the survey, become more familiar with how the survey works
  • Yes no are the fastest, check all that apply are next fast as they are viewed as yes no questions
  • Experienced interviewers are faster
  • Scales with more answer categories take longer

%d bloggers like this: