The impact of questionnaire design on measurements in surveys #3 #ESRA15 #MRX 


Live blogged from #ESRA15 in Reykjavik. Any errors or bad jokes are my own.

We had 90 minutes for lunch today which is far too long. Poor me.  I had pear skyr today to contrast yesterday’s coconut skyr. I can’t decide which one i like better. Oh, the hard decisions I have to make!  I went for a walk which was great since it drizzled all day yesterday. The downtown is tiny compared to my home so it’s quite fun to walk from one end to the other, including dawdling and eating, in less than half an hour. It’s so tiny that you don’t need a map. Just start walking and take any street that catches your fancy. I dare you to get lost. Or feel like you’re in an unsafe neighbourhood. It’s not possible.

I am in complete awe at the bird life here. There are a number of species i’ve never seen before which on its own is fun. It is also baby season so most of the ducks are paired off and escorting 2 to 8 tiny babies. They are utterly adorable as the babies float so well that they can barely swim underwater to eat. I haven’t seen any puffins along the shore line. I’m still hopeful that a random one will accidentally wander across my path.

By the way, exceptional beards really are a thing here. In case you were curious.

  the Who: experimental evidence on the effect of respondent selection on collecting individual asset ownership information

  • how do you choose who to interview?
  • “Most knowledgeble person”, random selection, the couple together, each individual adult by themself about themself, by themself about other people
  • research done in uganda so certainly not generalizable to north america
  • ask about dwelling, land, livestock, banking, bequeathing, selling, renting, collateral, investments
  • used CAPI, interviews matched on gender, average interview was 30 minutes
  • challenges included hard to find couples together as one person might be working in the field, hard to explain what assets were
  • asking couple together shows differences from ownership incidence but the rest is the same
  • [sorry, couldn’t determine what “signficant positive results” actually meant. would like to know.😦 ]

Portuguese national health examination survey: questionnaire development

  • study includes physical measurements and a survey of health status, health behaviours, medication, income, expenses
  • pre-tested the survey for comprehension and complexity
  • found they were asking for things from decades ago and people couldn’t remember (eg when did you last smoke)
  • some mutually exclusive questions actually were not
  • you can’t just ask about ‘activity’ you have to ask about ‘physical activity that makes you sweat’
  • responses cards helped so that people didn’t have to say an embarrassing word
  • had to add instructions that “some questions may not apply to you but answer anyways” because people felt that if you saw them walking you shouldn’t ask whether they can walk
  • gave examples of what sitting on the job, or light activity on the job meant so that desk sitters don’t include walking to the bathroom as activity
  • pretest revealed a number of errors that could be corrected, language and recall problems can be overcome with better questions

an integrated household survey for Wales

  • “no change” is not a realistic option [i wish more people felt that way]
  • duplication among the various surveys, inefficient, survey costs are high
  • opportunity to build more flexibility into a new survey
  • annual sample size of 12000, randomly selected 16+ adults, 45 minutes
  • want to examine effects of offering incentives
  • survey is still in field
  • 40% lower cost compared to previous, significant gains in flexibility

undesired repsonse to sureys, wrong answers or poorly worded question? how respondents insist on reporting their situation despite unclear questionning

  • compared census information with family survey information
  • interested in open text answers
  • census has been completed since 1881
  • belle-mere can mean stepmother and mother in law in french
  • can’t tell if grandchildren in the house belong to which adult child in the house
  • ami can mean friend or boyfriend or partner or spouse, some people will also specify childhood friend or unemployed friend or family friend
  • can’t tell if an unknown location of child means they don’t know the address or the child has died
  • do people with an often changing address live in a camper, or travel for work?
  • if you only provide age in years for babies you won’t know if it’s stillborn or actually 1 year old

ask a positive question and get a positive answer: evidence on acquiesence bias from health care centers in nigeria

  • created two pairs of questions were one was positive and one was negative – avoided the word no [but the extremeness of the questions differed, eg., “Price was reasonable” vs “Price was too expensive” ]
  • some got all positive, all negative, or a random mix
  • pilot test was a disaster, in rural nigeria people weren’t familiar with this type of question
  • instead, started out asking a question about football so people could understand how the question worked. asked agree or disagree, then asked moderately or strongly – two stage likert scale
  • lab fees were reasonable generated very different result than lab fees were unreasonable [so what is reality?]
  • it didn’t matter if negatives were mixed in with positives
  • acquiescence bias affects both positive and negative questions, can’t say if it’s truly satisficing, real answer is probably somewhere in between [makes we wonder, can we develop an equation to tease out truth]
  •  large ceiling effects on default positive framing — clinics are satisfactory despite serious deficiencies
  • can’t increase scores with any intervention but you can easily decrease the scores
  • maybe patient satisfaction is the wrong measure
  • recommend using negative framing to avoid ceiling effects [I wonder if in north america, we’re so good at complaining that this isn’t relevant]
%d bloggers like this: