Manufacturing False Precision in Surveys #MRX

How do you create a survey question measuring frequency of behaviour that will generate the most accurate responses? Experience tells us to consider things like:

  • Should I include a zero or incorporate that into the smallest value?
  • Should I use use whole numbers  like ‘2 to 4’ or partial numbers like ‘2 to less than 4’?
  • Should I use 4 break points or go all out with 10 break points?

These are all smart considerations and will help you collect more precise data. But seriously? How accurate, how precise, how valid are these data anyways? Do you really think that survey responders are going to carefully and precisely calculate the exact number of days or minutes they do something?

When we ask responders to choose between these two options…

  • 2 to 4.99
  • 2 to less than 5

… do you really think that one or the other option will help responders think of estimates that are any more accurate? Let’s face it. They won’t. Yes, there’s statistical precision in the answer options we’ve provided, but we are manufacturing a level of accuracy that does not exist. It’s no different than reporting 10 decimals places where 1 is more than sufficient.

So what do I recommend? Make things simple for your responder. Use real language not hoity toity, decades of academic research language. Use language that makes responders want to come and take another survey.

%d bloggers like this: