Down with response rates!

Clients and researchers love response rates. If you forget to include them, you’re sure to get an anxious phone call very soon. And if you do include them, you’re sure to hear either “why is your rate so horribly low” or “look how great the rate is!”
Sadly, in much of the market research space, response rates are non-sensical. Tell me the rate you want and I will create it for you. If you want a response rate of 60%, I will choose only panelists who have completed another survey in the last couple of weeks. Easy peasy.
Panels are the problem. Well, they’re not a problem. Smart researchers know how to use them properly. But if you’re just talking response rates, panels are a problem.
Panels rely on recruitment to begin with. They rely on people responding in order to join the panel. They rely on people to continue to respond to remain on the panel. Panelists have jumped through so many response rate hoops before they arrive at your survey that your response rate means nothing. It means that they responded to an ad, that they responded to a profiling survey, that they responded to a screener survey, that they responded to other surveys, and that they responded to your survey too.
If you really want to know the response rate, first ask how many people were invited to join and profile and screen and then complete your survey.
Now that’s a response rate. But you won’t like it.

Written on the go

One response

  1. Robbie Y. Hopper

    Still, there are always people who insist that, prior to going online, they had a better response rate. But were all of those evaluations legitimate? In detailing its own process of moving to online evaluations, University of British Columbia made an interesting discovery that most institutions never consider. In an online survey, only students who are validly enrolled in a course can evaluate that course, and they are limited to evaluating one time. Several of the courses in UBC’s first few “test” terms, had response rates higher than 100% for their paper distribution. For comparison purposes in their paper, UBC simply reduced those figures to 100%, but it’s interesting to be reminded that response rates may be artificially high for paper evaluations (University of British Columbia, 2010).

%d bloggers like this: