Are Professional Responders the Real Enemy?

TNS Philippines

Image via Wikipedia

Heavy responders are people who answer lots and lots of surveys but there is no official definition of ‘lots’ of surveys. Some people think 1 survey per week is too much, while others feel that 1 or 2 or 3 per day is too much.
A couple of years ago, heavy responders were all the rage. I was one of many people working on projects determined to discover whether there was an issue with heavy responders, and if so, how severe the issue was. Here is the report that came out of that work.
TNS, Ipsos And The NPD Group Conduct Study To Address Market Research Industry Concern
The end conclusion was that well-run panels don’t have sufficient numbers of heavy responders for those responders to have any meaningful effect on the results. Well run panels have rules in place to monitor survey frequencies, incentive policies that don’t encourage heavy responding, and recruitment polices that focus on selecting quality responders. Of course, panels with less strict rules could very well have serious issues with heavy responders.
But here’s my point. Do we really care about heavy responders? What if someone does answer a survey every single day. Today it’s chocolate, tomorrow it’s electronics, the next day it’s canned vegetables, then financial products, then automotive products and so on. Let’s even say the cycle continues and they always answer the same seven categories, once per week, every week. Is that such a bad thing?
These people might skew awareness questions simply because they’ve had more opportunities to see up and coming brand names in other surveys. They might even be more aware of the issues surrounding a product, e.g., various flavours or colours or options. But, are they necessarily providing poor quality data? Are they providing false or misleading data? Are the deliberately skewing the data through random responding and straightling?
I would suggest that data quality, and not heavy responding, is the real problem to be concerned about. Ask your survey company about their data quality program. I would.

Related Articles

  • 5 random things I like about statistics and proof you are a dork
  • Using Social Media Research to Predict the Future: That’s me saying yes you can!
  • Probability Sampling – Proof that only telephone samples are quality samples
  • Edward Tufte: Your Presentation Sucks Cause Your Content Sucks
  • Ray Poynter – Overview of Online Research Trends #netgain5
  • Advertisements

    6 responses

    1. OK. Now forgive me for being really quite thick here but under the ‘wood for trees’ approach shouldn’t we question the approach/methodology/sample etc. But then I’m just a simple client/buyer.

      I’m guessing this is an issue in particular types of on-line research e.g. points/reward based providers like YouGov (UK) or Zoomerang (US).

      I think I still agree with your original piece.

      1. There are many many online research suppliers out there ranging from the highest quality to the lowest of low quality. The problem is that clients can’t tell which is which and it’s just not nice to point them out, partly because it’s in bad taste and partly because it’s debatable. You just have to strongly believe in the practices of your provider. So ask them.

    2. Brian LoCicero

      Well, I think the key statement above is “well-run panels” which is why clients and suppliers need to be close to the sample sources being used and understand everything they can about them and their practices.

      Of course cr*p research will still likely play a BIGGER part of cr*p data but it’s always easier to blame the boogie-man 🙂

      1. Cr*p research? Who does that? No one right? 🙂

    3. I’m sure I’m missing something fundamental but I can’t get too excited about ‘heavy responders’. So I guess I agree with you. Seems to be a touch of ‘wood for trees’ in the issue. Aren’t these guys just noise in the system? Any decent project will identify and filter.

      1. The problem is that some research has shown a tiny percentage of people are making huge contributions to surveys. An exaggeration would be that every single survey is being answered by the same 100 people. That’s certainly not a random sample from a population and it certainly won’t lead to generalizable results.

    %d bloggers like this: