The Dilemma that is Market Research Data Quality

Oh no, this one’s a doozy for me!
There are a number of high quality marketing research companies out there. Here are just five of them:
🙂 Annie’s Quality Company Awards 🙂
Harris Interactive
Ipsos (My personal bias as I spent almost two years creating their iPi4 data quality system)
NPD Group
These companies care strongly about things like data quality, survey design, recruitment methods, incentives, engagement, ethics, and all those other things that make a great company great. Every recommendation they give to clients and potential clients is with these things in mind. They will often try to change details about a client’s research project or survey because the topic is overly sensitive (e.g., Have you felt like killing yourself in the past 3 days) or the incentive is too high (you’re going to attract people who will lie just to get the money) or the survey is too long (people will get bored and not pay attention to their answers).
You will notice that I said TRY to change details of a client’s survey. What often happens, however, is a few questions were revised or the incentive was slightly decreased or a few questions were removed from the grid. But, generally, the original concern about the research is still a concern, it’s just slightly less so. Why, you ask, are these reputable companies doing research that they don’t whole heartedly agree with?
Well first, let’s acknowledge that sometimes, it’s simply not possible to make the change. Perhaps a survey has been done this specific way for years. If it was changed now, all the norms would completely change and it would be impossible to know whether any changes were the result of real shifts or the new survey. I’ll discredit this option as there are ways to get around it.
Other times, clients are unwilling to make the change. Perhaps the client has very carefully developed the survey to meet exactly their needs. They simply can’t remove any questions or they will lose valuable information. This is another option i will discredit. I will argue strongly that surveys developed to gather huge amounts of detail end up attracting a skewed sample of people, including those who are not truly paying attention to the questions.
Obviously, the problem does not simply belong to the client. The researcher is the expert. It is their task to explain the issues, the problems, to demonstrate why their suggestions will improve the research. If they do not succeed at this task, then of course surveys will continue to be too long, too boring, and too irrelevant. Researchers need to become better teachers. Teach clients and everyone wins.
So what happens when a research company takes a stand and says “We only do quality research. These suggestions must be implemented or we cannot support a survey that will not gather quality data.” Here’s what happens. Another company, one with very different views of how to do research takes the job. They take the 60 minute survey. They take the survey with 100 grid questions. They offer the $20 incentive. Which means the quality company loses business, the clients lose quality data, and research just isn’t the best that it can be.
I guess every industry deals with this so I should just shut my mouth. Nope. 😀
Anyways, I’m very curious to hear your thoughts. How do YOU think we can improve things?

Related Articles

2 responses

  1. I will be anonymous

    I work for a research company in the data analytics space and was once forced to conduct research that we knew was incomplete because the client couldn’t afford (or wouldn’t spend) the amount necessary to do the complete work.

    We ended up do, say 50% of what we recommended the client do and the information turned out to not be actionble because the other half was needed to assess the whole picture.

    In the end, the client ended up getting made at us because we “allowed” them to make a dumb decision.


    1. I know you’re not alone. It’s tough to know what the right thing to do is and have it shot down because of money or lack of knowledge.

%d bloggers like this: