Oh no, this one’s a doozy for me!
.
There are a number of high quality marketing research companies out there. Here are just five of them:
.
Annie’s Quality Company Awards
Harris Interactive
Ipsos (My personal bias as I spent almost two years creating their iPi4 data quality system)
NPD Group
Synovate
TNS
.
These companies care strongly about things like data quality, survey design, recruitment methods, incentives, engagement, ethics, and all those other things that make a great company great. Every recommendation they give to clients and potential clients is with these things in mind. They will often try to change details about a client’s research project or survey because the topic is overly sensitive (e.g., Have you felt like killing yourself in the past 3 days) or the incentive is too high (you’re going to attract people who will lie just to get the money) or the survey is too long (people will get bored and not pay attention to their answers).
.
You will notice that I said TRY to change details of a client’s survey. What often happens, however, is a few questions were revised or the incentive was slightly decreased or a few questions were removed from the grid. But, generally, the original concern about the research is still a concern, it’s just slightly less so. Why, you ask, are these reputable companies doing research that they don’t whole heartedly agree with?
.
Well first, let’s acknowledge that sometimes, it’s simply not possible to make the change. Perhaps a survey has been done this specific way for years. If it was changed now, all the norms would completely change and it would be impossible to know whether any changes were the result of real shifts or the new survey. I’ll discredit this option as there are ways to get around it.
.
Other times, clients are unwilling to make the change. Perhaps the client has very carefully developed the survey to meet exactly their needs. They simply can’t remove any questions or they will lose valuable information. This is another option i will discredit. I will argue strongly that surveys developed to gather huge amounts of detail end up attracting a skewed sample of people, including those who are not truly paying attention to the questions.
.
Obviously, the problem does not simply belong to the client. The researcher is the expert. It is their task to explain the issues, the problems, to demonstrate why their suggestions will improve the research. If they do not succeed at this task, then of course surveys will continue to be too long, too boring, and too irrelevant. Researchers need to become better teachers. Teach clients and everyone wins.
.
So what happens when a research company takes a stand and says “We only do quality research. These suggestions must be implemented or we cannot support a survey that will not gather quality data.” Here’s what happens. Another company, one with very different views of how to do research takes the job. They take the 60 minute survey. They take the survey with 100 grid questions. They offer the $20 incentive. Which means the quality company loses business, the clients lose quality data, and research just isn’t the best that it can be.
.
I guess every industry deals with this so I should just shut my mouth. Nope.
.
.
Anyways, I’m very curious to hear your thoughts. How do YOU think we can improve things?

Mar 25 2009

Survey Design Tip #3: Do You Encourage Straightlining? #MRX

This is easy to do as most market research surveys are already designed to accomplish it. If you’ve taken a survey recently, you’ve probably seen it. Here’s a great example:
.
Let me guess, you said ‘Extremely Important’ to every single item. And, you probably answered completely honestly. Know what? You just straightlined. Straightlining is a very bad thing in the world of survey design. Why? Because it’s hard to tell whether someone truly answered the questions carefully or they were simply clicking as fast as they possibly could without reading anything. Gimme that incentive baby!
.
But how you do get around this? The most important thing about designing high quality survey questions is ensuring the use of both positively and negatively keyed items. In other words, half of your questions need to be phrased in a way that make the product sound good and the other half bad. Here’s how it works…
.
Instead of saying ‘Is all natural,’ it could have said ‘Has artificial ingredients.’
Or, instead of ‘Is a good source of nutrition,’ it could have said ‘Is a poor source of nutrition.’
.
If it’s this easy to, why don’t we do it? Why do I constantly get pre-written surveys that are chock full of questions that encourage straightlining? Maybe they’ve done it like this for a long time and they don’t want their norms tampered with. Maybe they feel like they’re encouraging people to think negatively about the product. Or, by offering negative options, maybe they’re encouraging people to rate their products negatively even if they wouldn’t have otherwise. But in the end, don’t you want QUALITY data? Trustworthy data? Actionable data?
.
So follow the rules. If there is no option but to write a grid question. At least write a good grid question.