Survey Design Tip #3: Do You Encourage Straightlining? #MRX


badgrid

This is easy to do as most market research surveys are already designed to accomplish it. If you’ve taken a survey recently, you’ve probably seen it. Here’s a great example:
.
Let me guess, you said ‘Extremely Important’ to every single item. And, you probably answered completely honestly. Know what? You just straightlined. Straightlining is a very bad thing in the world of survey design. Why? Because it’s hard to tell whether someone truly answered the questions carefully or they were simply clicking as fast as they possibly could without reading anything. Gimme that incentive baby!
.
But how you do get around this? The most important thing about designing high quality survey questions is ensuring the use of both positively and negatively keyed items. In other words, half of your questions need to be phrased in a way that make the product sound good and the other half bad. Here’s how it works…
.
Instead of saying ‘Is all natural,’ it could have said ‘Has artificial ingredients.’
Or, instead of ‘Is a good source of nutrition,’ it could have said ‘Is a poor source of nutrition.’
.
If it’s this easy to, why don’t we do it? Why do I constantly get pre-written surveys that are chock full of questions that encourage straightlining? Maybe they’ve done it like this for a long time and they don’t want their norms tampered with. Maybe they feel like they’re encouraging people to think negatively about the product. Or, by offering negative options, maybe they’re encouraging people to rate their products negatively even if they wouldn’t have otherwise. But in the end, don’t you want QUALITY data? Trustworthy data? Actionable data?
.
So follow the rules. If there is no option but to write a grid question. At least write a good grid question.

Related Articles

2 responses

  1. I think part of the problem is you have people who have never considered everything that goes into creating a reliable and valid survey instrument. Instead, they do survey design by committee. They go around thee table and say, “Let’s ask this” and “Let’s ask this” etc. And you end up with a survey that is a collection of questions without consideration for design. So many times, I have had people ask me to make a survey for them thinking: Just put some questions together and we can send it out.

    I like reverse coded questions for multi-item measures. Another tactic is adequately separating questions that are asking about the same dimension. Many times I have checked someone’s survey and said: “Wow, these factor loadings and alphas are amazing.” Then I go back and look at the survey to see, “Of course they are perfect, the questions are back to back to back.”

  2. The thing is, if I see a list that long on an Internet questionnaire (as opposed to interviewer-administered), I just shut the questionnaire down. I don’t even take in the questions, I just think ‘that’s really long winded and most of the items are ridiculous’ and I give up.

    You combine that with a question that is genuinelly rather hard to make sense of ‘how important is it that a brand has…’ and I’m off. Need a huge incentive to get through it…