1 Topic 5 Blogs: DIY surveys suck or save the day #MRX

[tweetmeme source=”lovestats” only_single=false]

G’day all and welcome to this months issue of 1 topic, 5 blogs. Todays topic is DIY surveys. Links to my fellow bloggers Bernie Malinoff, Joel Rubinson, Josh Mendelsohn and Brandon Bertelsen can be found below.


I will admit. From a client point of view, DIY surveys have a ton of advantages. You can get the survey out quickly without any middle-man holding you up. Without a middle-man, the cost can be greatly reduced. And, you get exactly what you want without pleading and whining and complaining. If you have some solid experience or training in survey design, I am quite in favour of DIY for short 5 minute surveys.

But, when it comes to substantial surveys, I have many, many issues. The issues revolves mostly around “getting exactly what you want.” Any time I review survey drafts from survey newbies, and EVEN from people experienced in survey design, I find many of these terrible problems.

1) Leading questions: The writer usually has no intention of doing this, but they almost always give away the answer they want to receive. And whether consciously or unconsciously, the responder recognizes that and happily provides it.

2) Missing options: The writer is so focused on the options they are interested in that they forget there are options that are far more popular. They make sure that Brand A and Brand B are represented, but completely forget about Brand C, Store brand, Never buy, and Don’t know. This is another way to get the exact answer you want purely by bad survey design.

3) I can usually find a bunch of category jargon including words I don’t even recognize. On numerous occasions, I have seen surveys ask something like “Do you plan to purchase the X6000?” I am left trying to remember if the previous question was about televisions or motorcycles in hopes that X6000 will suddenly mean something.

4) I can usually find survey jargon.  People don’t purchase, they buy. People don’t use gum, they chew gum. People don’t purchase the 75g bar soap package, they buy the two pack.

Even if you much prefer DIY, I 100% believe that everyone needs a middle man. Whether that middle-man is a reputable survey company or another person in a completely different department, you absolutely must have fresh eyes. Fresh eyes find the annoying mistakes that you can no longer see because you’re tired of reading your survey. Fresh eyes find logic errors, spelling mistakes, and unclear questions. Market researchers will criticize far more harshly resulting in a far better survey, but fresh eyes of any sort are always in your favour.

In the end, it doesn’t matter how much money you saved by going the DIY route if your survey results are useless.

Links to the other 4 blogs coming shortly:

Bernie Malinoff of element 54
Joel Rubinson of the ARF
Josh Mendelsohn of Chadwick Martin Bailey
Brandon Bertelsen

4 responses

  1. Hey Annie… great post as usual. The one thing I’d add about DIY is that we really have no idea how good or how poor it is. Personally, I’ve seen very few examples of DIY surveys in action and can’t judge the quality of the writing overall. I do know that there are some pretty crappy surveys written by the so-called experts. So is the crap percentage greater or lesser with DIY? All we really know is that there are horrible examples on both sides of the aisle.

  2. vancemarriner

    This is a great post. I especially agree with the point you made about the need for a middle man. If you’re interested, here’s a link to a blog post I wrote a while back that dealt with many of the same issues:


  3. Neil Wilkinson

    Don’t remember where I read this but it rings true:

    SURVEY GENERAL’S WARNING: DIY Surveys Done Poorly Cause Erroneous Conclusions, Bad Decisions, And May Cost You Customers.

  4. […] you’re designing a survey Annie at LoveStats has sage advice: Even if you much prefer DIY, I 100% believe that everyone needs a middle man. […]

%d bloggers like this: