Tag Archives: survey length

Quick tips for writing a quality survey

Designing a quality survey seems simple but for anyone who’s tried, the questions you have increase exponentially with every attempt. Here are just a few quick tips.

1) Start with a precise, well thought out purpose. If a survey question does not specifically answer a purpose, cut it.
2) Write questions that can be measured quantitatively. It will save you time, money, and peace of mind if you those numbers map directly to specific company goals.
3) Keep your survey short. This will lead to higher response rates, less self-opting out, and greater generalizability. When I say short, I mean 15 minutes. Absolutely no more.
4) Keep your questions short. This will lead to higher reading comprehension, greater accuracy, and greater data quality. Ditto for short answers.
5) Use real words. Forget consumption and purchase intent. Talk about eating and buying. We’re not all marketers and we don’t all get those fancy words.
6) Use negatives cautiously and sparingly. The human brain has a unique fondness for NOT seeing this word. Avoid tempting fate. If you must use a negative, try to use a capitalized NOT.
7) Get yourself a survey question design book and learn the art. You might as well do it right and get the right data. You will be amazed at all the other strange things the brain does when it digests a survey.

Good luck!

WRITE‘ by karindalziel via Flickr
Image is licenced under a Creative Commons Attribution licence

Related Posts

How to Shorten a Survey

Actress demonstrating initial reactions of fea...

Image via Wikipedia

I know, I know, you NEED all those questions. You can’t gather the learnings you require without every last one of those questions. Well, here are some tips that I hope will help you shorten your surveys and increase your data quality.
1) Remove or reduce all the questions that are there because they are interesting but never actually do anything about. You know what these are. The 40 item grids that people straightline through anyways. The questions that have so much jargon and marketing speak that you barely understand them. The questions that go into such minute detail that no sane person would be able to answer them. (No peeking, what is the exact fabric content of the shirt you are making right now.)
2) Remove all but two concepts. Seriously, do you think that by the time responders are on the third concept that they’re actually reading it? I doubt it. It’s pretty boring to read through the same series of questions over and over. I probably wouldn’t.
3) Chop the survey in half and deliver each half to a different set of people. Chances are, the survey is written as modules anyways. You can probably separate it out into two or three separate surveys and still achieve your overall goals without the loss of analysis. A good researcher will be able to do this.
4) Take advantage of the demographics gathered through profiling surveys. Most survey panels gather hoards of data from panelists when they join. This information is then updated on a regular basis, including automatic aging of people. If you can save ten minutes of survey time by piping in variables, just imagine what you can put in there instead. Even better, don’t replace those ten minutes.
5) Forget the 7 or 10 point scale and stick to a 5 point scale. Sure, you’ve given responders more options and yourself more data points. BUT, remember that you are going to average those scores across hundreds of people anyways. A 8.4 out of 10 is the same as 4.2 out of 5. You really really do have exactly the same number of gradients in both cases. Save some time, avoid annoyance, and just use 5 point scales.
6) Using branching and hiding skip patterns does NOT shorten a survey. It simply hides length. That is not a solution.
Hm… Now that I think about it, the topic of this chatter is not survey length but data quality. Fancy that.

Related Articles


  • You are too old to understand social media so don’t even try
  • #MRA_FOC #MRX Effective Data Visualization by Naomi Robbins, Part #1
  • CAUTION: The Scariest Thing EVER
  • 1topic5blogs: The only thing cell phone surveys are good fer
  • I don’t give a rat’s ass about probability sampling
  • Building a bad reputation before we even start: Privacy in social media research
  • My Theory on Declining Response Rates


    It’s quite simple.
    MANY research companies — You can’t answer everyone’s surveys
    Increasing survey lengths — Who has a spare hour?
    Boring surveys — grid after grid after grid
    Old fashioned surveys — Hello… it’s 2009. Have you seen the wicked cool options that are available?
    Competition — Facebook or survey, Twitter or survey, YouTube or survey, FAMILY or survey
    Sensitive surveys — Why is it now ok to ask about the most intimate details of a person’s life
    Marketing speak — Com’on now, my gum chewing needs? get serious.
    Bad + bad + bad + bad = more and more and more annoyances = more and more and more people just giving up
    What do we do?
    We spend more money on recruitment. We spend more money quicker. We spend money gaining the trust of people who will get frustrated and leave. We DON’T spend money maintaining the trust of people who have decided they do want to trust us. I think there is a cardinal rule of business that it is cheaper to retain good customers than to recruit new ones.
    The solution, though simple, is long term. Let’s improve surveys. Shorten them, improve the quality, make them real. The most incredible thing is that we already know how to do this. Over time, people will begin to see the change. They will start to appreciate marketing research surveys again. A new generation of responders will see what we are offering and choose to be a part of it. It always feels good to know that you’ve made an important contribution. It feels even better when that contribution was fun to make.
    Greece seems to have turtles at every archeological site, we called them guard turtles. Let’s hire guard turtles at every MR agency and put them in charge of guarding against surveys that don’t promote responder engagement. That would be fun!

    Related Articles


  • Will Goodhand: Social Media Research and Digividuals #netgain5 #mrx
  • There is no quality insight. There is only insight.
  • Twitterversity: It’s University in Pajamas! #MRX
  • Quick tips for writing a quality survey
  • Brian Singh: Insights from the Nenshi Campaign #netgain5 #mrx #li
  • Screener keeners or rejection correction?
  • The Dilemma that is Market Research Data Quality

    Oh no, this one’s a doozy for me!
    There are a number of high quality marketing research companies out there. Here are just five of them:
    🙂 Annie’s Quality Company Awards 🙂
    Harris Interactive
    Ipsos (My personal bias as I spent almost two years creating their iPi4 data quality system)
    NPD Group
    These companies care strongly about things like data quality, survey design, recruitment methods, incentives, engagement, ethics, and all those other things that make a great company great. Every recommendation they give to clients and potential clients is with these things in mind. They will often try to change details about a client’s research project or survey because the topic is overly sensitive (e.g., Have you felt like killing yourself in the past 3 days) or the incentive is too high (you’re going to attract people who will lie just to get the money) or the survey is too long (people will get bored and not pay attention to their answers).
    You will notice that I said TRY to change details of a client’s survey. What often happens, however, is a few questions were revised or the incentive was slightly decreased or a few questions were removed from the grid. But, generally, the original concern about the research is still a concern, it’s just slightly less so. Why, you ask, are these reputable companies doing research that they don’t whole heartedly agree with?
    Well first, let’s acknowledge that sometimes, it’s simply not possible to make the change. Perhaps a survey has been done this specific way for years. If it was changed now, all the norms would completely change and it would be impossible to know whether any changes were the result of real shifts or the new survey. I’ll discredit this option as there are ways to get around it.
    Other times, clients are unwilling to make the change. Perhaps the client has very carefully developed the survey to meet exactly their needs. They simply can’t remove any questions or they will lose valuable information. This is another option i will discredit. I will argue strongly that surveys developed to gather huge amounts of detail end up attracting a skewed sample of people, including those who are not truly paying attention to the questions.
    Obviously, the problem does not simply belong to the client. The researcher is the expert. It is their task to explain the issues, the problems, to demonstrate why their suggestions will improve the research. If they do not succeed at this task, then of course surveys will continue to be too long, too boring, and too irrelevant. Researchers need to become better teachers. Teach clients and everyone wins.
    So what happens when a research company takes a stand and says “We only do quality research. These suggestions must be implemented or we cannot support a survey that will not gather quality data.” Here’s what happens. Another company, one with very different views of how to do research takes the job. They take the 60 minute survey. They take the survey with 100 grid questions. They offer the $20 incentive. Which means the quality company loses business, the clients lose quality data, and research just isn’t the best that it can be.
    I guess every industry deals with this so I should just shut my mouth. Nope. 😀
    Anyways, I’m very curious to hear your thoughts. How do YOU think we can improve things?

    Related Articles

    Survey Design Tip #2: Short and Sweet by the Unintelligencer

    Burrowing Owl, a diurnal owl of open country.

    Image via Wikipedia

    [tweetmeme source=”lovestats” only_single=false]The Unintelligencer

    So how was yore dai today? Done yall grab an coffee, sit at yor desk, en than nots move until lunch? ore, Do joo spend 20 minuteses duin quick check o’ your email, followed by an 10 minute discussion of new product, followed by an 2 are meetin durrin which tiem ewe checked your email & twittered? Did yoo wach TV last night? Did ytou patiently sit 4 da entire 60 minuteses watchin aw de commercials or did yuo git up at erry commercial for a drink, a snack, a peck onna cheek, an email, or a…. um…. pee break? My guess iz you chose the second opsion ins boff caseses.
    So Y do wii ekspect survey responders too b able 2 sit thru a 45 minute survey? Why do wee ekspect thems tew do it 1ce a week, eveyr week? wut could possibly cause them too be interested n a survey for that lawng when you don’t even do it in your regular life?
    Awl we’re reely doin wif theeses lawng surveys is givin our precious survey participants reason to reconsider sharin their opinions, reason to let their attension wander, reason to move oan to somethin else. Sure, long survey gievs you lots of detailed data, in depff informasion, and plenty of opportunity to run fancy schmancy multivariate statistics. Butt, wiff response rateses followin such a scary declyn, it is hi time the survey research industry reconsiders what a long survey is. On that note, tell mee what YOU think is too long for a survey.

    Related Articles


  • 6 reasons why you should start a blog now
  • Pie-Packing by Mario Klingemann: More fascinating pie chart art
  • Bursting my market research bubble
  • MRA Boston Day 1 – Tweeple save the day! Turning down business is good for business
  • 5 random things about me
  • A Do Not Call list for the internet. Primed for failure? #MRX
  • Survey Design Tip #2: Short and Sweet

    Burrowing Owl, a diurnal owl of open country.

    Image via Wikipedia

    So how was your day today? Did you grab a coffee, sit at your desk, and then not move until lunch? Or, did you spend 20 minutes doing a quick check of your email, followed by a 10 minute discussion of a new product, followed by a 2 hour meeting during which time you checked your email and twittered? Did you watch TV last night? Did you patiently sit for the entire 60 minutes watching all the commercials or did you get up at every commercial for a drink, a snack, a peck on the cheek, an email, or a…. um…. pee break? My guess is you chose the second option in both cases.
    So why do we expect survey responders to be able to sit through a 45 minute survey? Why do we expect them to do it once a week, every week? What could possibly cause them to be interested in a survey for that long when you don’t even do it in your regular life?
    All we’re really doing with these long surveys is giving our precious survey participants reason to reconsider sharing their opinions, reason to let their attention wander, reason to move on to something else. Sure, long survey give you lots of detailed data, in depth information, and plenty of opportunity to run fancy schmancy multivariate statistics. But, with response rates following such a scary decline, it is high time the survey research industry reconsiders what a long survey is. On that note, tell me what YOU think is too long for a survey.

    Related Articles


  • 6 groups of people who need to follow research standards
  • Down with quant! Long live qual!
  • Keep the ESOMAR faith
  • #MRA_FOC #MRX MRA/IMRO Social Media Research Guidelines
  • Survey Design Tip #1: Responders DON’T CARE

    You live, breathe, and eat your brand. You love nothing more than to discuss the most intricate and minute details of how amazing your brand is, how revolutionary it is, how much better it is than any other competitive brand out there. And, you can talk about it for hours on end and always have something else to talk about. I don’t blame you. I’ve tried your brand. It is totally amazing.

    But, do you SERIOUSLY think that survey participants care even 1% as much as you do? They’re just grabbing the pads of paper on the top shelf at Office Depot because they ran out of paper. Do you SERIOUSLY think they care so much about paper that they can remain engaged in a 30 minute survey, discussing how your brand of paper meets all of their paper and paper accessory needs? How it fills their desire to know at the depths of their heart that this paper is statistically significantly better than some other paper?

    Perhaps I’m stretching things a bit, but I suspect you see my point. My favourite real example is a survey about gum. Grid after grid after grid. Does it “meet your daily chewing needs?” Does it “satisfy your nutritional requirements?” For all I care, they might as well ask me if it “meets my needs for world peace.” All I want is something that tastes good to chew on while I wait for the bus.

    Think about that next time you’re designing the most grammatically correct, professionally phrased, and comprehensively detailed (long) survey. Why aren’t you seeing good data quality with all those efforts?

    Related Articles

    %d bloggers like this: