The Last Blog Post: Speak Loudly My #MRX Friends #LastBlog

The Last Blog Post is an idea set up by Daniel Burstein. His request: “If you had one last blog post to send out to the marketing world, what would you say?” There are already many posts available to read and you can follow them with the Twitter hashtag #LastBlog. Here is my post, albeit sent out to the marketing research world. ***

You have been blessed with the ability to understand research. It is a gift that gives you the ability to make the world a better place in your own unique way. Where some people see reams of numbers swirling in a haze of fear, you see t-test results and chi-square tables and regression equations that make perfect sense. You know how to interpret confidence intervals and sampling error and outliers to come to accurate and meaningful conclusions. You know how to design questionnaires that produce valid numbers and how to select samples that give generalizable results.

Use your gift for the common good. Speak loudly when statistics are being interpreted too strictly, too loosely, or just plain incorrectly. Speak loudly when surveys are too long, too boring, or poorly designed. Speak loudly when samples are selected with little care. Speak loudly when charts and illustrations are being used to entertain instead of educate. Speak loudly when you see our market research industry being wrongly trod upon.

Speak loudly my dear #MRX friends.

Read these too

  • Survey Design Tip #3: Do You Encourage Straightlining?
  • Laugh at yourself and then cry at our flailing industry
  • In Honor of Infographics. #MRX
  • This is why Twitter will die
  • Only crazy people are on twitter
  • Screener keeners or rejection correction?

    How much is too much? How much is too little? There are lots of things in market research that require a healthy balance between doing the right thing and conducting business. Deciding how many screeners to offer to potential survey responders is one of them.

    Most survey panels recognize that screening people out of surveys, no matter why, is bad for two reasons. First, and completely justified, it ticks off panelists who feel their time has been wasted and their opinions ignored. Second, it’s a waste for panels that just used up one survey in their data quality rule of “one survey invite per week” and they didn’t even get a complete in return.

    For both of these reasons, many panels strive to handle the problem by offering up a number of surveys in a row to panelists. Panelists receive an invite and then proceed through one or more consecutive screeners until they qualify for a survey. (Let’s not consider what this means for probability sampling.)

    But what is the right number of screeners? Is it ok to send someone through ten minutes of screeners? Is it ok to give them two or three screeners?

    Photo credit: xenia from morguefile.com

    I just spoke with someone who said their company takes people through up to five screeners before they say enough is enough. Panelists are even compensated for each screener they complete. I worry that even though they are being compensated, it is annoying to panelists. Screeners are obviously not surveys. Panelists can tell that they’ve been rejected once, twice, three, four, and five times. Imagine being rejected by five screeners every time you try to participate. It’s just one more source of rejection, something none of us need now or ever.

    In fact, I even wonder if there is a rejection effect for which I have a two tailed hypothesis. Does increased rejection cause decreased survey scores due to the annoyance or does increased rejection cause increased survey scores due to the satisfaction of finally getting a survey to answer. I’d love an answer to that!

    So what does your experience tell you? Are responders keeners for screeners?

    Read these too

  • #MRA_FOC #MRX Karaoke by Jim Longo. I dare you.
  • If Frozen Meat Isn’t a Bribe, What About Raw Meat?
  • It’s Here! Twitterversity is Here! #MRXU
  • Proof that sampling social media data is critical, but you already knew that
  • If you DARE – the chart test of death
  • Esomar Chicago Day 2: Four days later!

    Chicago Harbor Lighthouse

    Image via Wikipedia

    Day 2 brought lots of interesting topics as well. We started off with updates on the ESOMAR 26 document, the ARF quality initiative, and the ISO process. I did not know that the first ESOMAR code was developed in 1948! I did not know that people from 26 countries were involved in the ISO process. It was not a surprise, however, to hear the findings of the ARF quality study .

    • data quality is not the serious issue that was previously thought
    • panel duplication is really only about 16%
    • heavy responders provide good data
    • response rates are not the best indicators of quality data.

    After the ARF presentation indicating a complete lack of ability to identify a weighting system that would equalize panels, one unlucky speaker had the unfortunate time slot of presenting his work on how the CART system can indeed weight data sources to be equal. Personally, I’m still on the side that panels cannot be weighted to be equal. I’ve participated in so many parallel studies for so many different types of surveys and categories of products  and have yet to seen a perfectly successful case. Sure, you can always weight a few variables into comparability, but you lose out on a bunch more. Maybe CART is the magic solution. Time will tell.

    Once again, survey panels took the hit on not being representative of the general population. Really, come on folks, what method of marketing research IS representative of the general population. None. End of story. In the end, the only kind of representativity that matters is making sure your sample suits the purpose. And, if your results never predict the marketplace, get out of the business.

    Kim Dedeker was kind enough to share a few thoughts on the state of the industry since she first caused a good storm. Her advice – the industry can only be sustained through quality, we need to continue having and creating leadership through getting involved.

    The last major topic of the day was mobile research. I must confess that I am STILL not a believer. I just don’t see how the ability to answer a 20 minute survey on a 2 inch screen as you ride the bus or pay for your new purchase is going to bring the survey industry back from the deadly response rate dive.

    Feel free to also read a few other thoughts I had about ESOMAR Chicago in a piece I wrote for research-live.

    Below is Kim speaking and a presentation about social media that mentioned the tweet-up at Lux Bar.

    Kim dedeker speaks at esomar chicago #cet gets mentioned

    Related Posts

     

  • Esomar Chicago: Day 1
  • Tagxedo for colorful shapeable saveable word clouds
  • Kama Sutra Statistics
  • What’s Your IQ?
  • It’s your turn to discover the nuances of social media research
  • Why Do I Always Screen Out of Surveys

    Books about survey research and survey design.

    Image via Wikipedia

    If you are a regular survey participant, you’ve probably experienced this. You start answering a survey, and before you get any real questions, you’re told…
    .
    screenout
    .
    Most people don’t like to be screened out of surveys. You started answering the survey because you wanted to finish it. So why does this happen? There are both good and bad reasons for this.
    .
    Sometimes when researchers expect that 100 twenty year old women will complete the survey, they find out that 150 twenty year old women want to complete it. And other times, when they expect to get 100 participants, they only get 50 participants. Then they’re in trouble because they have to find an additional 50 people and still meet the client’s timelines.
    .
    Estimating how many people will want to answer a survey is a tough thing. Researchers know in a general sense how many people will complete a survey. They predict it based on the age, gender, and other demographics of people they sent the survey to. They also predict it based on the topic of the survey and the length of the survey. In the end though, it is just an estimate and sometimes, the number is over or underestimated.
    .
    Never forget that researchers value your participation in surveys. So, when you see that screen-out question, view it as the researcher treating you with respect. They don’t want you to spend your time answering a survey when your opinions won’t be used. Why? Because, as soon as that survey is answered by the number of people required, the data is downloaded and quickly shipped off for analysis. It’s a busy world built on ‘give it to me yesterday’ and marketing research is in the same boat as every other business. I certainly wouldn’t want to spend 30 minutes answering a survey only to find out that my results weren’t downloaded and analyzed.
    .
    This isn’t to say that this is a good or right way to do things though. Most survey companies have many, maybe even hundreds, of surveys in the loop. Wouldn’t it be nice if you never screened out? If you were automatically sent to the next survey which was relevant to you, and you didn’t even see the transition? Ah, can’t wait for that technology to get there for everyone! It’ll come, just not soon enough for me.

    Related Articles

    Paid surveys! Make a living taking surveys!

    Money!

    .

    Aren’t you excited? If you do a quick search on google or twitter for “paid surveys,” you will find a huge listing of sites that claim you can make a living or makes lots of money taking surveys. Google found me 1,670,000 opportunities. WordPress found 845 blog entries hoping to entice me. Please, please, please, accept my public service announcement. You CANNOT make a living taking surveys. Most of these claims are exaggerations, some are outright fabrications. If it sounds too good to be true, it is.

    .
    That’s not to say there are no benefits to answering surveys. You might earn $5 or $10 now and then. You might even be so lucky as to win a TV or a car, but of course, though your chances of winning these prizes are far better than your chances of winning the lottery (trust me, I know), your chances of making it big are pretty much null.
    .
    So why bother answering surveys if you CAN’T rake in the bucks? Well, they are interesting, you learn about new products on the market, you might get to try new products, and you are helping manufacturers improve products so that they better meet your needs. Look at the money as a bonus. It may not be billions and trillions of dollars but ‘ll take 5 bucks now and then!

    Related Articles

     

  • Survey Design Tip #4: Brands are people too
  • Oh oh, I Have Another Favourite Statistic!
  • TV Shows From When I Was Short
  • Radical Stats
  • Ode to a Boxplot
  • Survey Design Tip #2: Short and Sweet