Tag Archives: survey panel

Screener keeners or rejection correction?

[tweetmeme source=”lovestats” only_single=false]How much is too much? How much is too little? There are lots of things in market research that require a healthy balance between doing the right thing and conducting business. Deciding how many screeners to offer to potential survey responders is one of them.

Most survey panels recognize that screening people out of surveys, no matter why, is bad for two reasons. First, and completely justified, it ticks off panelists who feel their time has been wasted and their opinions ignored. Second, it’s a waste for panels that just used up one survey in their data quality rule of “one survey invite per week” and they didn’t even get a complete in return.

For both of these reasons, many panels strive to handle the problem by offering up a number of surveys in a row to panelists. Panelists receive an invite and then proceed through one or more consecutive screeners until they qualify for a survey. (Let’s not consider what this means for probability sampling.)

But what is the right number of screeners? Is it ok to send someone through ten minutes of screeners? Is it ok to give them two or three screeners?

Photo credit: xenia from morguefile.com

I just spoke with someone who said their company takes people through up to five screeners before they say enough is enough. Panelists are even compensated for each screener they complete. I worry that even though they are being compensated, it is annoying to panelists. Screeners are obviously not surveys. Panelists can tell that they’ve been rejected once, twice, three, four, and five times. Imagine being rejected by five screeners every time you try to participate. It’s just one more source of rejection, something none of us need now or ever.

In fact, I even wonder if there is a rejection effect for which I have a two tailed hypothesis. Does increased rejection cause decreased survey scores due to the annoyance or does increased rejection cause increased survey scores due to the satisfaction of finally getting a survey to answer. I’d love an answer to that!

So what does your experience tell you? Are responders keeners for screeners?

Read these too

  • #MRA_FOC #MRX Karaoke by Jim Longo. I dare you.
  • If Frozen Meat Isn’t a Bribe, What About Raw Meat?
  • It’s Here! Twitterversity is Here! #MRXU
  • Proof that sampling social media data is critical, but you already knew that
  • If you DARE – the chart test of death
  • It’s your turn to discover the nuances of social media research

    Infographic on how Social Media are being used...

    Image via Wikipedia

    Remember the good old days when every offline study was parallel tested online? Remember all the headaches of changing baselines and bizzareness out of nowhere? The nature of data is that it doesn’t stay put even when you yell at it.

    Social media research is the same. If you’re used to seeing results from online surveys a certain way, you will see shifts when you start doing social media research. Some of these shifts will fall into the bizarreness out of nowhere category. There is no way to explain where they came from and you will never be able to.

    On the other hand, some differences will be very real. There are many reasons why.
    • Your new dataset is comprised of people who’ve probably never been an active member of an online survey panel. Up until now, they’ve never been represented in research. They hung up on phone surveys and closed all the “Take survey now” pop-ups. These folks now have a voice.
    • Your new data allows people to express themselves in a way never before permitted. Any topic, any tangent, any words, any slang, any rudeness. If people feel it is important enough to communicate it, it will be captured. Think of all the surveys, even the 60 minute surveys, that just weren’t long enough to capture that minute topic.
    • Your new data is measured on a different scale. People are used to forcing their opinions into a box, whether it’s the “strongly agree” box or the “yes I do” box. Now, their words fit into an unlimited number of boxes and they don’t have to feel like they just can’t box their answer.

    Don’t be afraid of different results. Embrace them as discovering nuances in your data. It’s a martha stewart good thing. Now have a warm homemade cookie.

    Read these too

  • Check Out the Statistical Outlier on my LinkedIn Cluster Analysis
  • Why do surveys ask the same question 8 billion times?
  • How cool is market research? #mrx Social media research is the new one size fits all
  • How to upset me by generating leads with market research surveys
  • Data Tables: The scourge of falsely significant results #MRX
  • 1 topic 5 blogs: Embracing the evolution of listening
  • A Review of NetGain 4 2010: Some old and finally some new

    [tweetmeme source=”lovestats” only_single=false]

    First, thank you to the MRIA for hosting this event. It was well attended by about 160 people, most of whom stayed until the very end. THAT is a key performance measure with substance.

    Second, thank you to Ronald McDonald House Charity, RMHC, for permitting me to share some of their social media research story. I think that seeing real data helps people really understand how research works.

    Alrighty! We started the morning session with a focuse on online panel quality. We had some great name brand speakers who shared a lot of very important industry information. Unfortunately, I must be honest. I didn’t hear one single new piece of information. In fact, the info was three years old. In my opinion, all researchers should know these things hands down:

    • Panels are different
    • Different panels generate different data
    • Don’t switch trackers between panels
    • There are fraudsters in panel data
    • Panels aren’t probability samples

    I guess if you are a brand new researcher, it would have been helpful. I was disappointed. 😦

    After lunch of butter tart squares and chocolate mousse bites (I think there was meat and veggies too but I don’t remember that part), the afternoon began with a focus on social media. Finally something really new and actually innovative. This information was new for many, if not most, people. Many attendees admitted to not being on twitter. In fact, only about 10 of 160 said they were on twitter. But, I met with several people who said that because of today, they were going to try it out. If you’re in this group, go try it now and give it two full weeks of real testing before you give up.

    We learned about some ways social media can be used for research.

    • Online communities are huge now and provide a way for researchers to talk to consumers about focused topics.
    • Facebook groups are great for talking to really low incidence target groups.
    • Virtual currencies can be used as incentives.
    • Social media research is still research though it uses a different dataset.

    I found the afternoon more interesting because new ideas were being talked about. It wasn’t online surveys or panel or cell phone surveys. It was the next generation of research. I’m excited to see that the MR industry is finally branching out into new areas. Let’s keep it up.

    Finally, it was my pleasure to speak at the conference. So many people said so many nice things to me about my presentation. I even met a few tweeters I’d never met before. Thank you to everyone who encouraged me and left me feeling appreciated.

    Read these too

  • In Honor of Infographics. #MRX
  • This is why Twitter will die
  • Only crazy people are on twitter
  • Mugging, Sugging and now Rugging: I take a hard stance on privacy
  • I’m sorry but representative samples are 100% unattainable
  • Probability Sampling – Proof that only telephone samples are quality samples

    Histogram of sepal widths for Iris versicolor ...

    Image via Wikipedia

    I completely disagree with my own title!
    My world is online. I started out writing online surveys in 1996 when I bugged the computer helpdesk at my graduate school to set me up with an online database. No one else at the university had ever done such a thing and i confused the heck out of them. I wrote my own html code which allowed me to specify font sizes, font colours, page colours, radio buttons, check boxes and text boxes. ooooooo….. so sophisticated. I’d be embarrassed to tell a scripter now that “I write my own code.”
    Online research has never tried to say it uses probability sampling but, other methods of research have. There has been a debate over the last year specifically directed at online panels. Well, not really a debate. Some folks have been outraged that online panels do not use probability sampling and therefore they do not qualify to use statistics. To go even further, they suggest that telephone samples do use probability sampling and so results from that type of research are the most valid.
    Let me offer up some ideas…
    Telephone research – Do you always answer your phone? Is your phone number unlisted? Do you return phone calls? Do you politely tell telephone interviewers that you are busy when in fact you are nursing a bag of cheetos?
    Mail research – Do you just throw out all the junk you get in the mail? Do you fill out surveys AND mail them?
    Online research – Are you signed up for an online survey panel? Do you click on the survey banners that appear after you run a search and then finish every survey?
    It seems to me that no matter how hard you try to use probability sampling, human beings just cannot cooperate. We’re not worms or mice or molecules. People choose when they wish to pay attention or participate. It’s not online panels. It’s research with human participants.

    Probability sampling of people? No such thing.

    Related Articles

    %d bloggers like this: