Survey Panel Questions – Enough Already!

Survey panels are big right now. If you want to launch a survey to hundreds or thousands of people across the country, chances are you go to a survey panel company. They have pre identified permission based access to people who are ready and willing to take surveys at the drop of a hat. When it comes to panel companies, I’ve had to explain the following two issues so many times I thought I’d just lay them out right here.

1) Your panel size is too small to meet our needs
Then how can one panel company have over 1 million panelists and another company stays in business with only 200k panelists? How does that work?
Well, let’s look at an example.

Company A has 1 million panelists and their return rate is 5%. If this panel was to launch to every single panelist right now, they would get 50 000 completes.
Company B has 200 000 panelists and their response rate is 25%. If they were to launch to everyone, they would also get 50 000 completes.
So basically, these two panels are identical! Different panel sizes, exact same outcome.

Why is Company A so much larger? Well, panels recruit thousands and thousands of people every year. Technically, they could advertise the size of their panel to be anything they wanted to. But, there’s a little thing called the panel rule that determines the real size of a panel.

A company could let anyone be ‘active’ on their panel as long as they clicked (and didn’t even finish!) a survey in the last 12 months. This panel will be really big, but since many of their panelists never even finished a survey, their response rates will be pretty low. On the other hand, another company could use a much stricter rule. Maybe it’s something like people are ‘active’ as long as they finished a survey in last 3 months. In this case, a lot fewer people meet the qualification, but they are all survey completers. This means their response rates are quite a big higher.

So, it comes down to what’s your panel size AND what’s your response rate. That will tell you the real size of a panel.

2) Your panel doesn’t look anything like what I need
This one usually comes to me as “I need to know the demographic profile of your panel so that i can determine whether you can run my survey.” Well, the reason panel companies have such huge panels is so that they can pick and choose from among their panelists to create the group that you need. Even if their panel is only 40% male, they can easily choose a sample for you that is 50% male or even 80% male.

Related Articles


2 responses

  1. Definitely. The ARF is a great source for data quality issues. Would have attended if I was still representing Ipsos!

  2. the definitive research on research project about online research data quality has been conducted by the ARF and will be presented at our annual conference. If you want the facts, come to NY March 30-april 1.

%d bloggers like this: