The Best Panel is a Census Rep Panel, NOT! #MRX

When you’re commissioning a new survey project, it can be hard to select the best survey panel for the job. There are many criteria to judge including response rates, data quality processes, panel sizes, and panel make-up.

Response rates are clear. If panelists don’t answer surveys, you get no completes. Data quality is clear. If panelists are speeders or random responders, you get garbage data. If the panel isn’t large enough, you don’t get enough completes. But what about panel make-up?

Logo for the 2010 United States Census.

What does the ideal panel look like? One of the most common misconceptions is that a survey panel should be census rep. Therefore, a Canadian panel should have the same demographics as the Canadian census and a US panel should have the same demographics as the US census. Unfortunately, this is NOT the ideal make-up of a survey panel.

Let’s think about the kinds of samples that we want to survey. Certainly, many survey projects are interested in census rep samples. Political surveys and social surveys for sure need to understand how a census representative population feels. But think a bit more. How often do you need samples of 1) males 18 to 34, 2) mothers of teenages, 3) mothers of babies, and 4) adults aged 65 and more. Those types of samples couldn’t be further from census rep and yet they are more representative of the types of samples that researchers need, the types of people they need to have as part of a survey panel.

So here is what the ideal panel looks like. It looks like what survey researchers need. And if researchers send 25% of their surveys to people aged 18 to 24, then 25% of their panel should be aged 18 to 24. (It’s actually more complicated than this as we must take into account that young people have lower response rates and therefore the panel should probably be 35% aged 18 to 24.) Similarly, since older people are less often the target of surveys, they should be underrepresented on a panel compared to census. (And even MORE underrepresented because their responses rates are much higher than average.)

The reason for this comes to the annoyance factor. If survey panels were census rep, we would have a lot of very annoyed, very frustrated younger people. They would be receiving far more surveys than other people and the demand on their time would be much more.  On the other hand, older people, who aren’t the focus of as many research objectives would receive far fewer surveys than other people and they would more easily become disengaged and disappointed at the lack of involvement. Neither of these situations is ideal.

So the next time you’re considering a research panel, don’t ask the providers if it’s census rep. Ask instead about the average number of invitations each panelist receives. Find out if some people receive 5 survey invites every week while other people receive only 5 invites every year. Find out if they treat their panelists nicely.

One response

  1. And the fact that panel smaple is still a non-probability sample so do not assume it is a representative of entire universe. As such, we really cannot report margin of error 🙂

%d bloggers like this: