Welcome to the virtual MRIA 2011 annual conference! This post reflects my personal musings and interpretations of this presentation. It was written during the presentation and posted minutes afterward. Any inaccuracies and silliness are my own.
Keynote Speaker: Kees de Jong, CEO Survey Sampling Int. “Panels are people?”
- Everyone is having trouble keeping up with the quality of presentations. Kees is up for the challenge!
- Kees suggests maybe panels are heading for extinction. If you put all your information – join rates, attrition rates, response rates – into Excel, you get a decreasing line. Panels are sick, hurt. Only good thing about panels is the community based panel. Panels are from the past, we don’t want to think about it, people are dropping out, hard to maintain.
- 8 dimensions of why panels are in bad shape.
- Business/Money: Panels were just a big database and money came out of it. 13euros per complete 10 years ago. That was attractive at the time. Recently, the price dropped to a couple dollars because of competition. Enormous focus on the monetary component. Panel buying belongs to the procurement department who just cares about price, not quality.
- Quality: Are panels in bad shape because of quality issues? Topic of probability samples has just arrived. :) Netherlands did a study of 29 panels. RR had no effect on data quality. Discovery of multiple panel membership was a shock. Ocean for fishing was smaller than they thought. Inattentiveness and professional respondents became new terms. Kim Dedecker now mentioned. :) US over-reacted and everyone jumped on this, not because intrinsically motivated, but because the woman with the wallet had spoken. And then tech companies tried to jump in.
- Discussions were from the wrong perspective. Whatever you focus on grows. If you focus on bad respondents, then bad respondents grow. We badmouth the millions of people doing the right thing and helping the industry.
Care: Will panels be extinct because we didn’t care enough? Panelists are most annoyed when they are invited to a survey and then they get screened out, or they screen out after a few questions. Or, when you’re english and get a german invite, etc. Broken links and sloppy work all show a lack of caring. we should have protected them better.
- Experience: The survey itself. Robert Bain felt the environment of surveys was clunky and disrespectful to panelists. Survey length is key, critical thing. Give people short surveys. It’s very simple. Did a study of 29 companies in Netherlands. Response rate had no issue. After 17 mintues, data quality decreased. We must decrease the length of surveys.
- Rewards: We need to understand what drives people. Points, money, it’s different for everyone.
- Are we headed for extinction? Maybe. Ten years ago, there was nothing else online but surveys. Surveys were fun. Now, the world has changed. Competition for attention is exploded. There are still people who love to take surveys but outside that group, it’s hard to get people engaged. We should worry about this.
- How do we solve this? We innovate. Not just technology but changing how we think and deal with issues.
- What if there are inmates in a prison taking surveys to make money for the prison? Scary! Technology takes care of that. Phew!
- There is no standard for validation even if some companies claim they have THE standard.
- New council RVC, Research Validation Council, for standards of practice to rate respondents and surveys, to be officially announced next week. We need some standardization. Has support from ESOMAR and CASRO.
- Rethink, redesign, rewrite surveys to get them shorter and better. Don’t screen people out! Route them to a secondary survey. This is the right solution.
- Last advice: EVERYONE should take their own survey. Why such long attribute lists? That’s not how humans work. And certainly not 20 grids for 20 brands. Stick to one topic. Teach clients that data quality decreases after 17 minutes. Rethink what you do with data. We are used to thinking records but maybe we should start thinking datapoint – not one 40 minute surveys but two 20 minute surveys or four 10 minute surveys. Increase the price of long surveys. Reward people who create great surveys.
- We focus on people who want to be in panels and this group is getting more and more skewed everyday. Normal aren’t on panels. Solution is not to focus on these people but on innovation. Focus on the billion people who are not on panels. Make the person, topic, and sender relevant. (Notice he didn’t say incentive.) Create dozens of environments so people will answer surveys in a relevant space.
- Social media and panels are different people. Sample from everywhere not just your panel, blend sample, but make sure your sampling stream is balanced.
- Holy grail is sample widest range of people based on location, behaviour, and regardless of modality – mobile, inbox, facebook, fully mixed mode.
- Annie Pettit, Chief #MRIA2011 Blogger (lovestats.wordpress.com)
- De Jong rebuffs NDP’s call for data on CareCard fraud (globaltvbc.com)
- Do not call list working for some – but not all – survey finds (theprovince.com)
- Speed Networking + Reception + Dinner #MRIA2011 (lovestats.wordpress.com)