“Are There Perils in Changing the Way We Sample our Respondents?”
Sample and panel providers are always looking to increase their active sample size. In recent years this has taken many companies out of email lists into real time sampling via ad banners or social networks. Research has revealed that panelists recruited by such methods are substantially different than the panelists that opt into online panels. This study addresses the various methods panels implement to generate additional sample, and the tradeoffs these methods require. While there is a clear short term gain of added panelists, there may be long term loss of data stability and panel tenure.
- Inna Burdein, Director of Analytics, The NPD Group, Inc.
- Is there a differences among people who take several surveys in a row versus taking surveys off your website
- Tested data from website survey, email survey, and follow-up survey
- 1400 completes per group
- Website takers are younger and newer.
- Website takers express more interest in surveys and incentives [or they just like clicking a lot]
- Website takers are more online, google a lot, lots of free time
- Completion rates are higher for website takers, and then follow-on surveys. Email takers are last.
- Website takers are more satisfied – easy, reasonable, interesting
- Website takers have more inconsistencies and not following instructions. Follow-ons are more likely to straightline and opt out of responding.
- Website panelists report more purchases, more store visits, more browsing stores, more online purchases, make home improvements, redecorate, go on vacation, invest in stock market
- [More likely to report purchases does not mean more likely to purchase]
- One follow on is kind of normal, but two follow-ons is where the differences happen, more unhappiness, more non-purchase, more straightlining, more use of none of the above
- Significant differences do emerge [but I wonder how many are truly meaningful, would you run your business differently if you got number A vs number B]
- Are there perils in changing the way you sample? It depends. Need enthusiastic responders and more representativeness. Tell people to answer on the website. Possibly balance on channel
- Follow-ons may hurt sample quality if no limit is set – time spent, number of surveys, what is the right rule?