“Reaching for the Holy Grail: Validation Techniques for Online B2B Panels”
When completing surveys online respondents have the ability to claim false credentials in order to qualify for higher paid surveys. This research seeks to apply to the online panel space the same methods used to keep telephone B2B sample clean. The presentation seeks to provide conference attendees with a better understanding of the importance of recruitment source to quality; which validation techniques are effective; and the legal and privacy pitfalls to watch out for when validating business sample.
Edward Paul Johnson, Director of Analytics, SSI
- Our clients want business professionals, qualified decision makers, informed on the topic, engaged and interested
- We know client lists are skewed
- We want it all online because it’s faster and convenient
- Looking for B2B people, it’s hard to first find them. Then you have to reach the boss. Then you have to weed out the fraud to get rid of people who just want the incentives. Then you need to create a lasting relationship because they are valuable for future research as well.
- Many times, you don’t even know the market size of the sample of people you are looking for – computer hardware purchasers?
- Fraud is normally very small, normally only 1 to 2% in the overall panel. But fraud has an advantage because they qualify for more than the honest people do. There might be 20% fraud among business people just because more fraudsters qualify.
- What are our weapons? Existing relationships like hotels airlines and credit cards, data mining the profile to find contradictions, social media linking, phone validation
- Every weapon is a two edged sword – Bigger panel means lower quality but smaller panel means higher quality. Only bigger panels allow more fraudsters and smaller panels eliminate honest people.
- Better to have multiple tests, one with high specificity an done with high sensitivity
- Data mining the profile removed about 15% of panelists – number of reports, company size were important variables. Unusual company size to number of computers was helpful. Over 45 years old and less than 1 year in the industry helped somewhat.
- LinkedIn validation – 600 people volunteered to connect to LinkedIn but profiles were often incomplete, email addresses were different. number of connections and skills was helpful but individual skills were too varied to be helpful. Fraudsters likely don’t volunteer to connect their accounts. Wasn’t a good method.
- Phone validation – Good confirmation test but it excluded good panelists. Some gave bad phone numbers or it was disconnected or they no longer work at that number. Good confirmation test but not a good entry test.
- Tips for phone validation – let them know you will call them at work. Call very close to when they joined, within 2 days. Keep the validation short, to 2 minutes, name company title. Use trained interviewers who know how to bypass gate keepers. The gate keeper might be able to validate this for you.
- DOES improve data quality. Existing relationships isn’t enough. Be careful of excluding good people, can do just as much damage with false positive and false negatives.
- It will never be perfect. There is no holy grail but you can improve it all the time.