Combining a probability based telephone sample with an opt-in web panel by Randal ZuWallack and James Dayton #CASRO #MRX
Live blogging from Nashville. Any errors or bad jokes are my own.
– National Alcohol Survey in the US, for 18 years plus [because children don’t drink alcohol]
– even people who do not drink end up taking a 34 minute survey compared to 48 minutes for someone who does drink. this is far too long
– only at 18 minutes are people determined to be drinkers or abstainers. [wow, worst screen-out position EVER]
– why data fusion? not everyone is online [please, not everyone is on a panel either. and what about refusals? this fascination with probability panels is often silly]
– RDD measures population percents
– web measures depth of information conditional on who is who
– they matched an online and RDD sample using overlapping variables
– problem is matching can create strange ‘people’ that doesn’t explain real people. however, in aggregate, the distributions work out. we think about it being right on an individual level
– “The awesome thing about having a 45 minute survey”…is the statistical analyses you can do with it [made me laugh. there IS an awesome thing? 🙂 ]
– [SAS user 🙂 Have I told you lately….. that I love SAS]
– There were small differences in frequencies between the RDD and web surveys for both wine and beer. averages are very close but significantly different [enter conversation – when does significantly different mean meaningfully different]
– heavy drinking is much much greater on web surveys
– is there social desirability, recall bias 🙂
– not everything lines up perfectly RDD vs web, general trends are the same but point estimates are different
– so how do you know which set of data is true or better?
– regardless, web does not reproduce RDD estimates
– problem now is which data is correct, need multiple samples from the same panel to test
Probability and Non-Probability Samples in Internet Surveys
Moderator: Brad Larson
Understanding Bias in Probability and Non-Probability Samples of a Rare Population John Boyle, ICF International
- If everything was equal, we would choose a probability sample. But everything is not always equal. Cost and speed are completely different. This can be critical to the objective of the survey.
- Did an influenza vaccination study with pregnant women. Would required 1200 women if you wanted to look at minority samples. Not happening. Influenza data isn’t available at a whim’s notice and women aren’t pregnant at your convenience. Non-probability sample is pretty much the only alternative.
- Most telephone surveys are landline only for cost reasons. RDD has coverage issues. It’s a probability sample but it still has issues.
- Unweighted survey looked quite similar to census data. Looked good when crossed by age as well. Landline are more likely to be older and cell phone only are more likely to be younger. Landline more likely to be married, own a home, be employed, higher income, have insurance from employer.
- Landline vs cell only – no difference on tetanus shot, having a fever. Big differences by flu vaccination though.
- There are no gold standards for this measure, there are mode effects,
- Want probability samples but can’t always achieve them
A Comparison of Results from Dual Frame RDD Telephone Surveys and Google Consumer Surveys
- PEW and Google partnered on this study; 2 question survey
- Consider fit for purpose – can you use it for trends over time, quick reactions, pretesting questions, open-end testing, question format tests
- Not always interested in point estimates but better understanding
- RDD vs Google surveys – average different 6.5 percentage points, distribution closer to zero but there were a number that were quite different
- Demographics were quite similar, google samples were a bit more male, google had fewer younger people, google was much better educated
- Correlations of age and “i always vote” was very high, good correlation of age and “prefer smaller government”
- Political partisanship was very similar, similar for a number of generic opinions – earth is warming, same sex marriage, always vote, school teaching subjects
- Difficult to predict when point estimates will line up to telephone surveys
A Comparison of a Mailed-in Probability Sample Survey and a Non-Probability Internet Panel Survey for Assessing Self-Reported Influenza Vaccination Levels Among Pregnant Women
- Panel survey via email invite, weighted data by census, region, age groups
- Mail survey was a sampling frame of birth certificates, weighted on nonresponse, non-coerage
- Tested demographics and flu behaviours of the two methods
- age distributions were similar [they don’t present margin of error on panel data]
- panel survey had more older people, more education
- Estimates differed on flu vaccine rates, some very small, some larger
- Two methods are generally comparable, no stat testing due to non-prob sample
- Trends of the two methods were similar
- Ppanel survey is good for timely results
Probability vs. Non-Probability Samples: A Comparison of Five Surveys
- [what is a probability panel? i have a really hard time believing this]
- Novus and TNS Sifo considered probability
- YouGov and Cint considered non-probability
- Response rates range from 24% to 59%
- SOM institute (mail), Detector (phone), LORe (web) – random population sample, rates from 8% to 53%
- Data from Sweden
- On average, three methods differ from census results by 4% to 7%, web was worst; demos similar expect education where higher educated were over-represented, driving licence over-rep
- Non-prob samples were more accurate on demographics compared ot prob samples; when they are weighted they are all the same on demographics but education is still a problem
- The five data sources were very similar on a number of different measures, whether prob or non-prob
- demographic accuracy of non-prob panels was better. also closer to political atittudes. No evidence that self recruited panels are worse.
- Need to test more indicators, retest
Modeling a Probability Sample? An Evaluation of Sample Matching for an Internet Measurement Panel
- “construct” a panel that best matches the characteristics of a probability sample
- Select – Match – Measure
- Matched on age, gender, education, race, time online, also looked at income, employment, ethnicity
- Got good correlations and estimates from prob and non-prob.
- Sample matching works quite well [BOX PLOTS!!! i love box plots, so good in so many ways!]
- Non-prob panel has more heavy internet users
- Thoughts on the CMRP designation #MRX #NewMR (mriablog.wordpress.com)
- Minimizing Nonresponse Bias (GREAT session) #AAPOR #MRX (lovestats.wordpress.com)
- The Roles of Blogs in Public Opinion Research Dissemination #AAPOR #MRX (lovestats.wordpress.com)
- AAPOR Women Leaders Share Their Insights #AAPOR #MRX (lovestats.wordpress.com)
[tweetmeme source=”lovestats” only_single=false]As promised on Twitter, here is the final chapter of my short book on probability sampling. 🙂
For those of us in the real world, where probability sampling is the impossible dream (see previous posts if you’re puzzled why I say this), what are we to do? What’s the point of doing any serious marketing research if we can’t even meet the most basic statistical requirements required for predictive analytics?
Let’s see. We’ve been doing heavy duty market research for at least 50 years now. We’ve been dang good at predicting the success of products and the failure of politicians. And, acknowledging the 5% of predictions we got wrong because we know there is always unpredictable error, we’ve been doing a dang good job WITHOUT absolutely perfect probability samples.
What? How is that possible? Predicting the future without using probability sampling? Of course it’s possible. Even if researchers can’t sample perfectly, we know how to sample really well with what we’ve got. We’ve been learning this skill for decades. We learn the idealized theory in school, we learn the practical theory on the job, and we tinker around with all the tidbits at our disposal to put together a pretty fine sampling job at the end.
We know how to identify flaws in data. We know how to clean data. We know how to write quality surveys (whether we always do is another story). We know how to interpret ambiguous data. We know our stuff.
We know our stuff so well that we are able to predict the future even when glitches creep into our research process.
So tell me again. Why all the failed efforts to prove we use probability sampling when we’ve taught ourselves to work smart with what we’ve actually got?
This rant has been brought to you by the letter P and the number 5.
Read these too
I completely disagree with my own title!
My world is online. I started out writing online surveys in 1996 when I bugged the computer helpdesk at my graduate school to set me up with an online database. No one else at the university had ever done such a thing and i confused the heck out of them. I wrote my own html code which allowed me to specify font sizes, font colours, page colours, radio buttons, check boxes and text boxes. ooooooo….. so sophisticated. I’d be embarrassed to tell a scripter now that “I write my own code.”
Online research has never tried to say it uses probability sampling but, other methods of research have. There has been a debate over the last year specifically directed at online panels. Well, not really a debate. Some folks have been outraged that online panels do not use probability sampling and therefore they do not qualify to use statistics. To go even further, they suggest that telephone samples do use probability sampling and so results from that type of research are the most valid.
Let me offer up some ideas…
Telephone research – Do you always answer your phone? Is your phone number unlisted? Do you return phone calls? Do you politely tell telephone interviewers that you are busy when in fact you are nursing a bag of cheetos?
Mail research – Do you just throw out all the junk you get in the mail? Do you fill out surveys AND mail them?
Online research – Are you signed up for an online survey panel? Do you click on the survey banners that appear after you run a search and then finish every survey?
It seems to me that no matter how hard you try to use probability sampling, human beings just cannot cooperate. We’re not worms or mice or molecules. People choose when they wish to pay attention or participate. It’s not online panels. It’s research with human participants.
Probability sampling of people? No such thing.