Comparing probability and nonprobability samples #AAPOR #MRX 


prezzie #1: how different and probability and nonprobability designs

  • nonprobability samples often get the correct rresults and probability samples are sometimes wrong. maybe they are more similar than we realize
  • nonprobability sampling may have a sample frame but it’s not the same as a census population
  • how do you choose, which factors are important
  • what method does the job that you require, that fits your purpose
  • is the design relevant, does it meet the goals with the resources, does the method gives you results in the time you need, accessability, can you find the people you need, interpretability and reliability, accuracy of estimates with acceptable mean square error, coherance in terms of results matching up with other data points from third parties [of course, who’s to say what the right answer is, everyone could be wrong as we’ve seen in recent elections]
  • nonprobability can be much faster, probability can be more relevant
  • nonprobability can get you right to the people you want to listen to
  • both methods suffer from various types of error, some more than others, must consider total survey error [i certainly hope you’ve been considering TSE since day 1]
  • driver will decide the type of study you end up doing
  • how can nonprob methods help prob methods, because they do offer much good stuff
  • [interesting talk, nice differentiation between prob and nonprob even though I did cringe at a few definitions, eg I dont see that quality is the differentiator between prob and nonprob]

prezzie #2: comparison of surveys based on prob and nonprob

  • limbo – how low can you go with a nonprob sample
  • bandwagon – well everyone else is doing nonprob sample [feelings getting hurt here]
  • statistical adjustment of nonprob samples helps but it is only a partial solution
  • nonprob panel may have an undefined response rate
  • need to look at point estimates and associations in both the samples, does sampling only matter when you need population point estimates
  • psychology research is often done all with college students [been there, done that!]
  • be sure to weight and stratify the data
  • education had a large effect between prob and nonprob sample [as it usually does along with income]
  • point estimates were quite different in cases, but the associations were much closer so if you don’t need a precise point estimate a nonprob sample could do the trick

prezzie #4: sample frame and mode effects

  • used very similar omnibus surveys, included questions where they expected to find differences
  • compared point estimates of the methods as well as to benchmarks of larger census surveys
  • for health estimates, yes, there were differences but where the benchmark was high so were the point estimates, similarly low or moderate point estimates, total raw differences maxed out around ten point
  • there was no clear winner for any of the question types though all highs were highs and lows were low
  • no one design is consistently superior
%d bloggers like this: