Do we need to control for non-quota variables? by Deb Santus and Frank Kelly #CASRO #MRX


Live blogged from Nashville. Any errors or bad jokes are my own.

Third author is Peter Kwok

we moved many offline sampling techniques to online sampling. now we have river and dynamic sourcing and routers.

– should we use one or other of both of outgo quotas or return quotas
– balancing quotas are set from sampling frames. usually region, age, gender, household size, often based on US census.
– survey quotas are determined by respondent profiles or subject category.
– some populations are really hard to find. not everyone is simply looking for genpop
– sample frames may not reflect the target populations
– females can respond 20 or more points higher than men
– with river or dynamic sampling, you don’t even know the demos that you’re getting

– router selection is efficient use of respondents but there’s not as much quota control compared to traditional sampling that uses outgo and return balancing
– traditional sampling focused on a specific person for a specific study

carried out a study using various sampling techniques. used interlocking age and gender, plus region.
– 10 minutes, grocery shopping habits, census quotas
cell 1 – 4 balancing variables including income, quotas for outgo
cell 2 – only used age, gender, region quotas on outgo

then weighted to census
– better weights on cell 1, better weight efficiency, minimum weights, maximum weights.
– every type of sample has skews [yer darn right! why do people forget this?]
– controlling for age, gender, region just wasn’t enough
– income and household size did not represent well when they weren’t initially balanced for, marital status also didn’t work well
– some of the profiling questions showed differences as well – belonging to a warehouse club showed differences, using a smartphone to help with chopping showed differences
– quotas do not guarantee a representative sample. additional controls are necessary on outgo. with more controls, weighting can even be unnecessary
– repetition is good. repetition is good. repetition is good (i.e., test-retest reliability is good!)

we need to retain our sample expertise. be smart. learn about sampling and do it well. keep the good things about the traditional ways.

[please please control on the outgo and returns if you can. weighting as a strategy is not the way to think about this. get the sample you need and fuss with it as little as possible through weighting]

%d bloggers like this: