Tag Archives: Frank Kelly

Do we need to control for non-quota variables? by Deb Santus and Frank Kelly #CASRO #MRX

Live blogged from Nashville. Any errors or bad jokes are my own.

Third author is Peter Kwok

we moved many offline sampling techniques to online sampling. now we have river and dynamic sourcing and routers.

– should we use one or other of both of outgo quotas or return quotas
– balancing quotas are set from sampling frames. usually region, age, gender, household size, often based on US census.
– survey quotas are determined by respondent profiles or subject category.
– some populations are really hard to find. not everyone is simply looking for genpop
– sample frames may not reflect the target populations
– females can respond 20 or more points higher than men
– with river or dynamic sampling, you don’t even know the demos that you’re getting

– router selection is efficient use of respondents but there’s not as much quota control compared to traditional sampling that uses outgo and return balancing
– traditional sampling focused on a specific person for a specific study

carried out a study using various sampling techniques. used interlocking age and gender, plus region.
– 10 minutes, grocery shopping habits, census quotas
cell 1 – 4 balancing variables including income, quotas for outgo
cell 2 – only used age, gender, region quotas on outgo

then weighted to census
– better weights on cell 1, better weight efficiency, minimum weights, maximum weights.
– every type of sample has skews [yer darn right! why do people forget this?]
– controlling for age, gender, region just wasn’t enough
– income and household size did not represent well when they weren’t initially balanced for, marital status also didn’t work well
– some of the profiling questions showed differences as well – belonging to a warehouse club showed differences, using a smartphone to help with chopping showed differences
– quotas do not guarantee a representative sample. additional controls are necessary on outgo. with more controls, weighting can even be unnecessary
– repetition is good. repetition is good. repetition is good (i.e., test-retest reliability is good!)

we need to retain our sample expertise. be smart. learn about sampling and do it well. keep the good things about the traditional ways.

[please please control on the outgo and returns if you can. weighting as a strategy is not the way to think about this. get the sample you need and fuss with it as little as possible through weighting]


Mobile and CAWI Parallel: Frank Kelly and Sherri Stevens #CASRO #MRX

… Live blogging from beautiful San Francisco…


Bite-Sized Chunks’—Mobile and CAWI Parallel Proposal by Frank Kelly, Global Director, Lightspeed Research  and  Sherri Stevens, Director, Millward Brown

  • Why modular designs for surveys? Long surveys have lower response rates, younger people don’t want to answer longer surveys, they’re platform agnostic
  • 65% of smartphone users would not want to spend more than 15 minutes completing surveys – even worse for people using smartphones vs computers and tablets
  • 80% of mobile responders say the experience is very easy and 34% are likely to take another survey. They did say that typing was the most difficult so consider the difficulties of diaries
  • Results are generally similar across platforms given the survey is mobile friendly and quotas are handled carefully
  • Chunk within respondents by giving a respondent several very short surveys. Or, chunk across respondents by giving each person only one of the segments and then put it all together afterwards.
  • Basic principles
    • Use specific questions or hooks to use across all cells to be able to fuse modules on the backend, not just awaress and intent
    • Determine what does and doesn’t need to be fused
    • Create a control cell for piloting
  • Without common questions, correlation matrices are insufficiently populated
  • In their research, the same four factors were extracted except they were in a different order
  • Continuous data seems to work better than categorical scale data
  • Less mentions of surveys being repetitive, liked the length more
  • Dropouts increase but it is relatively small 2% to 7% so be sure you can still use the partial data
  • Try to randomize modules but hold the core module in the first position
  • Fatigue effects can be reduced
  • Potential for fieldwork cost savings

If I could only pick three conference sessions to go to… #MRMW #MRX

If I can’t be there in person, I’ll certainly be there in spirit. MRMW Asia is sure to be a great event given the line up of speakers and topics in a fascinating city. And though I’d go to every session if I was there, if I could only see three, these are the three that I’d pick.

I’d love to see Frank Kelly’s presentation on the use of mobile devices in different countries. Canada, where I’m from, is known for its multiculturalism but that doesn’t mean I’ve had the opportunity to step into the homes and experience the lives of people who are so different from me. Anytime we can gain a better understanding of the differences between cultures means our research results will be better and more relevant to the right people. Even better, it means that as humans, we’ll have an increased understanding and more empathy for each other. And we all know we need more of that on this planet.

annie mobile

Self portrait via mobile for a facial coding exercise

I’m also thrilled to see that ethics and standards are on the agenda twice, first with a solo session by Mark Michelson and second with a panel moderated by Michelson. As a social media researcher, I see the problems of loose standards and ethics day in and day out. Companies have been charged with deceiving consumers, consumers have been furious over their loss of privacy, and clients have given up on methodologies because multiple vendors have misled them about validity and capabilities. This isn’t good for anyone, and certainly not good for an industry that prides itself on fairness and honesty. I’d love to see Ho, Ooi, Niles, and Foreman take a hard stand for consumer privacy and research ethics.

And last, I’d love to see Pankaj Jha’s presentation on facial coding. As a past Psychologist, I know there is validity behind the theory of facial coding. I know that researchers are also getting better at transferring the theory of facial coding into valid practice, though we aren’t there quite yet. If that’s not enough, let’s just see if we can take something that is already difficult and make it even more difficult by going mobile. Hopefully, the mobile picture I took of myself thinking about this conference will be easy to code – good lighting, clear expression. I’m sure every consumers’ image will make it this easy for Jha.

To those fortunate enough to be attending, keep the rest of us in the loop with your blog posts and tweets. We’ll be anxiously waiting for them!

%d bloggers like this: