After two days at CASRO, I learned the following:
- When you use a 5 point or 7 point scale, you will get different answers
- When you label or don’t label scales, you will get different answers
- When you use a web survey vs a mobile survey, you will get different answers
- When you gamify a survey, you will get different answers
- (And from the good ol’ days) when you run the same survey on two different panels, you will get different answers
What are we to gain from all of this? Well, no matter what you do or how you do it, you will get different results on surveys every time. There’s just no way around it. What we HOPE is that the results won’t be contrary, but rather simply different in magnitude. That rank orders will remain generally similar, that hates will remain hates, and loves will remain loves. Indeed, if we are lucky enough to run a single study across a number of different methods or styles and get similar rank orders every time, it’s a good indication that the conclusions we’ve drawn are both reliable and valid. Heaven.
What this problem also suggests is that there is and can be no right answer. The only right answer is the one in the responder’s head and given that people can’t even adequately describe what is going on in their head, it seems that we will never know the right answer. What we can do is develop clear and specific research hypotheses, and match them up with clear and specific research designs. That is best way to create reliable and valid answers.
We may not know the exact right answer, but we can know a good answer.
- Validity of Gamification: Sweeney, Goldstein, and Becker #CASRO #MRX (lovestats.wordpress.com)
- Cyborgs vs Monsters in modularizing surveys: Edward Paul Johnson and Lynn Siluk #CASRO #MRX (lovestats.wordpress.com)
- Shorter isn’t always better: Inna Burdein #CASRO #MRX (lovestats.wordpress.com)
Wait, was that a typo? Quantity over quality? Well, I meant what I said.
Question #1: What was the sample size of your last tracker? 30 per time frame? 50 per time frame? What about your last custom study? 300? 500?
Question #2: How many pages of questions and demos and cross-tabs did you flip through searching for any chi-square or t-test that was statistically significant? 100? 200?
Here’s the problem. We run ridiculously long surveys with far too few participants per test cell and we are ok with searching through far too many Type 2 errors.
Here’s the solution. Put your money into large sample sizes and not into question topic after question topic. Focus on sample sizes within demographic groups rather than questions with 4 or 8 people per cell. Trade variety of questions for reliability of results. Trade overly long surveys for properly sampled cells. Trade breadth of topics for validity of individual questions. Take money away from more and more questions and put it directly into more and more validity and reliability. Radical.
Please comment below. What was the sample size of your last study and what was the sample size within many of the cells?
- How to encourage speeding in your surveys
- Merry Christmas to all and to all a good sample size #MRX (lovestats.wordpress.com)
- Big Data? Big Deal. #MRX (lovestats.wordpress.com)
- New book! The Listen Lady: A novel and social media research guide baked into one #MRX (lovestats.wordpress.com)
Here’s your task. Read the following list of tasks and identify which ones are useless to brands and clients: – Watching how people interact with and actually use a product – Listening to how people talk about products with their peers – Learning which features people use to convince other consumers – Learning how consumers convince others to use a product
– Observing facial expressions of disgust and shame and love and peace – Watching for passion and complacency
Your second task: Make a list of all of the research methods that are error-free, risk-free and always give valid and reliable results.
There may be no perfect research method but there’s definitely a place for focus groups.
* The link seems to be unavailable now, sorry. If you find an active link, please do share it. *
I just found a quick little free bit of software for calculating reliability. Thanks to Skymeg Software and Dr. Neuendorf for making it available for a small cost of plugging it. Which I don’t mind doing.
It doesn’t have Krippendorfs alpha which is a shame, but free is free.
You can find PRAM here.
“We will not be held accountable for any damages that result from the use, install, download, or thinking about PRAM.”