Every time we test a new subject line, a new survey question, a new image, a new way-cool-and-so-much-better alternative, we always use a control group. That way, when we see the results from the test group, we can immediately see whether it was better, worse, or the same as the way we usually do things.
But let’s take a step back for a minute and remember how “random” sampling and probability works. Statistics tell us that even superior quality research designs are plagued by random chance occurences. That’s why when we report numbers, we always put a confidence interval around them, say 3 or 4 points. And then on top of that, we have to remember that five per cent of the time, the number we see will be horribly different from reality. Sadly, we can never know whether the number we’ve found is a few points away from reality or 45 points away from reality.
The only way to know for sure, or at least, nearly for sure, is to re-run the test 19 times. To hit the 20 times that allow us to say “19 times out of 20.”
And yet, we only ever use one single control group before declaring our test group a winner, or loser. One control group that could be wrong by 3 points or 43 points.
So here’s my suggestion. Enough with the control group. Don’t use a control group anymore. Instead, use control groups. Yes, plural. Use two control groups. Instead of waiting a week and redoing the test again, which we all know you haven’t got the patience for, do two separate control groups. Launch the control survey twice using two completely separate and identical processes. Not a split-test or hold-back sample. Build the survey. Copy it. Launch full sample to both surveys.
Now you will have a better idea of the kind of variation to expect with your control group. Now you will have a better idea of how truly different your test group is.
No, it’s not 19 repetitions by 19 different researchers in 19 different countries with 19 different surveys but it’s certainly better than assuming your control group is the exact measure of reality.
Nearly every day, I see a really cool statistic on TV or the interweeb. Everyone gets all excited about losing 312 pounds in four days, curing cancer, or eliminating measles forever. Candy is good for you! Coffee increases your memory! Drink more wine! Eat more Doritos! But if we paid ANY attention to the research methodology, you’d ignore the entire study. Here are a few of the biggest problems I see.
1) Significantly increased memory!!! Yes, when the sample size is large enough or the difference is large enough, anything is significant. So if 5 people in the control group remembered 5 things and 5 people in the test group remembered 8 things, the difference might be statistically significant. Or, if 1000 people in the control remembered 5 things and 1000 people in the test group remembered 5.2 things, the difference might be statistically signficant. Do you trust the results based on 10 people? Do you care about a difference of 0.1 points? I don’t. Get back to me when your sample sizes and effect sizes go beyond pre-test methodology sizes.
2) Cancer rates decreased by 75%!!! Yes, very nice finding. Especially when the cancer rate of the control group was 0.04% and the cancer rate of the test group was 0.01%. That is indeed a 75% decrease but will that massive decline of 0.03 points mean that you stop eating chocolate or start drinking wine? Doubt it. It’s not a meaningful difference when it comes to one single person. Get back to me when the rate decreases by 75% AND the base rate can be measured without any decimal places.
3) Chocolate makes you thin!!!! I’m sure it did. In that one, single study. That has never been replicated. Remember how we compare all our findings against a 5% chance rate? Well, that’s what you just discovered. The 5% chance where the finding occurred randomly. Run the research another 19 times and then get back to me when 19 of them say that chocolate makes you thin.
There are about 423 other cautions to watch out for, but today has been brought to you by the number three.
I’m on Twitter a lot. I tweet a lot, I read it a lot. But I seem to be gravitating more and more to Facebook even though the powers that be tell me Facebook is about to implode. But I’ll tell you why I’m dating Twitter less than I used to.
- It’s getting harder and harder to talk to a person. As people realize the value of branding, more and more Twitter accounts are being named after companies and brands. They tweet endlessly all day long and attempt to engage in conversation. But I really can’t remember the last time I picked up a box of Cheerios and had a fun and interesting conversation with it. I talk to people, not brands. Even though I deliberately follow people over brands, my Twitterstream is an endless list of brands and companies, as if every human has stopped tweeting. Why not sign each tweet with -Annie, or put -Annie in your user bio. Yup, change your user bio every day depending on who’s tweeting.
- More and more people are begging to be followed. Outright asking for people to follow up them. No, they aren’t buying followers and I do appreciate that. But I don’t appreciate tweets along the lines of “Hey LoveStats, I love your tweets. Please follow us.” You see, if you tweeted with me even a couple of times, I would have reviewed your tweets and determined for myself whether your tweets were of interest to me. In other words, if you’ve tweeted with me and I’m not following you, it’s likely because I don’t want to follow you.
- Fewer and fewer tweets are personal. As I already said, I prefer to talk to people. And have lunch and play with their kids and do embarrassing things. It’s fun to read these things because first of all, well, they’re fun. And second of all, it helps me get to know you as a person, you as someone I’d like to talk to. I firmly believe there is a healthy balance between professional and serious, and fun and friendly. As a market research community, we are losing the right balance.
In conclusion, please treat me like a person wearing pink socks and eating chocolate, not a robot that might open a wallet for you.
What is market research? Well, to put it simply, it’s surveys and focus groups and interviews and communities all pulling together to help us learn more about markets, marketing, and consumers.
It’s a field that’s taken a beating in the last few years as outsiders have brought innovative techniques to light. These new techniques have stolen share from market research and will continue to do so in the coming years.
Like it or not, big data, analytics, and data scientists have taken our jobs right out from under us.
But wait. What is market research again? It’s actually not surveys or focus groups. Market research is really the application of scientific methods to understand markets and consumers. It has nothing to do with specific techniques. It’s science. Just science. And big data is science. Data scientists use science. Indeed, market research by any other name is still science.
Is market research going to die? No. Is your job going to die? Maybe. Especially if you don’t keep up with the newest scientific methods. So go learn SAS and SQL and R. Go learn and your career in market research will never die. Your title might change but not your career.
–Written on the go
I have a problem. Anytime I think of the word platykurtic, I immediately think of this cartoon. I saw it many times as a youngster and for some reason always got a big kick out it. Now, the fact that platykurtic kind of sounds like FLATykurtic which describes the flat nature of a platykurtic normal curve makes it easy for me to remember that a platykurtic curve is a flatykurtic curve. I think that’s pretty clear so enjoy the cartoon yourself.
More cartoons from my childhood. Simon in the land of chalk drawings was another TV show I was allowed to watch. I still love the sing and you just might catch me singing it.
I have a number of favorite interview tests, and this is yet another one. Here’s the game: I draw two different charts, just like what you see here,except without the labels on the axes. One chart has a very slow rise and the other has a very steep rise. Then, I ask my poor interviewee which chart they would like to represent the raises they will receive from year to year.