Nearly every day, I see a really cool statistic on TV or the interweeb. Everyone gets all excited about losing 312 pounds in four days, curing cancer, or eliminating measles forever. Candy is good for you! Coffee increases your memory! Drink more wine! Eat more Doritos! But if we paid ANY attention to the research methodology, you’d ignore the entire study. Here are a few of the biggest problems I see.
1) Significantly increased memory!!! Yes, when the sample size is large enough or the difference is large enough, anything is significant. So if 5 people in the control group remembered 5 things and 5 people in the test group remembered 8 things, the difference might be statistically significant. Or, if 1000 people in the control remembered 5 things and 1000 people in the test group remembered 5.2 things, the difference might be statistically signficant. Do you trust the results based on 10 people? Do you care about a difference of 0.1 points? I don’t. Get back to me when your sample sizes and effect sizes go beyond pre-test methodology sizes.
2) Cancer rates decreased by 75%!!! Yes, very nice finding. Especially when the cancer rate of the control group was 0.04% and the cancer rate of the test group was 0.01%. That is indeed a 75% decrease but will that massive decline of 0.03 points mean that you stop eating chocolate or start drinking wine? Doubt it. It’s not a meaningful difference when it comes to one single person. Get back to me when the rate decreases by 75% AND the base rate can be measured without any decimal places.
3) Chocolate makes you thin!!!! I’m sure it did. In that one, single study. That has never been replicated. Remember how we compare all our findings against a 5% chance rate? Well, that’s what you just discovered. The 5% chance where the finding occurred randomly. Run the research another 19 times and then get back to me when 19 of them say that chocolate makes you thin.
There are about 423 other cautions to watch out for, but today has been brought to you by the number three.
I’m on Twitter a lot. I tweet a lot, I read it a lot. But I seem to be gravitating more and more to Facebook even though the powers that be tell me Facebook is about to implode. But I’ll tell you why I’m dating Twitter less than I used to.
- It’s getting harder and harder to talk to a person. As people realize the value of branding, more and more Twitter accounts are being named after companies and brands. They tweet endlessly all day long and attempt to engage in conversation. But I really can’t remember the last time I picked up a box of Cheerios and had a fun and interesting conversation with it. I talk to people, not brands. Even though I deliberately follow people over brands, my Twitterstream is an endless list of brands and companies, as if every human has stopped tweeting. Why not sign each tweet with -Annie, or put -Annie in your user bio. Yup, change your user bio every day depending on who’s tweeting.
- More and more people are begging to be followed. Outright asking for people to follow up them. No, they aren’t buying followers and I do appreciate that. But I don’t appreciate tweets along the lines of “Hey LoveStats, I love your tweets. Please follow us.” You see, if you tweeted with me even a couple of times, I would have reviewed your tweets and determined for myself whether your tweets were of interest to me. In other words, if you’ve tweeted with me and I’m not following you, it’s likely because I don’t want to follow you.
- Fewer and fewer tweets are personal. As I already said, I prefer to talk to people. And have lunch and play with their kids and do embarrassing things. It’s fun to read these things because first of all, well, they’re fun. And second of all, it helps me get to know you as a person, you as someone I’d like to talk to. I firmly believe there is a healthy balance between professional and serious, and fun and friendly. As a market research community, we are losing the right balance.
In conclusion, please treat me like a person wearing pink socks and eating chocolate, not a robot that might open a wallet for you.
What is market research? Well, to put it simply, it’s surveys and focus groups and interviews and communities all pulling together to help us learn more about markets, marketing, and consumers.
It’s a field that’s taken a beating in the last few years as outsiders have brought innovative techniques to light. These new techniques have stolen share from market research and will continue to do so in the coming years.
Like it or not, big data, analytics, and data scientists have taken our jobs right out from under us.
But wait. What is market research again? It’s actually not surveys or focus groups. Market research is really the application of scientific methods to understand markets and consumers. It has nothing to do with specific techniques. It’s science. Just science. And big data is science. Data scientists use science. Indeed, market research by any other name is still science.
Is market research going to die? No. Is your job going to die? Maybe. Especially if you don’t keep up with the newest scientific methods. So go learn SAS and SQL and R. Go learn and your career in market research will never die. Your title might change but not your career.
–Written on the go
I have a problem. Anytime I think of the word platykurtic, I immediately think of this cartoon. I saw it many times as a youngster and for some reason always got a big kick out it. Now, the fact that platykurtic kind of sounds like FLATykurtic which describes the flat nature of a platykurtic normal curve makes it easy for me to remember that a platykurtic curve is a flatykurtic curve. I think that’s pretty clear so enjoy the cartoon yourself.
More cartoons from my childhood. Simon in the land of chalk drawings was another TV show I was allowed to watch. I still love the sing and you just might catch me singing it.
I have a number of favorite interview tests, and this is yet another one. Here’s the game: I draw two different charts, just like what you see here,except without the labels on the axes. One chart has a very slow rise and the other has a very steep rise. Then, I ask my poor interviewee which chart they would like to represent the raises they will receive from year to year.