I just returned from two of the best marketing research conferences out there, ESOMAR and WAPOR, and was flipping through the notebook of rants and raves that I create as I listen to speakers. Interestingly, even at these conferences, where the best of the best speak, I heard a certain phrase repeatedly.
“The regression model indicated…”
“The data indicated…”
“The results indicated…”
Well you know what? The data indicated absolutely nothing. Zip. Zilch. Zero.
Data is data. Numbers in a table. Points in a chart. Pretty diagrams and statistical output.
The only thing that indicated anything is you. YOU looked at the data and the statistical output and interpreted it based on your limited or extensive skills, knowledge, and experience. If I were to review your data, My skills, knowledge, and experience might say that it indicates something completely different.
Data are objective and indicate nothing. Take responsibility for your own interpretations.
In quite a few presentations over the last few days I’ve heard people make claims like
– my regression showed that women are more likely to…
– my factor analysis showed there are three groups…
Big problem. Tests prove nothing. Statistical tests give you an idea of the likelihood a bunch of numbers have a distribution due to chance. That’s it. Numbers. Pretty numbers.
The interpretation you make of the numbers, well that’s a completely different story. Everyone will interpret data just a little bit differently. Significant or not.
What affects survey responses?
– color of the page
– images on the page
– wording choices
– question length
– survey length
– scale choices
All of these options, plus about infinity more, mean that confidence intervals and point estimates from any research are pointless.
And yet, we spew out out significance testing at every possible opportunity. You know, when sample sizes are in the thousands, even the tiniest of differences are statistically significant. Even meaningless differences. Even differences caused by the page colour not the question content.
So enough with the stat testing. If you don’t talk about effect sizes and meaningfulness , then I’m not interested.
Nearly every day, I see a really cool statistic on TV or the interweeb. Everyone gets all excited about losing 312 pounds in four days, curing cancer, or eliminating measles forever. Candy is good for you! Coffee increases your memory! Drink more wine! Eat more Doritos! But if we paid ANY attention to the research methodology, you’d ignore the entire study. Here are a few of the biggest problems I see.
1) Significantly increased memory!!! Yes, when the sample size is large enough or the difference is large enough, anything is significant. So if 5 people in the control group remembered 5 things and 5 people in the test group remembered 8 things, the difference might be statistically significant. Or, if 1000 people in the control remembered 5 things and 1000 people in the test group remembered 5.2 things, the difference might be statistically signficant. Do you trust the results based on 10 people? Do you care about a difference of 0.1 points? I don’t. Get back to me when your sample sizes and effect sizes go beyond pre-test methodology sizes.
2) Cancer rates decreased by 75%!!! Yes, very nice finding. Especially when the cancer rate of the control group was 0.04% and the cancer rate of the test group was 0.01%. That is indeed a 75% decrease but will that massive decline of 0.03 points mean that you stop eating chocolate or start drinking wine? Doubt it. It’s not a meaningful difference when it comes to one single person. Get back to me when the rate decreases by 75% AND the base rate can be measured without any decimal places.
3) Chocolate makes you thin!!!! I’m sure it did. In that one, single study. That has never been replicated. Remember how we compare all our findings against a 5% chance rate? Well, that’s what you just discovered. The 5% chance where the finding occurred randomly. Run the research another 19 times and then get back to me when 19 of them say that chocolate makes you thin.
There are about 423 other cautions to watch out for, but today has been brought to you by the number three.
Feeling a little lonely? Want to be in a relationship? Then join Match.com because you are 3X more likely to find a relationship if you do. Because, clearly, they are a much better program than any other dating program out there. Click on the image to watch their commercial and see for yourself.
But just hearing that “3X more” phrase makes me think:
- Does their system have three times as many people signed up?
- Are they comparing themselves to people who are using a crappy dating service?
- Are they comparing themselves to people who aren’t using a dating service at all?
- Have they considered that people who sign up for a service are more serious about finding a relationship?
The ad implies a causal link but there are so many correlational links that all I can do is completely discount the commercial.
I’m a big fan of dating services but not if they’re going to mess with statistics. Just as nobody puts baby in a corner, NOBODY messes with statistics.
It’s true. I read it in today’s newspaper. The article clearly stated that people are more likely to drown on hot days. Which obviously means that:
- People forget how to swim on hot days
- It’s more difficult to swim on hot days
- It’s harder to hold your breath on hot days
- People are less buoyant on hot days
- No one drowns on cold days
Of course, the article did fail to mention a few things. That people go in the water more often on hot days. That more people go in the water on hot days. Perhaps even that people are less likely to wear life jackets on hot days because “i’m just going in for a second.”
The media is fabulous at taking correlational relationships and presenting them as causation. Don’t be fooled. And wear a life jacket.
I hate vitamins. I hate the big fat pills you have to swallow and I hate the bad taste of the chewable vitamins despite the supposedly wonderful fruity flavours. But this form of self-medication is recommended by doctors and no matter how terrible they taste, I ought to take them.
Fortunately, now there are these lovely things called gummy vitamins for adults. I’ve been staring at them in the stores for a long time. No doubt, they taste fabulous but it disappoints me that they don’t contain all the vitamins that they ought to.
For instance, the gummy vitamins don’t contain any iron. Why not you ask? Well, I checked on the internet (everything on the internet is true) and it seems that the lack of iron is to ensure that should a child find a large bottle of “candy” just sitting around, that they don’t overdose on iron. Makes sense to me.
Well, I finally broke down and bought a bottle. I removed the safety seal and ate the prescribed dose of vitamins. Wow… Yum… They really do taste like candy. I closed the lid and stared at the bottle. Yum. Would two more hurt? Couldn’t possibly. But I shouldn’t. But would two more really matter? Nah. I stared at the bottle for a while longer and finally put it away. That’s why they don’t include iron in the adult gummy vitamins. Not because kids might OD on them, but because adults like me might OD on them.
And so we get to research. Doctors all around the world prescribe research as a valid and reliable method of learning more about consumers and brands. Heck, as a PhD qualified researcher, I prescribe large doses of research for myself all the time. But, if people are going to self-medicate with research, those without the appropriate academic qualifications and without dedicated on-the-job experience and training, need to have the iron removed. The dangerous parts of research should be safely tucked away to prevent harm as much as possible.
Without the appropriate training, DIY tools, like survey programs, sampling systems, statistical analysis and charting programs, should be carefully locked away to prevent surveys from being written incorrectly, samples drawn incorrectly, statistics interpreted incorrectly, and charts prepared incorrectly. If the “candy” can’t be consumed, then it can’t cause any damage.
For the safety and security of brand measurements, are you willing to lock it all up?
- SAS vs SPSS: Pick one and forever hold your peace #MRX (lovestats.wordpress.com)
- SmartyPants Gummy Vitamins Launches True All-In-One For Adults (prweb.com)
- Gummy vitamins business booming for Lowthers (insidehalton.com)
Any serious researcher will fight you to the death to convince you that their favourite stats program is THE best stats program. But, there are good and bad things about each of them.
SPSS: Great for people who rarely use statistics, who don’t remember code, who are scared by code, have no time to learn code, who just need to use some of the basic processes with no special adjustment, who work with normal data that never changes. It’s quick and easy to learn and use, and you don’t 4 feet of manuals to find what you’re looking for. You can just click through the menu to find all the basics you’ll ever need. Compared to SAS, SPSS is major el cheapo.
SAS: Great for people who love coding. SAS does have a menu version but if you like menus, then you should be using SPSS. SAS is great for data manipulation as in creating brand new variables, flipping cases into variables and vice versa, and re-running the same bit of code repeatedly with only a tiny change every time. The macros you can write are unlimited and awesome. And, if it doesn’t do the variation of a statistic you want, you can actually program that statistic into SAS. If you like collecting books, SAS can quickly contribute to that addiction. Buy them. You will need them.
R: What? A third choice? Oh yes. R is great for people who want to flaunt how anti-establishment they are. It’s open source and requires a lot of commitment to become competent. But it can do any statistic you’ve ever dreamed of and a billion more. And you can brag that you know hard core statistics programming. Do not attempt to learn it if Excel intimidates you. Otherwise, go and download it right now and get ready for a rip-roaring awesome 6 week holiday. It is free so prepare to salivate. Mmmmmm…. rrrrrrr. There are even a couple of self-help books now so you might want to find one of them. So, if you’re fresh out of school and no longer have the student version of SAS, R will be perfect for you.
What’s my preference? Well, I wish I was competent in R, but until then, SAS is the best thing since sliced bread.
- The Big Data Myth #MRX (lovestats.wordpress.com)
- The 6 Worst Market Research Mistakes #MRX (lovestats.wordpress.com)
- 6 market research fears that need to go the way of the Hostess Twinkie #MRX (lovestats.wordpress.com)
It’s probably safe to assume that every single research report you’ve ever written has been followed up with a single word – why.
Why did this result happen? Why did people give this answer? Why is this the winning option?
It’s easy enough to read through any report and be faced with lots of interesting questions. I can usually think of three or four contradictory answers for every question coming out of a report. And I can usually make any of them match up with the data. Data in, preferred answer out. Want an insight? I’ll make one up for you.
But which why is the right why? The problem is simple. Market research is rarely designed to answer the question why. Market research is usually designed to measure what. Surveys tell us what. Focus groups tell us what. Social media research tells you what. You see, even when you outright ask people to tell you why, you’re usually getting a why that has been massively skewed by deceiving memories and a variety of life experience. That’s not why.
Most market research is only designed to discover correlations which, I shouldn’t have to tell you, aren’t causation. Just because someone says they buy six cans of beans each week and they have six kids and they tell us they buy six cans because they have six kids does not mean that they buy six cans of beans because they have six kids.
The only way to measure why is with test control research. In the strictest sense, you must randomly create families with random numbers of random children. Randomly assign people to random families such that some of the families are two kid families while others are six or three kid families. Now you’ve got the correct conditions to observe whether families with more kids do indeed buy more beans. And then you’ll legitimately be able to say that having six kids causes families to buy six cans.
So until we can randomly assign people to families, to product offerings, to price differences, to political candidates, and more, we’re stuck with correlational results.
So keep on guessing why.
- Gamification of Surveys In The Real World #MRX (lovestats.wordpress.com)
- Does your market research supplier offer a conjoint product? I hope not. #MRX (lovestats.wordpress.com)
- In defense of research participants #MRX (lovestats.wordpress.com)
- A Cynic Ponders at the AMA Research Summit #AMAresearch #MRX (lovestats.wordpress.com)
Have you ever conducted a research project and NOT done any significance tests?
Have you ever run a series of significance tests and wondered why you bothered to do them?
Let’s think about why we do research projects and why we do significance testing. First of all, research isn’t worth doing unless the methodology is designed very carefully with appropriate sample sizes, great questions, and high standards of data quality. It should be designed with very clear research objectives in mind, with potential outcomes carefully thought out, with potential action steps carefully thought out. Quality research studies are conducted with measures of success clearly outlined before the research is carried out.
If all of these things are in place, then I challenge you to consider why you even bother with significance testing. A research study with clearly thought out objectives should be accompanied by specific hypotheses that lead to specific outcomes. Your well planned out study determined that Product A must generate scores that are at least X% better than Product B before Product A is identified as a success. If it does, then it makes sense to proceed with launching Product A.
So if you already know that you are seeking improvements of size X%, there is zero reason to conduct significance tests. Your measure of success has been predetermined. You already know that, based on your high quality research design, the difference is large enough to warrant moving forward with the launch.
In other words, if you need to run a signficance test to determine if a difference is important, then the difference is for sure not important at all. Significance tests aren’t required.
- Do Google Surveys use Probability Sampling? #MRX #MRMW (lovestats.wordpress.com)
- I hate social media research because: It’s not a rep sample #2 #MRX (lovestats.wordpress.com)
- That’s Me, The New Editor-In-Chief of MRIA’s Vue Magazine #MRX (lovestats.wordpress.com)
- SPSS Releases A New Extraordinary Version #MRX (lovestats.wordpress.com)