Just a few days ago, I moderated a webinar with four leading researchers and statisticians to discuss the use of margin of error with non-probability samples. To a lot of people, that sounds like a pretty boring topic. Really, who wants to listen to 45 minutes of people arguing about the appropriateness of a statistic?
Who, you ask? Well, more than 600 marketing researchers, social researchers, and pollsters registered for that webinar. That’s as many people who would attend a large conference about far more exciting things like using Oculus Rift and the Apple Watch for marketing research purposes. What this tells me is that there is a lot of quiet grumbling going on.
I didn’t realize how contentious the issue was until I started looking for panelists. My goal was to include 4 or 5 very senior level statisticians with extensive experience using margin of error on either the academic or business side. As I approached great candidate after great candidate, a theme quickly arose among those who weren’t already booked for the same time-slot – the issue was too contentious to discuss in such a public forum. Clearly, this was a topic that had to be brought out into the open.
The margin of error was designed to be used when generalizing results from probability samples to the population. The point of contention is that a large proportion of marketing research, and even polling research, is not conducted with probability samples. Probability samples are theoretical – it is generally impossible to create a sampling frame that includes every single member of a population and it is impossible to force every randomly selected person to participate. Beyond that, the volume of non-sampling errors that are guaranteed to enter the process, from poorly designed questions to overly lengthy complicated surveys to poorly trained interviewers, mean that non-sampling errors could have an even greater negative impact than sampling errors do.
Any reasonably competent statistician can calculate the margin of error with numerous decimal places and attach it to any study. But that doesn’t make it right. That doesn’t make the study more valid. That doesn’t eliminate the potentially misleading effects of leading questions and skip logic errors. The margin of error, a single number, has erroneously come to embody the entire system and processes related to the quality of a study. Which it cannot do.
In spite of these issues, the media continue to demand that Margin of Error be reported. Even when it’s inappropriate and even when it’s insufficient. So to the media, I make this simple request.
Stop insisting that polling and marketing research results include the margin of error.
Sometimes, the best measure of the quality of research is how transparent your vendor is when they describe their research methodology, and the strengths and weaknesses associated with it.
Welcome to Really Simple Statistics (RSS). There are lots of places online where you can ponder over the minute details of complicated equations but very few places that make statistics understandable to everyone. I won’t explain exceptions to the rule or special cases here. Let’s just get comfortable with the fundamentals.
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
What is sampling error? First, you need to understand what sampling is. Sampling is choosing a smaller set of data/people/things to reflect the entire population. For instance, instead of measuring the height of everyone in your office, you might just measure the height of ten people. Or, instead of asking every person in Canada who they intend to vote for, you choose a sample of 2000 people to ask.
In the process of sampling, you gather 10 heights instead of 100 heights, or you gather 100 opinions instead of 1000 opinions. Either way, you don’t gather every possible data point and that means the summary numbers you generate will probably not be exactly the same had you measured every data point. The process of sampling introduces error and it cannot be avoided.
In addition to sampling error, most research studies are affected by other errors that also take place during the sampling process. This includes coverage errors, non-response errors, self-selection errors, and more. Consider these obvious sampling biases:
- The ten tallest people in your office were away at a “Retreat for tall people” and you didn’t wait to include them in your height sample.
- The ten Asian people in your office were away at a “Retreat for Asian people” and therefore couldn’t be part of your height sample (hm…. aren’t Asian people know for being shorter than average?”
- When you were gathering opinions on voting intentions, you only asked people who were attending a gala for a particular political candidate
Running a survey and you’re positive your sampling plan is perfect?
- Does everyone have a telephone in order to respond to your telephone survey?
- Does everyone have a home where they can receive a mail survey?
- Does everyone have a computer where they can receive an email survey?
Running social media research and you’re positive your sampling plan is perfect?
- Does everyone feel comfortable leaving comments on blogs?
- Does everyone have a public facebook page?
- Does everyone use Twitter?
Of course, these are the obvious errors taking place during the sampling process. Tiny mistakes are always made in the sampling process, particularly when you must first decide from where to gather opinions. The trick is to ALWAYS assume that your sampling plan includes error.
- Really Simple Statistics: T-Tests
- Really Simple Statistics: p values
- Really Simple Statistics: Nominal Ordinal Interval and Ratio Numbers
- Really Simple Statistics: What is Ratio Data
- Really Simple Statistics: What is Ordinal Data?
- Really Simple Statistics: What is Nominal Data?
- Really Simple Statistics: What is Interval Data?
[tweetmeme source=”lovestats” only_single=false]As promised on Twitter, here is the final chapter of my short book on probability sampling. 🙂
For those of us in the real world, where probability sampling is the impossible dream (see previous posts if you’re puzzled why I say this), what are we to do? What’s the point of doing any serious marketing research if we can’t even meet the most basic statistical requirements required for predictive analytics?
Let’s see. We’ve been doing heavy duty market research for at least 50 years now. We’ve been dang good at predicting the success of products and the failure of politicians. And, acknowledging the 5% of predictions we got wrong because we know there is always unpredictable error, we’ve been doing a dang good job WITHOUT absolutely perfect probability samples.
What? How is that possible? Predicting the future without using probability sampling? Of course it’s possible. Even if researchers can’t sample perfectly, we know how to sample really well with what we’ve got. We’ve been learning this skill for decades. We learn the idealized theory in school, we learn the practical theory on the job, and we tinker around with all the tidbits at our disposal to put together a pretty fine sampling job at the end.
We know how to identify flaws in data. We know how to clean data. We know how to write quality surveys (whether we always do is another story). We know how to interpret ambiguous data. We know our stuff.
We know our stuff so well that we are able to predict the future even when glitches creep into our research process.
So tell me again. Why all the failed efforts to prove we use probability sampling when we’ve taught ourselves to work smart with what we’ve actually got?
This rant has been brought to you by the letter P and the number 5.
Read these too
[tweetmeme source=”lovestats” only_single=false]Ok, I give. You’ve got a study that actually uses probability sampling. By some magical slight of hand, you’ve identified every person in your desired population. Perhaps your population is the immediate members of your family, cancer doctors in your medical clinic or survey panel members.
You’ve managed to apply a random sampling method that gives each person an equal opportunity to participate. Maybe you picked numbers out of a hat. Maybe you used one of those books with 500 pages of random numbers.
You’ve managed to apply a process that gives every person an independent opportunity to participate. For this argument, let’s just assume that survey panels don’t kick certain people out because a housemate has also been selected.
Fine. You have a probability sample. You have covered off random error.
But folks, we aren’t in the business of hypothetical research. We make money from actual marketing research. Real people, real studies, real every day work. In my world, we just don’t do many of those one in a million studies that are capable of employing a reasonable semblance of probability sampling. Random error is not the whole picture.
Why does it seem like we always forget about non-random error? What about the vast majority of research that has 90% opt-out rates? Do we decide that those people weren’t part of the population to begin with? Does the lack of random error make non-random error ok?
I’m just having a hard time understanding the ongoing push to prove we are using probability samples when there remain other uneaten slices of the error pie.
Read these too
I completely disagree with my own title!
My world is online. I started out writing online surveys in 1996 when I bugged the computer helpdesk at my graduate school to set me up with an online database. No one else at the university had ever done such a thing and i confused the heck out of them. I wrote my own html code which allowed me to specify font sizes, font colours, page colours, radio buttons, check boxes and text boxes. ooooooo….. so sophisticated. I’d be embarrassed to tell a scripter now that “I write my own code.”
Online research has never tried to say it uses probability sampling but, other methods of research have. There has been a debate over the last year specifically directed at online panels. Well, not really a debate. Some folks have been outraged that online panels do not use probability sampling and therefore they do not qualify to use statistics. To go even further, they suggest that telephone samples do use probability sampling and so results from that type of research are the most valid.
Let me offer up some ideas…
Telephone research – Do you always answer your phone? Is your phone number unlisted? Do you return phone calls? Do you politely tell telephone interviewers that you are busy when in fact you are nursing a bag of cheetos?
Mail research – Do you just throw out all the junk you get in the mail? Do you fill out surveys AND mail them?
Online research – Are you signed up for an online survey panel? Do you click on the survey banners that appear after you run a search and then finish every survey?
It seems to me that no matter how hard you try to use probability sampling, human beings just cannot cooperate. We’re not worms or mice or molecules. People choose when they wish to pay attention or participate. It’s not online panels. It’s research with human participants.
Probability sampling of people? No such thing.