Just a few days ago, I moderated a webinar with four leading researchers and statisticians to discuss the use of margin of error with non-probability samples. To a lot of people, that sounds like a pretty boring topic. Really, who wants to listen to 45 minutes of people arguing about the appropriateness of a statistic?
Who, you ask? Well, more than 600 marketing researchers, social researchers, and pollsters registered for that webinar. That’s as many people who would attend a large conference about far more exciting things like using Oculus Rift and the Apple Watch for marketing research purposes. What this tells me is that there is a lot of quiet grumbling going on.
I didn’t realize how contentious the issue was until I started looking for panelists. My goal was to include 4 or 5 very senior level statisticians with extensive experience using margin of error on either the academic or business side. As I approached great candidate after great candidate, a theme quickly arose among those who weren’t already booked for the same time-slot – the issue was too contentious to discuss in such a public forum. Clearly, this was a topic that had to be brought out into the open.
The margin of error was designed to be used when generalizing results from probability samples to the population. The point of contention is that a large proportion of marketing research, and even polling research, is not conducted with probability samples. Probability samples are theoretical – it is generally impossible to create a sampling frame that includes every single member of a population and it is impossible to force every randomly selected person to participate. Beyond that, the volume of non-sampling errors that are guaranteed to enter the process, from poorly designed questions to overly lengthy complicated surveys to poorly trained interviewers, mean that non-sampling errors could have an even greater negative impact than sampling errors do.
Any reasonably competent statistician can calculate the margin of error with numerous decimal places and attach it to any study. But that doesn’t make it right. That doesn’t make the study more valid. That doesn’t eliminate the potentially misleading effects of leading questions and skip logic errors. The margin of error, a single number, has erroneously come to embody the entire system and processes related to the quality of a study. Which it cannot do.
In spite of these issues, the media continue to demand that Margin of Error be reported. Even when it’s inappropriate and even when it’s insufficient. So to the media, I make this simple request.
Stop insisting that polling and marketing research results include the margin of error.
Sometimes, the best measure of the quality of research is how transparent your vendor is when they describe their research methodology, and the strengths and weaknesses associated with it.
Well, crap happens and today was my turn. I had scheduled a webinar with five fantastic panelists to discuss using margin of error with convenience samples. Hundreds of people registered. Many people personally emailed me to say how they were excited about listening in. And what happened?
I’ve used the webinar software many times before with no problem but still took the morning to practice several times. The panelists and I all logged in early to make sure our sound was good and clear. Everyone was ready.
And when the webinar started, the audience heard not a word. I couldn’t type in the question box. I couldn’t type in the chat box. Nothing. I had to cancel the webinar.
Upon returning to my desk, I found over a hundred emails from people wondering if the webinar had been cancelled on them, or if the technical issue was on their side or my side. They just didn’t know what was happening. Clearly, an apology was in order.
Unfortunately, our webinar experienced technical difficulties today. We are working carefully to resolve the issue and will reschedule it as soon as possible. Thank you for your patience and we apologize for the inconvenience.
This is a very typical templated letter that any company might send so no worries there. It’s very formal and official, But boy, is it ever impersonal. And it just doesn’t sound like me. Here is what I actually wrote.
Thanks for checking in with me. I feel so terrible right now. We did a complete sound check ahead of time and the sound was perfect. Once the webinar went live, we lost all the sound and couldn’t figure out how to fix it.
We are going to reschedule and I’ll email you in case you’d like to give us a second chance.
I’m so sorry to waste your time today.
As you can see, the two letters are very, very different. The letter I wrote came from a human being. Me. I truly did (and still do) feel awful about disappointing hundreds of people. I didn’t know what the issue was but explained what I did know at the time. I know some people were inconvenienced. I know some people were annoyed.
Now, to be clear, I did copy and paste this message as a template. But I personally opened and read every single email. And I personally addressed each email. And I felt bad while I replied to each email.
And you know what happened? Many people emailed me back to say they were relieved to know that the technical issues weren’t theirs. Many people said they hoped the webinar would be rescheduled and that they were looking forward to it all the same. Empathy poured through.
I have to think that letting me come out in the email, rather than using a templated email, showed people that I cared. Perhaps treating each person who emailed me as human being relieved some of the annoyance.
It’s something to think about the next time you’re tasked with using a templated email. Maybe it’s time to drop the formal corporate talk and just say what’s in your heart.
(I’m still working on rescheduling the webinar and will leave a link here when I have it. Finding a time when five professionals can get together is tough!
**Here is the link to register for the re-scheduled webinar. THANK YOU for all the kind messages.)
All good things must come to an end and so to has WAPOR. But, today was still a full day of sessions so here are a few of my take aways.
- Anything other than Margin of Error is too hard to understand. AAPOR, and by association, WAPOR have been having a rough time lately with discussions, rants, arguments or whatever you want to call them regarding margin of error. In today’s business meeting, someone mentioned that using anything other than margin of error is just too difficult to understand. Well you know what, margin of error is difficult to understand yet we’re still all on the same page. The fact that it, or any other measure, is difficult to understand is absolutely no excuse. We aren’t stupid. Journalists are stupid. Pollsters aren’t stupid. Let’s find a measure that works, that makes sense, and start using it. I don’t care how ‘difficult’ it is.
- Should you debrief after observational research? It’s probably been a hundred years now that various North American associations have agreed that observational research does not require researchers to inform the people they observe. As long as the researchers do not interfere, don’t talk to, don’t manipulate, don’t affect the people around them, don’t sneak around, don’t hide, don’t misconstrue, they are free to listen and observe what people are saying and doing in public spaces. No permission required. But, I learned today that academics in Germany must debrief people afterwards. I’m having a hard time wrapping my head around this one. Chances are that at some point in the future, most if not all research organizations are going to have extremely similar ethics codes. How will we reconcile this one?
- Newspapers are the most trustworthy. I found this kind of humorous. Newspapers first, TV second, social media is further down the line. It kind of made me think that the longer it takes to take the news and make it public, the more likely people are to trust it. Hence, since daily newspapers generally take about 24 hours to turn news into the medium, there’s a lot of time to determine where an error was made and refrain from printing it. Television makes somewhat of an effort to broadcast news as quickly as possible but even they take some time. If an item doesn’t make “live, breaking news”, it still has to wait until 6pm or 10pm to be shared widely. Again, lots of time to discover and error and correct it. But this blog post? I could have written it the very second I heard each tidbit (and I normally do) which means I could have misheard or mistyped something without realizing it. Hit “submit” and that false news is out there.
- Peanut Labs Ask-Me-Anything with special guest Jim Bryson (web.peanutlabs.com)
- WAPOR Day 2: People don’t lie on government surveys #AAPOR #MRX (lovestats.wordpress.com)
- WAPOR Day 1: Why are Google, Facebook, and Microsoft so far ahead of us in research? #MRX (lovestats.wordpress.com)
- America: You aren’t representative of the world #MRX #WAPOR (lovestats.wordpress.com)
Toronto is in the midst of a heated contest with two major mayoral candidates, Rob Ford and George Smitherman. Which means we’re going to hear a lot of numbers being thrown at us. So here is a quick and easy guide to what those numbers mean.
In this case, a poll means that a bunch of people have been asked who they will vote for, perhaps hundreds or thousands of people. Maybe you find out that 30% plan to vote for one person and 35% plan to vote for the other.
When you ask only a few hundred people who they will vote for, you have a bigger chance of making a mistake than if you ask thousands. Pollers call that chance of mistake the margin of error. I call it the jiggle factor.
So, the 30% and 35% are jiggly numbers. If you only ask a few hundred people in your poll, those numbers will jiggle a lot. If you ask thousands of people, the numbers won’t jiggle a lot.
With a margin of error, or jiggle factor, of 3 points, the 30% might jiggle as low as 27% or as high as 33%. And, the 35% might jiggle as high as 38% or as low as 32%.
See how we applied the jiggle factor(margin of error) to both numbers? It means that the low number is actually somewhere between 27 and 33 and the high number is actually somewhere between 32 and 38. Notice that those two sets of numbers overlap on 32 and 33. This tells you that the 30% and the 35% are not different from each other and that our two candidates are in a dead heat.
Even though the one number is 5 points larger than the other number, they really are equal to each other. The important part is to apply the jiggle to both numbers.
In the end, the only way to know if they are different is to vote. So vote!
(It’s a lot more complicated than this, but hopefully the general idea is clear.)