Say it ain’t so! Banish average scores? Where is that coming from?
- Average scores fail to explain the whole picture. Here’s an example. Let’s say the Blackberry generates an average score of 4 out of 5, a moderately positive score. And, let’s say that the iPhone generates an average score of 4 out of 5, the same score. It would seem that people like the Blackberry just as much as the iPhone. (NOOOOOOO!) But wait… did you forget to look at standard deviations or box scores? What if you learned that the Blackberry received 80% neutral scores, 10% positive scores, and 10% negative scores. And what if the iPhone received 40% neutral scores, 30% positive scores, and 30% negative scores. Those are two very different love/hate stories. But you wouldn’t know it from the average score. Those two bar charts you see represent the exact same mean but oh so different box scores.
- Trends never change. What is the biggest joke researchers make about tracking studies? That nothing ever changes. Indeed, from day to day and week to week, the numbers are always the same. There’s almost no reason to even run a tracker. The average score has been 3.466364 since last week, since before the internet, since before the beginning of time. Since we’re working with a scale from 1 to 5, instead of a scale from 1 to 100, we don’t even give ourselves the chance to see if something has changed. Why do we even bother? Because we were told to run a tracker and never change the questions or formatting even if the market warrants revisions. Wow. Great research objective.
But wait. For some reason, I can’t let average scores go.
- Big trends scream. In the social media space, where we get to use sample sizes in the thousands and millions, serious opinion changes make for beautiful charts. Sure, the number has been 3.2 for 2 years now. But all of a sudden it went to 3.25 and then 3.3 and then 3.6. I can even pin down the exact day that some unknown event took place and shook up the market before returning opinions to normal. Box scores do this too, but you’ve got three lines to check instead of just one and, well, maybe I’m just a little lazy.
The moral of the story is this. If you’re going to use average scores, you absolutely must use them in concert with a measure of distribution, whether a standard deviation or box score. And, you must consider whether you are using a scale that is wide enough to actually let you see any changes that might truly exist. Otherwise, I’ll tell you now. Your score next week will be 3.355235263.
- Really Simple Statistics: What is a standard deviation?
- Radical Market Research Idea #6: Don’t calculate p-values
- Radical Market Research Idea #1: Banish probability sampling
- SMART Social Media Research Bootcamp Videos