Tag Archives: standard deviation

5, 7, and 9 Point Scales: Do You See The Difference? #MRX

It’s a highly debated question with quantitative data to support all sides. Are 5 point, 7 point, or 9 point scales better suited for generating quality data? Sure, the distribution of responses is slightly different in each case and your ability to conduct more complex statistical analyses can be improved. But I have a few very basic arguments all of which lead me to support scales with  fewer items.

  1. Scales with more points create differences where differences do not exist. Sure, I understand. You want to measure tiny differences. But do consumers REALLY see the difference between 5 and 6 on your 9 point scale? When it comes most ordinary products for most ordinary people, the answer is probably no. Soap is soap and butter is butter and only the brand manager sees the difference.
  2. Let’s assume, however, that people DO see the difference between 5 and 6 because people take the utmost interest in every product that ever existed. And let’s assume you’ve prepared an extremely comprehensive survey with a multitude of grids measuring a multitude of dimensions.  What’s better – 50 grid items using 5 point scales or 50 grid items using 9 point scales.  Those 9 point scales are creating nearly twice as much respondent fatigue and an entire group of people is now even less likely to answer the next survey you so carefully prepared.
  3. How does your analysis plan incorporate those extra scale points? Are you going to provide an average score and standard deviation? If that’s the case, then tell me, how is 3 out of 5 any different than 6 out of 10? I’ll tell you. It’s not. All you’ve done is given yourself a bigger number to work with. And chances are your analysis is incorporating extra error simply because of extra responder fatigue and a higher drop-out rate.

It’s quite simple. Stick with 5 point scales. You’ll generate just as many data points with lots of variability and your survey responders will thank you for it.

Banish average scores! #MRX

Say it ain’t so! Banish average scores? Where is that coming from?

  1. Average scores fail to explain the whole picture. Here’s an example. Let’s say the Blackberry generates an average score of 4 out of 5, a moderately positive score. And, let’s say that the iPhone generates an average score of 4 out of 5, the same score. It would seem that people like the Blackberry just as much as the iPhone. (NOOOOOOO!) But wait… did you forget to look at standard deviations or box scores? What if you learned that the Blackberry received 80% neutral scores, 10% positive scores, and 10% negative scores. And what if the iPhone received 40% neutral scores, 30% positive scores, and 30% negative scores. Those are two very different love/hate stories. But you wouldn’t know it from the average score. Those two bar charts you see represent the exact same mean but oh so different box scores.
  2. Trends never change. What is the biggest joke researchers make about tracking studies? That nothing ever changes. Indeed, from day to day and week to week, the numbers are always the same. There’s almost no reason to even run a tracker. The average score has been 3.466364 since last week, since before the internet, since before the beginning of time. Since we’re working with a scale from 1 to 5, instead of a scale from 1 to 100, we don’t even give ourselves the chance to see if something has changed. Why do we even bother? Because we were told to run a tracker and never change the questions or formatting even if the market warrants revisions. Wow. Great research objective.

But wait. For some reason, I can’t let average scores go.

  1. Big trends scream. In the social media space, where we get to use sample sizes in the thousands and millions, serious opinion changes make for beautiful charts. Sure, the number has been 3.2 for 2 years now. But all of a sudden it went to 3.25 and then 3.3 and then 3.6. I can even pin down the exact day that some unknown event took place and shook up the market before returning opinions to normal. Box scores do this too, but you’ve got three lines to check instead of just one and, well, maybe I’m just a little lazy.

The moral of the story is this. If you’re going to use average scores, you absolutely must use them in concert with a measure of distribution, whether a standard deviation or box score. And, you must consider whether you are using a scale that is wide enough to actually let you see any changes that might truly exist. Otherwise, I’ll tell you now. Your score next week will be 3.355235263.

Really Simple Statistics: What is a standard deviation? #MRX

really simple statisticsWelcome to Really Simple Statistics (RSS). There are lots of places online where you can ponder over the minute details of complicated equations but very few places that make statistics understandable to everyone. I won’t explain exceptions to the rule or special cases here. Let’s just get comfortable with the fundamentals.

** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **

Standard deviations are massively popular in all aspects of market research reporting. Any time someone tells you an average number, they’ll probably tell you  what the standard deviation is at the same time, even if you didn’t ask for it. At it’s most basic level, a standard deviation is a number that tells you how similar a set of numbers is.

For now though, let’s forget about all the technical language and think about a casual application. In your immediate family, most of the women are probably similar to each other in terms of their height. If your mom is 5 foot 3, chances are that many other women in your family are somewhere around 5 foot 3, and in fact most of them are probably within an inch or two of 5 foot 3. The “normal” woman is about 5 foot 3 and there is very little differentiation or deviation among the heights. The deviation is small.

On the other hand, get out the wooden ruler you’ve saved since public school, the one with your 4ever true love engraved on it, and hold it up to their hair. Some of the women have really long hair, others have shoulder length hair, while still others have short and snazzy hair. There’s a lot of  differentiation, a lot of disagreement, a lot of deviation in their hair lengths. Sure the average or normal length might be 8 inches, but the deviation from the norm could easily be 8 inches. The deviation is large.

In the market research space, you can look at standard deviations in a similar way. It can be interpreted as the amount of disagreement among people’s opinions. Let’s consider 100 answers to a purchase intent question asked on a five point scale from Definitely Will Buy all the way to Definitely Will Not Buy.

  1. If 50 people answered definitely will buy and 50 answered definitely will NOT buy, that’s a big difference among the answers, a lot of disagreement, a lot of differentiation. Half of the people are checking off the 5 and half of the people are checking off the 1. People haven’t come to any consensus on whether they agree or disagree. In technical words, that clear disagreement indicates a wide or large standard deviation. These wide standard deviations make our work as market researchers more difficult. It’s hard to recommend a new product when people can’t agree on whether they would buy it.
  2. But, if 90 people answered definitely will buy and 10 people answered probably will buy, there’s a lot of agreement there. 90% of the people are checking off the 5 and 10% of people are checking off the 4. People are generally agreeing with each other. They pretty much all intent to buy though some are a little more sure about that purchase than others are. That agreement reflects, inversely, very little differentiation, very little disagreement. It indicates a very narrow or small standard deviation. This is what market researchers love to see. We have a clear answer to our question and can proceed to recommend a product that most people would like to buy.

So here’s the general scoop:

  • Small standard deviation = Lots of agreement among the opinions
  • Large standard deviation = Lots of disagreement among the opinions

It’s that simple!

Really Simple Statistics: Nominal Ordinal Interval and Ratio Numbers #MRX

Welcome to Really Simple Statistics (RSS). There are lots of places online where you can ponder over the minute details of complicated equations but very few places that make statistics understandable to everyone. I won’t explain exceptions to the rule or special cases here. Let’s just get comfortable with the fundamentals.

What? There are different kinds of Numbers?

In statistics, the type of number you use determines the type of statistic you can use. Learn these and you’ll have an easier time deciding what statistic makes more sense to use. There are four basic types of numbers that we consider in statistics.

What are Nominal Numbers?

Nominal numbers make the least sense because they aren’t really numbers. Nominal numbers are simply numbers that are different. 1 is not 2. 3 is not 9. It really makes more sense to think of things like apples and oranges, or cookies with green sprinkles vs cookies with red sprinkles. There is no reason to assign apples to the number 1 or 3 nor does it make any sense to assign oranges to the number 2 or 9. We just assign numbers to things because it makes doing statistics and creating charts easier. It’s like a check all that apply question on a survey.

What are Ordinal Numbers?

With ordinal numbers, we have a little bit more information about the numbers. When we use ordinal numbers, we know that one of the numbers is bigger than another number. We know that 2 is bigger than 1, and 7 is bigger than 3. And it works the other way too. 1 is smaller than 2 and 3 is smaller than 7. We know which number is bigger, we just don’t know by how much bigger. One cookie is simply bigger than the other cookie. And I’ll have the bigger one. Like you could even yank it out of my hand. These types of numbers show up when we use Likert scale questions on a survey.

What are Interval Numbers?

Now let’s add in another piece of information. Interval numbers tell us everything we learned above, AND they tell about the spacing between the numbers. For instance, the amount of space between 1 and 2 is the same as the amount of space between 6 and 7. Or, the difference between 1 and 2 cookies is the same as the difference between 2 and 3 cookies. The difference in both cases is exactly one cookie. My cookie.

What are Ratio Numbers?

And lastly, this is where we thank Muhammad ibn Ahmad al-Khwarizmi. Ratio numbers incorporate the number zero. Now we know which number is bigger, and we know how much bigger, and we also know how to create none of it. This would be a survey question where you ask people to make sure their numbers add up to 100%. But I don’t dare illustrate what zero cookies looks like. The shock of it might kill me.

Data Tables: The scourge of falsely significant results #MRX

Histogram of sepal widths for Iris versicolor ...

Image via Wikipedia

Who doesn’t have fond thoughts of 300 page data tabulation reports! Page after page of crosstab after crosstab, significance test after significance test. Oh, it’s a wonderful thing and it’s easy as pie (mmmm…. pie) to run your fingers down the rows and columns to identify all of the differences that are significant. This one is different from B, D, F, and G. That one is different from D, E, and H. Oh, the abundance of surprise findings! Continue reading →

%d bloggers like this: