I hate social media research because: It’s not a rep sample #2 #MRX


I recently wrote a blog post citing ten of the biggest complaints about social media research. Today I address complaint #2.

It’s not a representative sample.

Part 1. Are we really going to go there? I guess we ought to. In 99.9% of market research, we aren’t using a representative sample in the strict sense of the word. Survey panels aren’t probability samples. Focus groups aren’t probability samples. Market research generally uses convenience samples and social media research is no different.

But here is the difference. We’ve all heard the statistic that a tiny percentage of people answer the majority of all market research surveys. In other words, most people aren’t participating in the survey experience and we never hear their opinion. Similarly, when we conduct social media research, we only listen to people who wish to share their opinions on Facebook, Twitter, YouTube, or any of the other millions of websites where they can write out their opinions. No matter what research method you choose, you only hear the people who wish to contribute their opinion in that mode.

Part 2. Who is talking about the brand anyways? Alright, so we know SMR doesn’t use rep samples. Big deal. One of the reasons we use rep samples in traditional research is to ensure we are talking to the right people. We do a rep sample because a product is used by a rep sample. We do a male only sample because a product is used by males only. In both cases, we choose a particular sample because it is most likely to reflect product triers and users. Guess what. The only people talking about your brand in social media are the people who care about your brand. Whether they hate your brand or love your brand, you have instantly reached the people who are relevant to your brand. They have raised their hand to tell you, “Listen to me. I have an opinion about your brand.”

If you require a rep sample, you ought to use a survey because that is the closest approximation. Always use the right method for the job.

True or False: True, but does it matter?

8 responses

  1. I am completely puzzled by your claims of lack of bi-modality or skewed data in social media research. Your data sources are clearly social medial collected, so you would not have the equivalent data from the general population to conclude you do not have bi-modal data. GIGO here I think. A biased sample will,by definition potentially hide bi-modality or sample skews. Just because you do not see it in your data doesn’t mean it isn’t there. you need to benchmark against a more representative sample. Actually logic will tell you that you are almost certainly looking at skewed data simply by definition, you get those most voluble and those most critical. And probably the youngest, the least moneyed, etc.

    1. I’ll be first to agree that social media data is not 100% representative of a census population. No argument there. I’ll also be first to agree that there is no research method that is 100% representative of a census population. Forcing volunteers into census designed demographic boxes does not make volunteered data representative of unvolunteered data. Are some methods better than others? Absolutely. In the end, you need to choose the method that best suits the specific need of a specific research project.

      But when it comes to bi-modality of data, this is something I have never seen in social media data, even after looking at thousands of datasets representing different brands, categories, people, things, ideas etc. Most people assume that social media data is stuffed with people who either abhor or adore the item under measurement and that simply isn’t the case. The data is indeed (generally) normally distributed around the mean, wherever the mean for that particular brand/item may be.

  2. What this doesn’t adress is that the only people who are talking about your brand is social media are those with a reason to talk about you. This often involves a very negative or very postive experience. The vast marjority of your customers have had neither and their point of view is not reflected in SM discussion.

    1. It’s true that only people who want to talk about you will talk about you. There are lots of brands I know but would never talk about online. However, it isn’t true that social media data is bimodal. In fact, the data is normally distributed. It’s just more fun to pay attention to the really angry and really excited comments. Thanks for sharing your thoughts. 🙂

  3. Harry LaRacuente

    Thus is a nice discussion. It leads to a further question of whether brand lovers versus brand dislikers are equally likely to voice their opinions at the same rate or at differential rates. For example: if those who dislike a brand are twice as likely to “articulate a view”; then a 50 – 50 split in pro versus con messaging is really closer to two to one split. I do not think I’ve seen this addressed.

    1. In my experience, social media sentiment is normally distributed. I’ve run distribution charts of sentiment across many (hundreds) of brands and I’ve not yet seen one that is bi model or anything vastly non normal. Yes skewed positive or skewed negative. Yes leptokurtic and platykurtic. But nothing bimodel. Thanks for sharing!

  4. Annie, I think the reason we use (or try to use) probability samples is that we want to be able to generalize to a given population with a known margin of sampling error. The issue with non-probability samples, to me, comes when it’s time to draw conclusions that extend beyond the people you actually talked to/heard from. That said, your point that we only hear from people who want to give us their opinion is a very good one. Even probability sample designs suffer from a kind of self-selection bias in that we can’t FORCE people who are randomly chosen to participate. They will only participate if they see value in doing so.

    1. Yup, as much as we like to think surveys use probability samples, that’s just not the case. Surveys generally use very fancy convenience sampling with lots of corrections and approximations. Not that it’s bad but we do not recognize the truth.

%d bloggers like this: