It’s quite simple really.
Vendors have been over-stating the advantages and under-stating the disadvantages.
Is demographic data available for social media data. YES! Absolutely! We’ve got tons of demographic data! Come buy our data! Of course, you don’t realize until you get your hands in the data that only a tiny percentage of verbatims actually have demos, and it’s only age or gender or region. Forget income, education, household size, religion, race, and all the other demographics you are used to seeing in survey research. But did you ask your vendor what percentage of verbatims had demographic data? Probably not. But you shouldn’t have had to.
Is there data for my brand? YES! Absolutely! What was your brand again? Some tiny, obscure brand that has a generic name like Target, Gap, or Apple? No problem! Here’s your 5 million records of which only 1000 actually reference your brand. Of course, you won’t know that until you get your hands in the data and realize it’s mostly garbage. Should that have happened? Absolutely not.
Is the sentiment scoring accurate? YES! Absolutely! We ran some tests to prove that our data is scored 99.9999% accurately, especially when we delete all the data we aren’t sure about so you aren’t actually getting to see all your data. Of course, once you get your hands in the data, you will find big, huge chunks of data that are scored completely wrong. Should it be a surprise to you that much of your data was deleted or scored incorrectly? Most definitely not.
You know, this is probably no different than getting a new Barbie or a new Teenage Mutant Ninja Turtle for your birthday. When a bright and shiny new toy is placed in front of us, it’s hard to see the negatives. It’s pretty, it looks fun, and we just want to play with it. It’s not until we rip off the wrapper and remove the marketing curtain that we truly see for ourselves what’s going on. Barbie’s lipstick is on crooked, she’s missing a shoe, and unlike the commercial, not a single one of the Turtles can jump 50 feet in the air without the aid of a firecracker.
Perhaps this is a call to action. If you find yourself in the trough of disillusionment, ask yourself why. Were you eager to believe something that was too good to be true? Did you use your market research skills to thoroughly probe and analyze and critique the messages you heard when you were in the process of buying listening research? Did you fall for marketing hype over good sense? Did your vendor make false promises? Sigh. False promises.
Related articles
- Social Media Research and the Trough of Disillusionment (Ray Poynter)
- A Halloween chocolate lesson in nominal, ordinal, interval, and ratio data #MRX (lovestats.wordpress.com)
- Gamification of Surveys In The Real World #MRX (lovestats.wordpress.com)
- The 6 Worst Market Research Mistakes #MRX (lovestats.wordpress.com)
The LoveStats Blog 
Why would you want to apply offline survey demographics in an environment in which behaviours and motivations are uniquely different? One of the foundations of the social web is that it allows people to create interest-based communities that are based on more personal motivations than income, education, household size, religion and race. Preference and perspective are how we need to segment a social audience, not old-world demographics. See Charles Leadbeater’s work on systempathy here about this new paradigm – http://www.slideshare.net/ibbt/charles-leadbeater-the-challenges-of-innovation
.
I think the point is that we’re so used to seeing demographics on every datapoint, that we are lost when we don’t see that data. We need to get away from that style of thinking. Fortunately, over time, more and more people are making their online demographics available which means we are now better able to create online demographic profiles. Same and different.
[…] Why Social Media Listening Research Hasn’t Lived Up to the Hype – Speaking of hype, Annie Pettit of Conversition lets you know why social media market research hasn’t lived up to it. […]
Your points are correct. But one shouldn’t rely entirely on tools. Tools are one thing, then it’s about how we use them and the methodology that we build upon. And then I know for a fact that’s there are solutions for circumvent these limitations.
Agree. Tools can never be the sole solution. The only solution is the human brain applied to data. Tools get you a few steps closer but only if they are properly and honestly explained.
Thanks, Annie. False promises are one thing but, importantly, is it good research?
Absolutely. Let’s start with no false promises and then move on to quality data, followed up with knowledgeable and realistic analysis. Couldn’t be an easier. 🙂