Tag Archives: honesty

Why social media listening research hasn’t lived up to the hype #MRX

Tales of the Teenage Mutant Ninja Turtles #3. ...

It’s quite simple really.

Vendors have been over-stating the advantages and under-stating the disadvantages.

Is demographic data available for social media data. YES! Absolutely!  We’ve got tons of demographic data! Come buy our data! Of course, you don’t realize until you get your hands in the data that only a tiny percentage of verbatims actually have demos, and it’s only age or gender or region. Forget income, education, household size, religion, race, and all the other demographics you are used to seeing in survey research. But did you ask your vendor what percentage of verbatims had demographic data? Probably not. But you shouldn’t have had to.

Is there data for my brand? YES! Absolutely! What was your brand again? Some tiny, obscure brand that has a generic name like Target, Gap, or Apple? No problem! Here’s your 5 million records of which only 1000 actually reference your brand. Of course, you won’t know that until you get your hands in the data and realize it’s mostly garbage. Should that have happened? Absolutely not.

Is the sentiment scoring accurate? YES! Absolutely! We ran some tests to prove that our data is scored 99.9999% accurately, especially when we delete all the data we aren’t sure about so you aren’t actually getting to see all your data. Of course, once you get your hands in the data, you will find big, huge chunks of data that are scored completely wrong. Should it be a surprise to you that much of your data was deleted or scored incorrectly? Most definitely not.

You know, this is probably no different than getting a new Barbie or a new Teenage Mutant Ninja Turtle for your birthday. When a bright and shiny new toy is placed in front of us, it’s hard to see the negatives. It’s pretty, it looks fun, and we just want to play with it. It’s not until we rip off the wrapper and remove the marketing curtain that we  truly see for ourselves what’s going on. Barbie’s lipstick is on crooked, she’s missing a shoe, and unlike the commercial, not a single one of the Turtles can jump 50 feet in the air without the aid of a firecracker.

Perhaps this is a call to action. If you find yourself in the trough of disillusionment, ask yourself why. Were you eager to believe something that was too good to be true? Did you use your market research skills to thoroughly probe and analyze and critique the messages you heard when you were in the process of buying listening research? Did you fall for marketing hype over good sense? Did your vendor make false promises? Sigh. False promises.

Do not disturb (but only when it’s convenient for you)

Skyline of Toronto

Image via Wikipedia

With the Toronto Mayoral election upon me today and two major candidates fighting for the lead (George Smitherman and Rob Ford), I came to appreciate just how important our Do Not Call list is. This is a list people can add their phone number to which tells telemarketers not to call them. If a telemarketer phones a number on the list, there will be penalties for the company. Market researchers, social researchers, and political pollsters have been granted special permissions which allow them to call numbers on that list without penalty.

And that brings me to this past weekend and the impending political vote. I received over ten phone calls from people “hoping to count on my support.” A couple of calls were real people but most were automated calls that droned on and on filling up my answering machine. A couple of calls were negative campaigns but most seemed to be professional. I have to say most because I turned down the volume of my phone and stopped listening to them once I realized what was happening.

I have full respect for the political process and urge everyone to vote (or spoil their ballot) in every election that they are eligible to vote in. But boy, did I take a second ponder at that Do Not Call list. If I didn’t put my name on the list, how much phone spam would I receive?

Which brings me to my real point. Social media research of course. Jeffrey Henning of Vovici tweeted me this weekend to ask my opinion about groups that are password protected but can be instantly accessed by anyone as soon as they create a password. I immediately thought of the Do Not Call list.

Both systems are examples of lists that are easy to sign up for and easy to ignore. I CAN call someone on a Do Not Call list. I know they don’t want to talk to me but their phone number isn’t physically broken. I CAN sign up for a password to a forum and receive instant access but I know the password means this information is not for public consumption. I CAN ignore people’s polite attempts to tell me that I’m not wanted but I can usually take a hint.

Just because you CAN doesn’t mean you should. What do you think?

Researchers do it… with ethics

If we can, for just a few minutes, let’s put aside how this discussion came to be so top of mind and focus on an underlying issue: observational research.

Observational research involves watching how people behave in their natural environment with no attempt to alter or guide it. It could include watching families interact with their children at the park, watching how people choose soap at the grocery store, watching how people choose a parking spot, or a whole host of other normal behaviours that take place in the public space where others would be able to see them.

Normally, in observational research, only a few people are observed. A few families, a few shoppers, a few drivers. The researcher takes notes, perhaps on their iPad, concerning things like how many people were there, did they speak loudly, did they pick up five different brands, did they back in and out several times.

Some people might feel uncomfortable knowing that they were a part of the research though chances are they would never know of it. Is this deceitful? Perhaps. Is it unethical. Perhaps. It’s a topic that many psychologists and sociologists have grappled with for over a hundred years. With exceptions, the agreed upon answer is that this type of research is ethical.

The greater issue comes down to scale. What if we watched millions of people at once? What if we had access to their names? This, of course, is social media research. People have decided to share intimate details of their lives in a public places (not password protected) where others can view the information and often their personal contact information as well. It’s observational research on a practically unimaginable scale.

Social media research isn’t going to go away. If market researchers don’t do it with ethics, validity, and reliability, other will do it without ethics, without respect, without honesty, without validity, without reliability.

Personally, I can’t tell you if it’s right or wrong. I can tell you that my heart doesn’t ache when I do the work. I bring respect for others to my work, an appreciation for ethics, a desire for solid research practices. I know I’m contributing to a process that brings better products to market without putting people through an arduous traditional research process.

You can’t control the behaviours of other people, but you can make sure that you sleep well at night. I do.

%d bloggers like this: