Surprise, surprise! A non-rep sample is as good as a ‘rep’ sample

Image via Wikipedia

A new study called Re -Examining the Validity of Different Survey Modes for Measuring Public Opinion in the U.S. by Brian Schaffner and Stephen Ansolabehere made a stunning discovery that non-representative samples accurately predict the market place. Why is this surprising?

Market research has never had the good fortune to use rep samples. Not everyone signs up for, let alone knows about, online survey panels. Mail surveys only go to people who have homes and often skip large apartment buildings. Telephone surveys, even random digit dial version, don’t give every person an equal and independent chance of participating.

Market research has always been about non-rep samples. That’s the nature of our business. We thrive on turning non-rep data into rep conclusions. So stop being surprised and start being proud. It’s what we do.

Related articles

• Fake Science: What market research works to avoid #MRX (lovestats.wordpress.com)
• #MRXblogs – It’s new and it’s fabulous #MRX #NewMR #NGMR (lovestats.wordpress.com)
• I’m a Lion in the Market Research Zoo #MRX (lovestats.wordpress.com)
• This is why your research sucks (lovestats.wordpress.com)
• 21 Market Research Tips (blogs.sitepoint.com)

This is why your research sucks #MRX

You might not want to admit it, but at one time or another, you were probably on a team responsible for some research that sucked. Wonder why? Let me help you out.

alvimann from morguefile

1) You didn’t have a trained, experienced researcher at the helm.
Researchers are not a luxury component of research projects. Researchers know what makes a quality, unbiased, nonleading, useful questionnaire and focus group. They know what the most appropriate sample sizes are and WHY those are the most appropriate sample sizes. They know which statistics are the right ones and WHY those are the right ones. Researchers know how to take a problem and funnel it into a measurable, valid, and generalizable project.

2) You failed to identify and follow through on specific objectives.
There are two places where it is essential to focus on your objectives. First, when designing your research, you need to have a problem to solve or a reason to do the research. Without a problem, you could write a 400 question survey and still be trying to add more. Second, you need to focus once you get your data. Most surveys result in 300 page data-tables which are completely overwhelming, even for great researchers. Without focus and silo-vision, you will never find an answer. You can search but you will not find.

3) You focused on price and speed rather than quality and quantity.
Sure, you can choose the research with the best price and speed. But validity and reliability depend on sample sizes, and data cleaning, and appropriate statistical testing, and quality research design. These things are not quick nor cheap but when you need to accurately predict future sales or which TV show will be canceled or which product test will succeed, this is how you must do it.

4) You don’t follow through on the results.
Lots of really great research actually does get done, in large and small companies, via surveys and focus groups and social media research. But research is just crap if you don’t follow through on the results. If you KNOW you aren’t going to follow through on a set of findings, don’t bloat your survey and fatigue your respondents with it. If you KNOW you won’t have the time to follow through on the results for six months, DON’T do the research for six months. Research in a drawer is money in the toilet. Or, you could just give that money directly to me. Paypal accepted.

Related Articles

• Advertising to Real People – Including Researchers #MRX (lovestats.wordpress.com)
• Assume the Worst of that Survey
• Really Simple Statistics: Mean, Median, Mode #MRX
• The Death Of Market Research
• Pick One: Pretty or Pretty Misleading #MRX

Fake Science: What market research works to avoid #MRX

Within the market research community, we have ongoing discussions about whether market research is irrelevant and useless, usually because someone knowledgeable about the field has discovered a poor quality study. Focus groups aren’t scientific, survey panels aren’t probability samples, social media research isn’t representative. To clarify, a well-planned focus group can be scientific, surveys panels will never be probability samples, and social media research will be somewhat representative at some point. In every case, though, the distinguishing feature is that no method is perfect. But, when you put the methods in the hands of an experienced, high quality researcher, you will be sure to get the best possible outcome.

There is no dispute that market research is not a 100% scientific method. But we can easily avoid the quality of scientific research that went into the research shared on “Fake Science.” It’s a shame to hear about the candy corn – that is a favourite of mine.

Related Articles

• What is Online Research Sample? (questionpro.com)
• Laugh at yourself and then cry at our flailing industry
• To zero or not to zero, that is the question
  
• Cronbach’s Alpha, My Favourite Statistic
• Why do surveys ask the same question 8 billion times?

Proof that sampling social media data is critical, but you already knew that

There are lots of tools on the internet to gather consumer opinions. OpinionAided is a new one that’s quick to set up and quick to get results from. The results are absolutely unscientific but hey, everyone needs a little fun now and then.
.
Let’s have our own fun as researchers and see how the lack of any rules around sampling affects the results. First, answer the three PollDaddy questions here. Then, read below to see what the answers were to the same question on OpinionAided. Same? Different? This, of course, is a highly scientific and valid methodology. Oh, just have a little fun!

.

.

.

.

.

.

Results from OpinionAided

Samples around 150 each
Facebook 39% concerned
Twitter 17% concerned
YouTube 21% concerned

Read these too

Building a bad reputation before we even start: Privacy in social media research

2011 Market Research Unpredictions #MRX

Six things that terrify me in market research #mrx

To Mid-Point or Not To Mid-Point, That is the Question

Where is the privacy fallout?

StatsSuck. That is all.

Cover via Amazon

At the recent ARF audience measurement conference in New York, a couple of controversial statistical ideas were raised. Controversial in the sense that people reading my tweets couldn’t tell if I believed the idea or not.

1) The point was made that we should forget the 95% significance value and focus instead on 80%. I do agree that some people get so hung up on that 95% that they fail to see the forest for the trees. We need to understand the theory of statistics so that we know when it makes sense to go against them. As always, once you know why you’re breaking the rules, it’s ok to break them. I see 80% as a good theory building, hypothesis testing, do I bother to keep trying number. And then, 95% is a good confirmatory test. But with human discretion applied.

2) We focus a lot of our energies on trying to build the most accurate samples we possibly can, split by many demographics and complicated sampling strategies. But the problem is that we know we can never achieve that perfect sample. Ever. So let’s approach this from a different point of view. Acknowledge the flaws in a sample, and be wary of and smart abut the weaknesses they bring to the results. If you want to achieve new heights, curious outcomes, and innovation, simply press on. Innovation comes from taking risks. Working with a less than perfect sample just might create a situation for innovation.

I dare you.

Read these too

Why market researchers can never be marketers

Survey Design Tip #3: Do You Encourage Straightlining?

Laugh at yourself and then cry at our flailing industry

In Honor of Infographics. #MRX

This is why Twitter will die