“Does Sample Size Still Matter?”
David Bakken, Chief Operating Officer, KJT Group, Inc. and Megan Bond
- Healthcare research is often forced into very small sample sizes
- How do you determine how many interviews should be conducted for a health product?
- Level of precision
- Best guess of current market share
- Sampling error is the primary source of uncertainty, as you collect more data, error approaches zero; uncertainty is expressed as long-term frequencies
- [whoop! missed an entire section here because I was paying close attention 🙂 ]
- People routinely use samples that are too small even though they know they shouldn’t
- 30 used to be the rule of thumb for subsample sizes
- We normally draw probability samples from non-probability frames
- Classic paradigm
- start with a population with known parameter values
- draw samples
- estimate sample parameters
- compare samples to population
- (you can even do this with fake monte carlo populations)
- With a population of 800, they tested samples of 25 to 450.
- 25 had unusually higher variance but seemed to level off around samples of 250 [but 250 is a huge sample of 800]
- Smaller samples have more extreme errors when it comes to measures like max and min values
- Sample size does still matter
- Things get a lot better when samples are greater than 100
- Bayes rules help reduce uncertainty in small samples it requires a change in our thinking
Welcome to Really Simple Statistics (RSS). There are lots of places online where you can ponder over the minute details of complicated equations but very few places that make statistics understandable to everyone. I won’t explain exceptions to the rule or special cases here. Let’s just get comfortable with the fundamentals.
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
I’m going to guess that the #1 question every researcher has when they start a research project is this: How many people do I need to measure? If you want a simple answer, then you need to measure 1000 people per group. Unfortunately, that’s not the answer most people like. Unlike the hope the post title gave you, there really isn’t a simple answer.
Do you plan to look at subgroups?
If you plan to split out subgroups in your data, then you need to make sure each group will have a large enough sample size. Do you plan to compare men and women? Do you want to see if older people generate different results than younger people? Are you comparing TV commercial #1 with TV commercial #2?
If you only have the budget to measure 100 people but you plan to split that group into people aged 18 to 34 and 35+, and then by gender, then you will only have 25 people in each group. That simply isn’t enough to be sure that the results you find are more likely to be real than due to chance. Every single group you look at needs to have a large enough sample size to ensure the results aren’t due to chance. And if that means each of your 15 groups needs to have at least 100 people each, then you’ll need to increase your budget or decrease the number of groups you look at.
How big of a difference are you expecting?
If you think an important difference between your groups will be small, then you will need a large sample. For instance, if you’re testing the effectiveness of a health and wellness campaign, any small difference will make a big improvement in people’s lives. You don’t care if the improvement is small, perhaps an increase in effectiveness of 1% or 2%. You care that 1% or 2% of people are doing better. We know pure chance can easily give us numbers that are 2% different. To try to counter random chance, we need to use very large sample sizes. Hundreds or thousands is probably the more appropriate number.
And vice versa – if you think an important difference will be big, then you can get away with a smaller sample. Perhaps you’re testing a new scent of air freshener. Really, you don’t care if 1% of people like it more than the existing scents. You only care if 10% or 15% of people like it more than the existing scent. It’s much harder for random chance to create sets of numbers that are 10% different so this time, we don’t need to use such a large sample size. You might be able to get away with just a couple of hundred.
Are you measuring once or several times?
If you are measuring something more than once, perhaps tracking it on a weekly or monthly basis, each sample size can be smaller. For instance, you might determine that a one time measure should be 300 but a weekly measure need only be 100 per week for 6 weeks. As before, it’s hard for random chance to produce similar results every single week for 6 weeks so we don’t need as large a sample size each time.
If you’re looking for some specific direction, then check out this list of statistical calculators. Be prepared for some very technical terms though!
Wait, was that a typo? Quantity over quality? Well, I meant what I said.
Question #1: What was the sample size of your last tracker? 30 per time frame? 50 per time frame? What about your last custom study? 300? 500?
Question #2: How many pages of questions and demos and cross-tabs did you flip through searching for any chi-square or t-test that was statistically significant? 100? 200?
Here’s the problem. We run ridiculously long surveys with far too few participants per test cell and we are ok with searching through far too many Type 2 errors.
Here’s the solution. Put your money into large sample sizes and not into question topic after question topic. Focus on sample sizes within demographic groups rather than questions with 4 or 8 people per cell. Trade variety of questions for reliability of results. Trade overly long surveys for properly sampled cells. Trade breadth of topics for validity of individual questions. Take money away from more and more questions and put it directly into more and more validity and reliability. Radical.
Please comment below. What was the sample size of your last study and what was the sample size within many of the cells?
- How to encourage speeding in your surveys
- Merry Christmas to all and to all a good sample size #MRX (lovestats.wordpress.com)
- Big Data? Big Deal. #MRX (lovestats.wordpress.com)
- New book! The Listen Lady: A novel and social media research guide baked into one #MRX (lovestats.wordpress.com)
Everybody deserves a little recognition once in a while. Many of us do good deeds on a regular basis and are never recognized for those accomplishments. Today, however, is your chance to shine.
Do you qualify for any of these awards? Then be proud and claim them! Copy the link and post the awards on your website. But be honest. Falsely claiming awards will result in your name being published on Santa Clause’s naughty list. Just sayin.
Proper English, proper attire, proper format. Everything needs to be proper. At the recent Research Live social media research conference in London, I kept hearing a phrase that sounded something like this: “we didn’t do proper research.”
So what is proper research? Is it test and control groups? Large sample sizes? Quantitative data and probability based statistics? Surveys and randomized groups? It made me wonder what was improper about the research, why their research was so bad that it needed special caveats.
Research isn’t test-tubes and skinner boxes. Research is a living, breathing process by which people discover relationships among variables. Research comes in all forms from qual to quant, from covert to overt, from 1 group to 51 groups, from surveys to n=1 case studies, from exploratory to confirmatory.
The only form of proper research I know is this: a scientific approach to discovering answers to questions. This process involves knowing the risks, the pros and cons, the biases and skews of the method you’ve chosen. It involves knowing how to handle the resulting data properly, without preconceived biases, without expectancy effects. Proper research isn’t a specific type of research. It’s simply smart research.
If you want to be an excellent market researcher, you need to know a lot about many different topics. You need to know what makes a good survey question or focus group discussion guide and how to avoid writing a horrid one. You need to know about research methods, sampling, weighting, and sample size determination. Knowledge of statistics is essential and it must go beyond t-tests, chi-squares, and p-values. There is a ton of very detailed, complicated information you must know to do your job well.
But here is the problem. When people ask for research advice, they don’t always want an essay on the pros and cons of various options and techniques. They know they’re asking a complicated question with a complicated answer but sometimes they just want a quick and simple answer. They want to know that they’re pointing in the right direction, that they’re generally thinking the right thing.
So what do we do? We don’t try to understand whether it’s a request for a simple answer or an in-depth consultation. No matter what they’re looking for, we give people a three hour lecture about the intricacies of research and make everything far more complicated than it needs to be. Our strange technical languages serves to scare off some people and bore others to tears.
Isn’t it time we considered what people really want? Perhaps just a simple answer to a simple question?
- Surprise, surprise! A non-rep sample is as good as a ‘rep’ sample (lovestats.wordpress.com)
- This is why your research sucks (lovestats.wordpress.com)
- Really Simple Statistics: 1-Tail and 2-Tail tests #MRX (lovestats.wordpress.com)
- Really Simple Statistics: Chi-Square (lovestats.wordpress.com)
You might not want to admit it, but at one time or another, you were probably on a team responsible for some research that sucked. Wonder why? Let me help you out.
1) You didn’t have a trained, experienced researcher at the helm.
Researchers are not a luxury component of research projects. Researchers know what makes a quality, unbiased, nonleading, useful questionnaire and focus group. They know what the most appropriate sample sizes are and WHY those are the most appropriate sample sizes. They know which statistics are the right ones and WHY those are the right ones. Researchers know how to take a problem and funnel it into a measurable, valid, and generalizable project.
2) You failed to identify and follow through on specific objectives.
There are two places where it is essential to focus on your objectives. First, when designing your research, you need to have a problem to solve or a reason to do the research. Without a problem, you could write a 400 question survey and still be trying to add more. Second, you need to focus once you get your data. Most surveys result in 300 page data-tables which are completely overwhelming, even for great researchers. Without focus and silo-vision, you will never find an answer. You can search but you will not find.
3) You focused on price and speed rather than quality and quantity.
Sure, you can choose the research with the best price and speed. But validity and reliability depend on sample sizes, and data cleaning, and appropriate statistical testing, and quality research design. These things are not quick nor cheap but when you need to accurately predict future sales or which TV show will be canceled or which product test will succeed, this is how you must do it.
4) You don’t follow through on the results.
Lots of really great research actually does get done, in large and small companies, via surveys and focus groups and social media research. But research is just crap if you don’t follow through on the results. If you KNOW you aren’t going to follow through on a set of findings, don’t bloat your survey and fatigue your respondents with it. If you KNOW you won’t have the time to follow through on the results for six months, DON’T do the research for six months. Research in a drawer is money in the toilet. Or, you could just give that money directly to me. Paypal accepted.
[tweetmeme source=”lovestats” only_single=false]I really hate standards.
I am far too independent and far too concerned about doing the right thing to like it when someone tries to tell me what to do. When it comes to carrying out market research, I’m irked when I’m told to follow standards and annoyed when I’m required to meet specifications. So if standards aren’t for me, then who are they for?
Market research standards are for
• people who believe in the letter of the law, not the spirit of the law
• new researchers to learn about basic and broad important features
• research buyers who don’t yet understand the nuances of every market research method and don’t always know if they’re buying fluff or substance
• small research companies that need backing to grow in the marketplace
• big research companies that might gradually ride the slippery slope
• every researcher and research company to learn from and ponder about
I guess that means research standards are for me.
Read these too