Radical Market Research Idea #3: Insist on quantity over quality #MRX


Wait, was that a typo? Quantity over quality? Well, I meant what I said.

Question #1: What was the sample size of your last tracker? 30 per time frame? 50 per time frame? What about your last custom study? 300? 500?

Question #2: How many pages of questions and demos and cross-tabs did you flip through searching for any chi-square or t-test that was statistically significant? 100? 200?

Here’s the problem. We run ridiculously long surveys with far too few participants per test cell and we are ok with searching through far too many Type 2 errors.

Here’s the solution. Put your money into large sample sizes and not into question topic after question topic. Focus on sample sizes within demographic groups rather than questions with 4 or 8 people per cell. Trade variety of questions for reliability of results. Trade overly long surveys for properly sampled cells. Trade breadth of topics for validity of individual questions. Take money away from more and more questions and put it directly into more and more validity and reliability. Radical.

Please comment below. What was the sample size of your last study and what was the sample size within many of the cells?

2 responses

  1. it’s about time. sample size can solve a lot of problems.
    we need shorter questionaires and bigger samples

  2. Totally agree on length. Not so on the sampling stuff – although definitely true from a purist POV….. I think that insights coming from all over the place in companies will simply overrun MR professionals if we’re not nimble and pragmatic. I dislike the dichotomoy quality vs quantity. We need to do more research, it needs to be high quality but reliability and validity aren’t the be all and end all of every single study. Our voices need to be heard often, with authority, but sometimes simply saying – small sample, highlight potential risk, and remain part of the conversation rather than saying “that or nothing”. Pragmatic.

%d bloggers like this: