Weighting is a very common process in the research process. You might even use it every single time you run a study. I’m going to go against the grain here and challenge you to think about it more carefully.
Let’s look at a simple example. Let’s take a data set that is made of 40% men and 60% women. The men produced an average score of 38% and the women an average score of 48%. That gives a raw score of 44%. But, because the population is 50% male and 50% female, we want to weight the results back to that target. That gives us a weighted score of 43%. So, the raw score is 44% and the weight score is 43%. Is it really all that different? Does that really change the business decision? The answer to this question should be “Absolutely not because my confidence interval is 3 points.” Makes sense, doesn’t it. If the raw score is basically equal to the weighted score, what are you doing weighting data?
Now, i’m not saying don’t weight your data. I’m just saying think twice before you weight your data. UNDERSTAND how weighting works before you use it. Here are some thoughts in relation to weighting:
1) Do not expect your scores to change very much.
2) If your scores are changing a lot, your sample is too different from the population and your weighted scores are probably not very reliable. You probably have tiny sample sizes that should be thrown out, not weighted.
3) If your scores aren’t changing very much, why are you weighting? Data varies and comes with confidence intervals. You’re probably just shifting the score around within it’s confidence interval. So why bother.
4) If you are using weighting, do not weight because you didn’t get enough of a particular demographic group. Weight because one group was too large.
Moral of the story: Use the largest sample you can afford, and pull it so that it will be as representative as possible when you are done.