I received my research organization’s magazine today. Inside were many lovely articles and beautiful charts and tables. I quickly noticed one particular article because of all the charts it had, but the charts are not what caused my fury.
The article was YET ANOTHER one on panel quality. Yes, random responding, straightlining, red herrings. The same topic we’ve been talking about for years and years and years.
Now, I love panel quality as much as the next person and it is an absolutely essential component for every research panel. We know what the features of low quality are and how to spot them and how to remove their effects. We even know the demographics of low quality responders (Ha! Really? We know the demographics of people who aren’t reading the question they’re answering?) But this isn’t the point.
Why do we measure panel quality? Because the surveys we write are so bad, we turn our valuable participants into zombie. They want to answer honestly but we forget to include all the options. They want to share their opinions but we throw wide and long grids at them. They want to help bring better products to market but we write questions about “purchase experience” and “marketing concepts.”
I don’t want to hear about panel quality anymore. It’s been done to death. Low panel quality is OUR fault.
Tell me instead how you’re improving survey quality. How have you convinced clients that shorter is better and simpler is more meaningful? What specific techniques have you used to improve surveys and still generate useful results? Tell me this and I’ll gaze at you with supreme admiration.
The Mahalanobis Distance
This is a lovely little statistic that folks should take advantage of a lot more. If you’re doing a lot of survey data quality work, you should know about the Mahalanobis Distance (MD). It can help you decide whether a responder is answering questions honestly or not simply by comparing their set of responses to other sets of responses. If someone responds to a series of questions in a way that doesn’t match how other people are responding, then that set of data gets picked out.
So, MD finds people who are straightlining because straightlining is usually not a normal way to respond to data. Of course, finding straightliners is easy to do anyways so who cares. BUT, and far more importantly, this statistic helps to pick out random responders, people who are just haphazardly clicking all over the place. You can find random responders if you read through an individual’s responses, but it takes a lot more personal attention. MD does it automatically, instantly, without all that extra time.
If you’d like to have a read that goes way over most people’s heads, follow this link to “On the generalised distance in statistics” by P.C. Mahalanobis.
Oh, and I have the coolest story to go along with this. I MET THE GREAT NIECE OF MAHALANOBIS HIMSELF!!!! In fact, she and I were coworkers for a short while. I did my best to discover all the amazing details that one might want to know about her celebrity uncle, you know, things like his favourite colour or his favourite brand of ketchup, but she respected his privacy and refused to budge. I did manage to get an AUTOGRAPHED (by my friend Renee) copy of his publication though.
Ok, ok, maybe I shouldn’t get all excited about it, but I am. And if you think that’s just silly, well you’re just jealous.