The power of cognitive interviewing and what qualitative research can learn from Behavioral Economics by Gina Henderson #Qual360 #QRCA
Live blogging from the Qual360 conference in Toronto, Canada. Any errors or bad jokes are my own.
- People don’t always do what they way they are going to do
- motivations and needs are only half the picture, we need to understand behaviour more holistically
- BE has made popular th eideas that people do not believe rationally, intuition and emotion play a significant role in our behaviors, choices are affected by context
- our choices are endless, consumers don’t give the effort to consider every possible choice, they consider what is available as opposed to what they really want
- how do qual and BE relate? people don’t say what they mean or mean what they say, behaviour is driven by unconscious, words are poor tools, intuition and emotion are important – that why we use projective techniques
- this is still only part of the story
- BE says behaviour is automatic and unthinking, satisfying vs maximizing, real choices of what people actually do, contemporary, adaptive unconscious, capable of learning complex information better and faster, use behaviour as an entry point
- Qual says choices are outcome of enduring needs preferences and beliefs, ideal solutions, feelings, perceptions, attitudes, psychoanalytical view, raw, untamed, use meaning as an entry point
- Consumers are ok with good enough because they don’t have time to make the extra effort for the ideal solution
- diaries and ethnographies help us learn about current behaviours
- Cognitive interviewing – from 1970′s, police used it a lot to get as close to the actual experience as possible, this is what qual researchers want also
- Some people believe memory failure is a failure of retrieval, we just have to know the right codes to find it, triggers could be a memory, image, smell, taste, sound, emotion, location – like smelling cookies and remembering gramma
- Don’t ask why – consumers will give you an answer but they can’t recall in a meaningful way, it’s not conscious
- experiment – students rated jams the exact same way as jam experts ranked them, until the students were asked WHY, then the ratings were all different
- Goal of cognitive interviewing is to recreate the context – anchor them in time, find out what else was going on at that time, what time of the year was it, ask them about the building, the atmosphere, then ask who was there, who did you talk to, what did you talk about. Can let responder meander, they don’t need to stay on topic, don’t interrupt them, allow a freeflowing conversation, a lot of silence is okay
- Horlicks case study – a milk additive – why did people stop using the product – learned about user habits, the environments they were in, context of using product, where the product was available, learned about whether the product could be soy or dairy
- BE is not the answer to everything, it’s another tool
- Peanut Labs Ask-Me-Anything with special guest Tom Ewing
- Peanut Labs Ask-Me-Anything with special guest Kristin Luck
- What is Vue magazine?
- Innovation for insights into the Millennial Moms’ Online Shopping by Annie Iverson and John Williamson #Qual360
- Designing with packaging usability in mind by Claudia Del Lucchese #Qual360
- Behavioural Economics Can Finally Explain Human Behaviour
- What is a convenience sample?
Well, once you manage to catch your breath after laughing solid for 4 minutes, let’s really think about all the people involved in this little prank.
1: Interviewer: First of all, this interviewer deserves a raise, a bonus, and a promotion for going through this interview without laughing, getting upset, or antagonizing the survey responder. I’m sure he deals with this sort of thing, whether real or fake, all day long every day. And yet, the utmost professionalism on his part. Kudos for a great job.
2: Responder: How did our industry get to such a state where surveys are written so poorly that people leave a tape recorder at their telephone waiting for researchers to call in order to make fun of them? This is nothing for us to be proud of.
3: Data Analyst: How exactly is the data analyst going to handle data which is clearly horrible quality? Will the analyst think of checking for outliers in each question? Will the analyst review the entire set of responses to recognize that it is an across the board outlier and probably a troublemaker? Will these responses lead to completely invalid analysis and conclusions?
4: Survey Author: Of course, we understand the need to use standardized questions in surveys. But, no matter how convinced you are, the world does not consist of people who know how surveys work. There are absolutely people out there who need to be taken through a survey with far more care than what we
permit when writing surveys. Telephone surveys need to be written so that interviewers can speak naturally and help those people who actually need some help. That’s where good data comes from. I’m really curious if the survey author left a place for the interviewer to indicate that this instance was possibly an outlier.
So, enjoy. But the next time you write a survey, keep this in mind. Are you antagonizing yet another survey responder or are you responsible for creating a more positive market research experience?
I’ve been asked on a number of occasions how I found a job in market research after completing a Phd in psychology. You’d think the opportunities are endless but, like any career path, there are always obstacles.
Interviewers have told me to my face that they refuse to hire or they don’t like Phds. They’ve even given me strange tests to determine whether I’m human or robot, like “What does this abstract painting on my wall mean to you?” (Honestly, that piece of crap is major ugly.)
So here is my advice. It’s free advice so the confidence intervals are wide. Please do ask questions and I promise I’ll answer all of them.
- The fact that you have a Phd means you know research and statistics. Don’t waste your cover letter proving this.
- The fact that you have a Phd means employers think you don’t know the real world and that you can’t speak casual english. Prove this wrong in your cover letter. Write in business dialect not in dissertation dialect. This is one case where fancy words do NOT impress.
- Forget the stats speak. When they ask you what a t-test is, don’t tell them it’s an analysis of mean scores and confidence intervals of a quantitative variable for a second qualitative variable. Speak english. Say it’s a way to determine if two groups of people differ on a measure like height or weight.
- Join a few online survey panels so you can get a lay of the land. What questions get asked? How are they asked? Do you want to shoot yourself during the survey because it’s so horrid? This will give you insight about the business you think you want to get into and…
- … something to talk about during interviews. In which you will speak like a human being not a professor.
- Go to a used bookstore and buy a Market Research 101 textbook. Learn it.
- If your field isn’t psychology, you would do well to take a course in social psychology or personality psychology. It will give you great insight into survey question design.
- Learn either SAS or SQL. Not the menu driven kind, the syntax programming type. Even if you don’t end up using it on the job, you will be better able to talk to the statisticians and get what you need in the time you need.
- Accept that in the business world, projects take 14 days not 14 months, with sample sizes of 200 not 2000, and conclusions that are final not proposals for 8 more years of research. Now is not the time to try to convince your gracious hosts otherwise. They aren’t stupid.
You are already qualified. You just need the right vocabulary and the right perspective on research for business. Any questions?
Read these too
After asking someone what a t-test is and whether they watch American Idol, I have another question I like to ask interviewees to see if they know their stuff. I ask them to create a chart that they would present to a client. Ahhh, the variety of answers this reveals.
1) Some folks tell me that this version of Excel doesn’t do charts but they can do it on their version. (Honestly. This one astounds me.)
2) Others create a chart. That’s it. Nothing else. When I re-prompt that it’s to give to a client, I still get a default chart in default colors with a default no title and default lack of zero on the axis.
3) A few create a chart and try to pick the right chart, try to put a title, and try to fix the axis.
Right away, I can see that Person #1 has weak problem solving skills and great excuse skills. There are multiple ways to chart in excel and if you can’t get one of them to work, I can’t help you. Thanks for coming out.
Person #2 pays no attention to detail and barely wants to get the job done. Thanks for coming out.
Person #3 is my person. Whether they choose the right options, I know they are curious and want to do the right thing. If they don’t know what the right thing is, I can teach them and they can learn.
If I ever get to ask you this question, please make sure your chart has a non-misleading scale, a good title, non-florescent colours, and isn’t 3D. My sincere thanks.
Read these too
Is research an art or a science? Personally, I think it’s both. Actually, I think it’s a magical combination of the two. So, when I’m hiring a new researcher, I have a few standard tests that help me determine whether I’ve got an artist, a scientist, or a magician in front of me. Let me share one of my tests with you.
I like to ask people to consider that I know nothing about research and statistics and then describe to me what a t-test is. I tend to get one of three answers.
1) A 500 word technically stunning and accurate monologue incorporating as many statistical terms that can be thought of in the time allotted.
2) A series of ums and awws followed by an admission that they didn’t so well in statistics but they think it has something to do with numbers.
3) A round about discussion about how the average guy is taller than the average girl.
The first answer is the scientist. Even after I prod and give examples, they still can’t find their voice. I know this person knows the answer, but I wouldn’t dare ask them to write a report, answer an email, or talk to a client. Imagine the motion sickness that client would experience!
The second answer is the artist. Even though the jobs ad clearly said “knows statistics,” this person has creatively managed to work around such an unimportant detail. They attempt to stun me with a selection of phrasing in hopes that some combination of words will mean something.
The third answer is the magician. They have been gifted with that rare skill of actually doing what they’ve been asked to do. They have proven to me that they know the answer AND I know they will be able to talk to clients. They are a magical combination of scientist and artist. And then, only then, I ask my real interview question, “Are you addicted to American Idol?”
Would you pass my test?
Read these too