Tag Archives: interview

Point Counterpoint: Telephone and CATI Interviewing should die #MRX #NewMR 

Market research isn’t straight forward. As much as you believe one thing, I’ll completely disagree and believe another. Which brings us to this series of posts. Over the coming weeks, a colleague and I are going to tackle an important research question. I’m going to make one argument and they’re going to argue the exact opposite.  Do we each have the same opinion, do we each believe in our own argument, are we arguing against our real opinion? You figure it out. Here goes!

Keza Kyanzaire: Telephone interviewing should die

CATI or Computer Assisted Telephone Interviewing Software has been helping market research companies interrupt people during dinner since the dawn of time (the proliferation of the internet). I admit, there are advantages to this form of interviewing, especially considering that when it was originally conceived, the alternatives included going door to door with a clipboard. However, it’s past time to put telephone interviewing to pasture.

According to the CRTC’s 2016 Communications Monitoring Report, more Canadian households exclusively use mobile phones over landline phones, and the shift away from landline phone usage continues to increase. Telephone interviewers are increasingly beginning to reach respondents on mobile phones. Respondents on mobile are often younger, and age can have an adverse effect on the willingness to participate in research and the quality of the data. As well, a respondent reached on a mobile phone may not be in an environment conducive to completing an interview.

Let’s not even mention the fact that telephone interviewers are often mistaken for telemarketers, which comes with its own set of issues. Market researchers have always had to fight to distance themselves from telemarketers in order to establish trust with participants, and legitimacy in the work we do.

Web-based surveys are an effective alternative to telephone interviews. Not only are they a cost saving alternative, (staffing call centres can be extremely costly and they tend to have high turnover rates) they allow the respondent to decide where and when to participate. Particularly in the case of MROC’s or panels, respondents can choose to participate in surveys that are of interest to them and are more engaged and active. Moreover, web-based surveys allow for visual stimuli and more interactive elements that not only make for a better experience for the respondents, but also allows for a depth in the types of questions that can be asked, and data that can be gathered.

Telephone interviewing has been a great resource for the industry, but with the advantages of web-based methods, why are we still running call centres in 2017. It’s past it’s prime and it is time that they are replaced by more effective and advanced methods.

Keza has an Honors Bachelor of Psychology with a Specialization in Cognition. She also completed Humber College’s Research Analyst Postgraduate Program, where she learned how to conduct both social and market research. Keza put her education into practice as a Research Analyst Intern at Numeris, where she conducted statistical analysis and helped inform business decisions. She is now a Researcher at PATTISON Outdoor. And she’s an experienced speaker having taken the stage at #IIeX in Amsterdam. 

Annie Pettit: Telephone interviewing is the best thing since sliced bread

I’m not a gambling person but I’ll bet you know about research panels. Well, you’re in the minority. Most people who are not in the market and social research industry don’t know that panels exist. Which begs the question -who are these people who know about research panels? Are they just people desperately seeking places to earn money? What causes some people to seek out incentive based activities? One thing I know for a fact is these people have internet. They can afford monthly internet charges, maybe even high speed internet charges, as well as the cost of a device to access the internet. Internet might be a right but it still is a luxury for many people. Let’s check our academic privilege at the door. So how do we gather opinions from people who aren’t aware of research panels or couldn’t participate in them even if they wanted to?

One alternative is mall or central location research. Send your best face to face interviewers to the nearest mall and have them randomly interrupt passersby.  Well, first of all, not everyone lives near a location that is large enough to warrant sending an interviewer. Sorry residents of Nunavut, you’ll never be asked to participate in mall research – your town isn’t large enough for your opinion to matter. You live in Vancouver, Calgary, Saskatoon, Montreal, or Halifax? Your chances are pretty good of meeting up with a mall interviewer – unless you’re one of those people who takes a ten minute detour through the parking lot to avoid the person with a clipboard. Besides, anyone who’s taken an introduction to research class knows what accidental sampling or convenience sampling is. Goodbye ability to generalize to a larger population if mall research is your game. 

Which is why I’m a huge fan of telephone research. As it has been since the invention of polls, telephone research continues to lead the fight against self-selection bias. If you’ve never heard of research panels, if you can’t afford high speed internet, if you don’t live in a major city, if you avoid people at the mall, telephone methods still value and seek out your opinions. We are currently living through a time where people don’t trust the polls. Let’s not revert to methods that make it even easier to distrust polls. 

If you’re an early career researcher, data scientist, data visualizer, marketer, or similar, and would like to write a #PointCounterpoint article with me, please send a gmail to my full name anniepettit and let me know what topic interests you. Maybe I’ll pick yours! 


The power of cognitive interviewing and what qualitative research can learn from Behavioral Economics by Gina Henderson #Qual360 #QRCA

Live blogging from the Qual360 conference in Toronto, Canada. Any errors or bad jokes are my own.qual360

The power of cognitive interviewing and what qualitative research can learn from Behavioral Economics 
Gina Henderson, Director, Qualitative Research, TNS
  • People don’t always do what they way they are going to do
  • motivations and needs are only half the picture, we need to understand behaviour more holistically
  • BE has made popular th eideas  that people do not believe rationally, intuition and emotion play a significant role in our behaviors, choices are affected by context
  • our choices are endless, consumers don’t give the effort to consider every possible choice, they consider what is available as opposed to what they really want
  • how do qual and BE relate? people don’t say what they mean or mean what they say, behaviour is driven by unconscious, words are poor tools, intuition and emotion are important – that why we use projective techniques
  • this is still only part of the story
  • BE says behaviour is automatic and unthinking, satisfying vs maximizing, real choices of what people actually do, contemporary, adaptive unconscious, capable of learning complex information better and faster, use behaviour as an entry point
  • Qual says choices are outcome of enduring needs preferences and beliefs, ideal solutions, feelings, perceptions, attitudes, psychoanalytical view, raw, untamed, use meaning as an entry point
  • Consumers are ok with good enough because they don’t have time to make the extra effort for the ideal solution
  • diaries and ethnographies help us learn about current behaviours
  • Cognitive interviewing – from 1970’s, police used it a lot to get as close to the actual experience as possible, this is what qual researchers want also
  • Some people believe memory failure is a failure of retrieval, we just have to know the right codes to find it, triggers could be a memory, image, smell, taste, sound, emotion, location – like smelling cookies and remembering gramma
  • Don’t ask why – consumers will give you an answer but they can’t recall in a meaningful way, it’s not conscious
  • experiment – students rated jams the exact same way as jam experts ranked them, until the students were asked WHY, then the ratings were all different
  • Goal of cognitive interviewing is to recreate the context – anchor them in time, find out what else was going on at that time, what time of the year was it, ask them about the building, the atmosphere, then ask who was there, who did you talk to, what did you talk about. Can let responder meander, they don’t need to stay on topic, don’t interrupt them, allow a freeflowing conversation, a lot of silence is okay
  • Horlicks case study – a milk additive – why did people stop using the product – learned about user habits, the environments they were in, context of using product, where the product was available, learned about whether the product could be soy or dairy
  • BE is not the answer to everything, it’s another tool

Other Posts

Laugh at yourself and then cry at our flailing industry

[tweetmeme source=”lovestats” only_single=false]

Well, once you manage to catch your breath after laughing solid for 4 minutes, let’s really think about all the people involved in this little prank.

1: Interviewer: First of all, this interviewer deserves a raise, a bonus, and a promotion for going through this interview without laughing, getting upset, or antagonizing the survey responder. I’m sure he deals with this sort of thing, whether real or fake, all day long every day. And yet, the utmost professionalism on his part. Kudos for a great job.

2: Responder: How did our industry get to such a state where surveys are written so poorly that people leave a tape recorder at their telephone waiting for researchers to call in order to make fun of them? This is nothing for us to be proud of.

3: Data Analyst: How exactly is the data analyst going to handle data which is clearly horrible quality? Will the analyst think of checking for outliers in each question? Will the analyst review the entire set of responses to recognize that it is an across the board outlier and probably a troublemaker? Will these responses lead to completely invalid analysis and conclusions?

4: Survey Author: Of course, we understand the need to use standardized questions in surveys. But, no matter how convinced you are, the world does not consist of people who know how surveys work. There are absolutely people out there who need to be taken through a survey with far more care than what we

permit when writing surveys. Telephone surveys need to be written so that interviewers can speak naturally and help those people who actually need some help. That’s where good data comes from. I’m really curious if the survey author left a place for the interviewer to indicate that this instance was possibly an outlier.

So, enjoy. But the next time you write a survey, keep this in mind. Are you antagonizing yet another survey responder or are you responsible for creating a more positive market research experience?

From academia to a job: Free advice from a Phd

Werner Erhard and Associates v. Christopher Co...

Image via Wikipedia

[tweetmeme source=”lovestats” only_single=false]I’ve been asked on a number of occasions how I found a job in market research after completing a Phd in psychology. You’d think the opportunities are endless but, like any career path, there are always obstacles.

Interviewers have told me to my face that they refuse to hire or they don’t like Phds. They’ve even given me strange tests to determine whether I’m human or robot, like “What does this abstract painting on my wall mean to you?” (Honestly, that piece of crap is major ugly.)

So here is my advice. It’s free advice so the confidence intervals are wide. Please do ask questions and I promise I’ll answer all of them.

  1. The fact that you have a Phd means you know research and statistics. Don’t waste your cover letter proving this.
  2. The fact that you have a Phd means employers think you don’t know the real world and that you can’t speak casual english. Prove this wrong in your cover letter. Write in business dialect not in dissertation dialect. This is one case where fancy words do NOT impress.
  3. Forget the stats speak. When they ask you what a t-test is, don’t tell them it’s an analysis of mean scores and confidence intervals of a quantitative variable for a second qualitative variable. Speak english. Say it’s a way to determine if two groups of people differ on a measure like height or weight.
  4. Join a few online survey panels so you can get a lay of the land. What questions get asked? How are they asked? Do you want to shoot yourself during the survey because it’s so horrid? This will give you insight about the business you think you want to get into and…
  5. … something to talk about during interviews. In which you will speak like a human being not a professor.
  6. Go to a used bookstore and buy a Market Research 101 textbook. Learn it.
  7. If your field isn’t psychology, you would do well to take a course in social psychology or personality psychology. It will give you great insight into survey question design.
  8. Learn either SAS or SQL. Not the menu driven kind, the syntax programming type. Even if you don’t end up using it on the job, you will be better able to talk to the statisticians and get what you need in the time you need.
  9. Accept that in the business world, projects take 14 days not 14 months, with sample sizes of 200 not 2000, and conclusions that are final not proposals for 8 more years of research. Now is not the time to try to convince your gracious hosts otherwise. They aren’t stupid.

You are already qualified. You just need the right vocabulary and the right perspective on research for business. Any questions?

Read these too

  • Kama Sutra Statistics
  • What’s Your IQ?
  • It’s your turn to discover the nuances of social media research
  • The Joy of Stats with Hans Rosling #MRX
  • My Theory on Declining Response Rate
  • What Ethnographers Do: Interview Guide (via Patterns of Play)

    [tweetmeme source=”lovestats” only_single=false]This is the first time I’ve reblogged via wordpress so who knows what will happen!

    Anyways, I thought this was an interesting way to remind people what social media research is. It’s not monitoring or reading or counting or answering questions. It’s formulating a plan around structured questions in order to come up with answers and possibly more questions. It’s “good old fashioned” ethnography brought into the world of social media.

    Read and appreciate!

    Part of the documentation we assemble before every project is an interview and observation guide. It’s supposed to serve as a checklist for supplies, as well as a guideline for how to conduct the interview. It includes the information we are trying to get at, as well as the questions we’re using to do so. Here is our interview guide for our current project. If you would like a copy of our interview guide, you can download it here as a Word docume … Read More

    via Patterns of Play

    Read these too

  • 1 topic, 5 blogs: Rich Media in Surveys
  • 3 Reasons Why Researchers Hate Focus Groups #MRX
  • 1 Topic 5 Blogs: DIY surveys suck or save the day
  • Paul the Octopus, Phd in Statistics, Lettered in football
  • Pie Charts – Our Evil Friend
  • If you DARE – the chart test of death

    Former logo of American Idol from 2002 to 2008.

    Image via Wikipedia

    After asking someone what a t-test is and whether they watch American Idol, I have another question I like to ask interviewees to see if they know their stuff. I ask them to create a chart that they would present to a client. Ahhh, the variety of answers this reveals.

    1) Some folks tell me that this version of Excel doesn’t do charts but they can do it on their version. (Honestly. This one astounds me.)
    2) Others create a chart. That’s it. Nothing else. When I re-prompt that it’s to give to a client, I still get a default chart in default colors with a default no title and default lack of zero on the axis.
    3) A few create a chart and try to pick the right chart, try to put a title, and try to fix the axis.

    Right away, I can see that Person #1 has weak problem solving skills and great excuse skills. There are multiple ways to chart in excel and if you can’t get one of them to work, I can’t help you. Thanks for coming out.

    Person #2 pays no attention to detail and barely wants to get the job done. Thanks for coming out.

    Person #3 is my person. Whether they choose the right options, I know they are curious and want to do the right thing. If they don’t know what the right thing is, I can teach them and they can learn.

    If I ever get to ask you this question, please make sure your chart has a non-misleading scale, a good title, non-florescent colours, and isn’t 3D. My sincere thanks.

    Read these too

  • Art, History, and Culture in Chicago Two Balloons and a Great Song
  • Survey Panel Questions – Enough Already! Word Cloud of my Resume – Wicked Cool!
  • Survey Design Tip #4: Brands are people too
  • Oh oh, I Have Another Favourite Statistic!
  • TV Shows From When I Was Short
  • Are you a scientist, an artist, or a magician?

    Is research an art or a science? Personally, I think it’s both. Actually, I think it’s a magical combination of the two. So, when I’m hiring a new researcher, I have a few standard tests that help me determine whether I’ve got an artist, a scientist, or a magician in front of me. Let me share one of my tests with you.

    I like to ask people to consider that I know nothing about research and statistics and then describe to me what a t-test is. I tend to get one of three answers.

    1) A 500 word technically stunning and accurate monologue incorporating as many statistical terms that can be thought of in the time allotted.
    2) A series of ums and awws followed by an admission that they didn’t so well in statistics but they think it has something to do with numbers.
    3) A round about discussion about how the average guy is taller than the average girl.

    The first answer is the scientist. Even after I prod and give examples, they still can’t find their voice. I know this person knows the answer, but I wouldn’t dare ask them to write a report, answer an email, or talk to a client. Imagine the motion sickness that client would experience!

    The second answer is the artist. Even though the jobs ad clearly said “knows statistics,” this person has creatively managed to work around such an unimportant detail. They attempt to stun me with a selection of phrasing in hopes that some combination of words will mean something.

    The third answer is the magician. They have been gifted with that rare skill of actually doing what they’ve been asked to do. They have proven to me that they know the answer AND I know they will be able to talk to clients. They are a magical combination of scientist and artist. And then, only then, I ask my real interview question, “Are you addicted to American Idol?”

    Would you pass my test?

    Read these too


  • So you think you have a probability sample
  • Cronbach’s Alpha, My Favourite Statistic
  • Permissible pie charts and 1 bar chart
  • Esomar Chicago: Day 1
  • Tagxedo for colorful shapeable saveable word clouds
  • %d bloggers like this: