Tag Archives: psychology

Qualitative Market Research is Bunk #MRX

A photo of a group conducting psychotherapy.

Image via Wikipedia

I have the greatest of respect for psychologists. My original training was as an experimental psychologist, and as such, I obtained permission to take a graduate level class in ethics alongside students training to become clinical psychologists. I learned the ins and outs of the ethics involved with working with people in the most fragile state of their lives, and trust me it’s very complicated. What I learned there has come with me through my entire career. Any time ethical issues arise in the field of market research, I always look back on what I learned in my training to become a psychologist to guide me down the right path. It’s a good place to start because academic Psychologists have some of the strictest ethical codes out there. People’s lives are on the line.

Psychology ethics codes sound very much like a medical doctor’s code. Do no harm. Respect the individual. Care for people in situations where they may be unable to care for themselves. This is what I seek in market research even when some people say that I am being naive and idealistic.  But I’m OK with that. I’d rather be too careful than risk someone being hurt. I sleep well because of this.

Is it an issue of informed consent?

So when I saw the MRS paper on proposed privacy guidelines, I was very concerned. My interpretation of it suggests that it disregards the legitimacy of naturalistic observation research (i.e., practice of scientifically observing people in their natural environment without interfering with them or drawing attention to yourself). Here is one quote (all the quotes in this post are out of context) from the MRS paper:

“That is why as the real and virtual worlds converge further, MRSB believes that researchers must apply the same ethical standards to research online as they do in real life. That is: Obtaining the informed consent of all persons from or about whom data is collected”

To me, this flies in the face of traditional ethical research methods that have been practiced by psychologists, sociologists, anthropologists, and market researchers for decades. All of these codes permit naturalistic observation and recognize it as a valid and ethical practice. Here are just a few excerpts from various ethics codes of psychology organizations.

Canadian Psychological Association Code of Ethics
“Obtain informed consent for all research activities that involve
obtrusive measures, invasion of privacy, more than minimal risk of
harm, or any attempt to change the behaviour of research

American Psychological Association Online Research Ethics
When conducting research online, researchers need to contend with changes in the technology, the ways the technology is typically used, and the norms surrounding this use, because this context is integral to assessing anonymity, privacy, risk and the like. For example, the concept of minimal risk depends upon a comparison of the risk associated with research participation to risk in everyday life. The concept of privacy depends upon participants’ reasonable expectations about whether others will be allowed access to information about them. As online behavior and norms change, the nature of minimal risk and the very concept of privacy themselves change.

American Psychological Association Code of Ethics
“8.05 Dispensing with Informed Consent for Research
Psychologists may dispense with informed consent only (1) where research would not reasonably be assumed to create distress or harm and involves (a) the study of normal educational practices, curricula, or classroom management methods conducted in educational settings; (b) only anonymous questionnaires, naturalistic observations, or archival research for which disclosure of responses would not place participants at risk of criminal or civil liability or damage their financial standing, employability, or reputation, and confidentiality is protected; or (c) the study of factors related to job or organization effectiveness conducted in organizational settings for which there is no risk to participants’ employability, and confidentiality is protected or (2) where otherwise permitted by law or federal or institutional regulations”

British Psychological Society Code of Ethics
“Unless informed consent has been obtained, restrict research based  upon observations of public behaviour to those situations in which  persons being studied would reasonably expect to be observed by  strangers, with reference to local cultural values and to the privacy of  persons who, even while in a public space, may believe they are unobserved.”

Department of Psychology, University of Sheffield
9. Observational Research
9.1 Studies based upon observation must respect the privacy and psychological well-being of the individuals studied. Unless those observed give their consent to being observed, observational research is only acceptable in situations where those observed would expect to be observed by strangers. Additionally, particular account should be taken of local cultural values and of the possibility of intruding upon the privacy of individuals who, even while in a normally public space, may believe they are unobserved.

Is it an issue of permissions?
This for sure is a no. Ethical monitoring and measuring and research platforms would not exist if it was an issue of permissions. Websites have two very simple ways of ensuring their data remains private. First, they can insert code into their website that essentially says “please don’t gather our data.” This code is readily respected by ethical companies. Second, websites can choose to NOT create APIs that permit the sharing of data with external companies. Since Facebook and Twitter and most other major websites provide APIs to give data to third parties, they are the ones who determine what can be shared within their privacy policy. Ethical companies won’t, and physically can NOT, take data that isn’t permitted.
Is it an issue of masking verbatims?

On a separate note, the MRS paper indicates this about the cloaking of data: “While data masking or cloaking may be offered as a privacy solution, from a methodological point of view it is rather unsatisfactory. This approach could distort results if the original wording was materially altered (and even a single word change can do this).”

This, I believe to be an affront to the skills, expertise, and training of most qualitative researchers who perform this task on a daily basis with the utmost of quality in workmanship. Are we to disregard qualitative researchers? Push them outside of the market research world? Ignore the contributions of people whose skills are qualitative because they are incapable of properly summarizing or describing opinions? I can only hope that this was an unintended side effect. Masking is truly such a minor issue that I immediately begin picturing babies and bathwater. Masking is a widely used process that can be taught and learned. So let’s teach it and learn it.

Whether you agree or disagree with me, whether my opinion is in the extreme minority, I implore you to share your opinions with the MRS. Our industry is being shaped right now and your voice matters. Please speak up.


If you think ads don’t influence you, you’re only fooling yourself

Babies, brains and relationships 075

Image by Baha'i Views / Flitzy Phoebie via Flickr

Implicit memory is a fascinating aspects of human psychology. Even when you think you aren’t paying attention to something, your brain is still taking it all in. All those ads on TV that you talk over, all the billboards along the highway that you ignore, all the signs in store windows that you gloss over, all of them are still being registered by your brain even if you actively try to ignore them.

Implicit memory is the memory you have for things that you have experienced before but don’t actively remember. It essentially causes you to do something, think of something,  buy something, recommend something, try something that you’ve already heard about before even if you can’t remember hearing about it before.

It’s the favorite colour you have for buying clothes because your mom used to dress you in that colour when you were a baby. It’s your favourite food because Grandma gave it to you once as a reward even though you don’t remember the event at all. It’s all th95% of events that pass you by every day that you don’t pay the slightest attention to.

What this means is your opinion of statistics will be just a little bit more positive after leaving this website. Of course, it may just mean that your hate score of 1.1 on a five point score will only improve to 1.2.

6 Myths of Sentiment Validity

1) You can tell how valid a sentiment scoring system is by evaluating as few as 20 records
2) You can accurately judge validity by examining the originally assigned score and deciding if you agree with it
3) If data for one brand is valid, data for all the brands are probably valid as well
4) You can judge validity by checking twitter data as it is the lowest common denominator
5) If the system is based on natural language processing, you know the sentiment is valid
6) If the sentiment scoring is manual, you know it’s perfectly valid.

Sentiment Analysis is THE BOMB! #mrx

What was once thought to be an impossible goal has now been achieved. Sentiment analysis is now quick to do and extremely easy. For numerous decades, the best scientists in the field have worked hard to develop automated systems that can achieve validity in the range of 100%. This level of precision and intelligence was a dream for many years and now has been achieved. The illustration below shows some people celebrating this amazing accomplishment.

Join me in celebrating April Fools day, but please remember this: If you hear someone saying their sentiment system validates at 90%, 95%, or 100%, run. Run quickly.  In the opposite direction.

Huh? I never pay attention to you anyways. #MRX

We expect a lot from people. We ask them tell us why they like certain products, why they purchase certain products, which products they intend to purchase. We ask them whether they’d prefer blue or yellow sprinkles on their donuts, and whether they like the courier or time new roman font on the package. We expect them to pay attention to and consciously remember every minute detail of their lives. Continue reading →

From academia to a job: Free advice from a Phd

Werner Erhard and Associates v. Christopher Co...

Image via Wikipedia

[tweetmeme source=”lovestats” only_single=false]I’ve been asked on a number of occasions how I found a job in market research after completing a Phd in psychology. You’d think the opportunities are endless but, like any career path, there are always obstacles.

Interviewers have told me to my face that they refuse to hire or they don’t like Phds. They’ve even given me strange tests to determine whether I’m human or robot, like “What does this abstract painting on my wall mean to you?” (Honestly, that piece of crap is major ugly.)

So here is my advice. It’s free advice so the confidence intervals are wide. Please do ask questions and I promise I’ll answer all of them.

  1. The fact that you have a Phd means you know research and statistics. Don’t waste your cover letter proving this.
  2. The fact that you have a Phd means employers think you don’t know the real world and that you can’t speak casual english. Prove this wrong in your cover letter. Write in business dialect not in dissertation dialect. This is one case where fancy words do NOT impress.
  3. Forget the stats speak. When they ask you what a t-test is, don’t tell them it’s an analysis of mean scores and confidence intervals of a quantitative variable for a second qualitative variable. Speak english. Say it’s a way to determine if two groups of people differ on a measure like height or weight.
  4. Join a few online survey panels so you can get a lay of the land. What questions get asked? How are they asked? Do you want to shoot yourself during the survey because it’s so horrid? This will give you insight about the business you think you want to get into and…
  5. … something to talk about during interviews. In which you will speak like a human being not a professor.
  6. Go to a used bookstore and buy a Market Research 101 textbook. Learn it.
  7. If your field isn’t psychology, you would do well to take a course in social psychology or personality psychology. It will give you great insight into survey question design.
  8. Learn either SAS or SQL. Not the menu driven kind, the syntax programming type. Even if you don’t end up using it on the job, you will be better able to talk to the statisticians and get what you need in the time you need.
  9. Accept that in the business world, projects take 14 days not 14 months, with sample sizes of 200 not 2000, and conclusions that are final not proposals for 8 more years of research. Now is not the time to try to convince your gracious hosts otherwise. They aren’t stupid.

You are already qualified. You just need the right vocabulary and the right perspective on research for business. Any questions?

Read these too

  • Kama Sutra Statistics
  • What’s Your IQ?
  • It’s your turn to discover the nuances of social media research
  • The Joy of Stats with Hans Rosling #MRX
  • My Theory on Declining Response Rate
  • That’s not my job… or is it? Someone please tell me!

    [tweetmeme source=”lovestats” only_single=false]
    Where does the line fall?

    As a researcher, my skills are carefully slotted into the design, analysis, and interpretation of research. And by interpretation, I mean explaining what the numbers on a chart are, explaining if/why they are significantly different, explaining how they relate to each other.

    But where does explaining the consequences of the numbers come into play? Who is the right person to draw conclusions about implications and action items? Is it the researcher? Is it the client? Is it a collaboration?

    I’m just asking.

    Read these too

  • Six things that terrify me in market research #mrx
  • To Mid-Point or Not To Mid-Point, That is the Question
  • Where is the privacy fallout?
  • Geico chucks wood
  • 5 random things I like about statistics and proof you are a dork
  • A Box Score Lesson for Psychology Students

    A plot of a normal distribution (or bell curve...

    Image via Wikipedia

    I was in school for a bunch of years, and took a bunch of research design courses and a bunch of statistical analysis courses. Easy ones, hard ones, and a few really interesting ones. Surprisingly, one thing I never learned about was box scores, a statistical staple in the market research world.

    Box scores are a way of talking about and working with Likert scales or other types of categorical scales so that everyone knows whether you are talking about the positive end of the scale (top box, top 2 box), the middle of the scale (middle/neutral box), or the negative end of the scale (bottom box, bottom 2 box).

    Instead of calculating average scores from the Likert scale responses, box scores are reported as the percentage out of the total number of people who answered the question.(If 10 out of 50 people chose strongly agree, top box score is 20%) Box scores let you clearly identify how many people fall into a subgroup – people who are happy, unhappy, or just don’t care about your product.

    Why do box scores matter? In a sense, they do report the same type of information as average scores. But, unless standard deviations are near and dear to you, average scores often appear very similar between groups. It’s hard to explain to a client why scores of 3.6 and 3.9 are very different because there is no intuitive difference between those numbers.

    But, let’s think about box scores now. Can you intuitively understand the difference between 30% of people liking your brand and 40% of people liking your brand? I’m pretty sure you can. And you don’t need to understand what a standard deviation is either. I’m not in favour of dumbing down statistics but I am in favour of people understanding them.

    Here’s another reason box scores are good. The average score calculated for a result that is 10% top box, 10% bottom box and 80% middle box is exactly the same average score you would get for a result that is 40% top box, 40% bottom box, and 20% middle box. I’d certainly like to know if 10% or 40% of people hated my product. That’s a pretty important difference to be aware of and I wouldn’t want it getting lost because someone had a weak understanding of what a SD is.

    So now, psychology/sociology/geography majors, go forth and prosper as market researchers!

    Read these too


    Be All That You Can Be: Be A Research Mentor

    Baby with book

    Image via Wikipedia

    In Canada, there are very few schools that specialize in teaching the skill of market research at the undergraduate level. In fact, I only know one.

    Now, there are many, many programs that inclde a couple courses in statistics, or research design, or marketing. These at least provide some fundamental knowledge so that when you hear a term later on, you at least recognize that it is a term. If you can’t find an MR program, the next best thing is to do a degree in psychology, sociology, geography, or marketing. I may be biased but I think the best option is a major in psychology with a minor in marketing. You can see though, that even if you create an optimal program, none of these focus on the art and science of MR as its own academic area.

    Even those folks who go on to earn graduate degrees fall into the same bucket. Psychology graduate students do their research on psychology topics and probably never take a marketing course. Their research skills are top notch but an internalized perspective on marketing is lacking. And, marketers do their research in marketing and don’t have the background in social psychology to better understand why people buy the way they buy.

    What it means is that most new market researchers come to the table with serious gaps in knowledge. They must resort to learning on the job. If they’re lucky, the person who trains them is a wonderful mentor with many years of experience. But those folks are few and not always readily available to the junior folk. What is more likely the case is that someone barely senior to them tells them just enough to get the job done because they are still trying to learn the skills themselves. In my case, the only mentor I had was an intro marketing textbook that I picked up at a used book store.

    We are fortunate that our MR societies have ongoing training courses and certification. Unfortunately, these cost money and new graduates just don’t have that kind of cash. Nor do their employers have money to invest in a newbie. Which means a lot of people in the MR industry are not as skilled as they should be.

    Maybe this is partly why our industry is struggling through data quality issues. Not enough people understand the psychology behind survey answering. Not enough people understand the myriad precise techniques of writing survey questions. This lack of MR skills leads to bad surveys which leads to bad survey experiences and results in declining response rates.

    So, here’s my idea. It’s not new. If you work with newbies, be that missing link. Be a mentor. Teach them everything you can. Send them to conferences. Make the time to set up lunch and learns not because you have to, but because its the right thing to do. Invest in your company by financing their CMRP certification. This will lead to a better research product, happier employees, and a stronger company.

    I thank you, and your newbies thank you too!

    Related Posts


  • #MRA_FOC #MRX MRA Articles of Incorporation, no longer 53 years old
  • Art, History, and Culture in Chicago Two Balloons and a Great Song
  • Survey Panel Questions – Enough Already! Word Cloud of my Resume – Wicked Cool!
  • Survey Design Tip #4: Brands are people too
  • Oh oh, I Have Another Favourite Statistic!
  • The Psychological Theory of Positive Reinforcement

    In psychological theory, there are various ways to change, maintain, or create behaviours. There is a two by two chart (!!!! calm down) that illustrates the giving (positive) of treatment to change behaviour, the taking (negative) of treatment to change behaviour, the encouragement (reinforcement) of behaviour, and the discouragement (punishment) of behaviour. Most people are familiar with (negative) punishment to make kids behave better, including taking away privileges.
    However, I just don’t know how this one fits into the scheme of things. When I was in school, we were asked to choose one of our own behaviours, choose a reward or punishment, and record our change (or non-change) in behaviour. Well, I knew that I needed to watch less TV so choosing the behaviour was a done deal. The second part was choosing a reward. THAT, was the tough one. I was dirt poor and so telling myself that I would get myself a magazine or coffee or a movie night as reward was out of the question. I needed a non-financial reward. And I found one. I was FAR more interested in seeing the results of this highly scientific study than anything else. In other words, I want to see a change in the chart that I was going to create from the results. Needless to say, I vastly decreased by television viewing. It was a great chart. 🙂
    But I digress. I confess that it has been more than one week since my last blog. My chart of blog views showed a decline over that period, which in itself was fun to see, but now the standard deviation around my average blog views has increased resulting in larger confidence intervals around the mean. I hope you can forgive me. But what a nice chart it is!


    Related Articles


  • Probability Sampling – Proof that only telephone samples are quality samples
  • Edward Tufte: Your Presentation Sucks Cause Your Content Sucks
  • Ray Poynter – Overview of Online Research Trends #netgain5
  • An Acceptable Use of Pie Charts: Van Gogh Color Distributions #MRX
  • The Lost Art of Qualitative Research
  • %d bloggers like this: