Live note taking at the #IIeX conference in Atlanta. Any errors are my own.
Panel: The Next Generation of Market Research & Insights Creation
; Moderated by Leonard Murphy (GreenBook) with panelists Chris Enger (Periscope by McKinsey), Tamara Char (Periscope by McKinsey), & Simon Chadwick (Cambiar)
- Periscope by McKinsey is a suite of tools for collecting learnings, analytics
- Our entire industry is fragmented, over half of companies that source data did not exist ten years ago and they may not exist ten years form now
- Technology is not the driver of change, client needs and circumstances are the drivers of change, they are being asked to do far more with budgets lower than they used to be, they much get creative
- Behavioural data and analytics techniques to analyze that data is suddenly easily available and analyzable, this changes everything about being able to identify insights and work in an agile way, can get to 80/20 answers more quickly, we don’t need the 100% answer, we need to make progress on problem solving
- Are analytics pushing the business forward, are the ‘researchers’ falling behind and failing to get seat at the table?
- Need to elevate the quality and consistency of data so that the leadership is never getting three answers to the same question nor are employees hearing diverging answers
- You must have a c-suite leader and hopefully the chief financial officer who has a longer tenure in a company, not the chief marketing officer
- The CMO needs to spend time developing strategies not waiting to get data, let the machines do the heavy lifting so the team can spend their time strategizing
- What is the role of the methodologist, understanding fit for purpose of all the tools, this is why we’re seeing so much fragmentation,
- In the USA, people are attracted by tools. In the EU, they are more focused on ideas and creativity, and try to be creative all through the entire process. Need to be less technologically focused in the USA.
- Try assigning various people on th c-suite to BE a person in a segment, have them go shopping for her, experience her, all to get them to empathize more clearly, because c-suite lives are so completely different from their segments
- Is automation a dirty word? Machine learning templates and speeds everything up, may eliminate bias of an individual person although it will perpetuate bias that exists within the data
- We need to present data for ten minutes and then discuss the oilers and solutions for the remaining 50 minutes
Panel: The GRIT Report & Future Impacts
; Moderated by Leonard Murphy (GreenBook) with panelists Aaron Reid, Ph.D (Sentient Decision Science), Patricia Chapin-Bayley (Toluna), Rick Kelly (Fuel Cycle) & Isaac Rogers (20|20 Research)
- Automation is mostly used for analysis of surveys data, charting and infographics, analysis of text data, analysis of social media, sampling
- “My clients aren’t asking me for social media data” no they aren’t, they’re asking someone else
- Automation frees up time to expand capacity and do more, many things will soon be automated. We must adapt to this or fall by the wayside.
- Buyers are slow to adopt automation, automation is a dirty word because they think it is DIY and it will be more work. It will actually free up resources and allow you to do more once you are trained and moving forward.
- Do you want to be at a data collection conference in five years or at an insights conferences? Your business must adopt automation.
- People don’t CARE if you automate, they want better research insights and thinking. You must have automation to get there.
- Automation may not cut your budget but it allows you to move your budget into higher value endeavours.
- What should samplers do? Advise on representativity, enforce length of interview limits, consult on questionnaire design, restrict to mobile only, forbid mobile-unfriendly. it is an absutive relationship – clients don’t want to pay for consumer friendly and respectful questionnaires.
- There is no such thing as a non-mobile study. Every device must work and work well. You cannot run a survey without mobile respondents or you are guaranteed a nonrepresentative sample. Why is this even a conversation?
- If you aren’t thinking mobile first, you are being stupid. We spend half of our time on our devices. It is a data quality issue. [Cannot agree with this comment enough]
- Educating the researcher of the future – they need critical thinking and storytelling skills. We all need to be critical thinking experts, you shouldn’t in the business without that. We need to train the current workforce on how to do this. We’ve trained people on how to run cross-tabs but they need training on storytelling and turning insights into action.
- Quick research doesn’t have to be quick and dirty or poor quality
- The technology doesn’t matter, the platform doesn’t matter, we need to stop talking about the technology and focus on consultation, understanding the problem
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
Plus c’est la même chose, The Future of Market Research Education
Reg Baker, Executive Director of MRII
- People don’t trust us with their data, we need to have this conversation with them
- Most of us are in the business by accident
- If we knew what we were doing, it wouldn’t be called research
- forces shaping research – from data scarcity to abundance, from asking to listening, from analyzing to synthesizing
- The argument – there is a set of principles that distinguish good from bad regardless of method – science
- Clients have expectations of accuracy, how bad they are willing to accept, how good to use for predictions
- MR education is way too focused on training to do a task and not enough on teaching principles to apply across technologies
- Most training is “how to write a survey” and “how to run a focus group”
- Training is not education. Training is acquiring knowledge for specific competencies. This is bringing new people into our organizations. People learn a tracking project and they populate the graphs.
- Education is knowledge, skills, habits, beliefs, formative beliefs, ideals
- How do people become educated in market research? There are 5 programs in the USA, a couple dozen people per year. Associations help with webinars that are sometimes sales pitches of one method, Continuing education people like Reg’s organization. Employers do the bulk of the training to make people productive – the smart ones will figure it out over time.
- What should a program teach its students? What about social listening, predictive analytics, management consulting, big data, consumer research?
- The firm of the future needs – specialists (data scientists, survey researchers, neuroscientists), business consultants, polymaths (he who knows much, generalists who understand how all the methods work together and assign the right method to the right objective)
- Joan Lewis – we need to be methodology agnostic. The answer to every business problem is NOT a survey [WHAT!?!?!]
- We need to teach people how to see noise.
- Market research is easy. There are just a few easy steps. Understand the business problem. Know the full range of methods and data sources that might be used. Gather the right set of data. Understands the strengths and weaknesses of the data and resolve the inconsistencies. Create an actionable narrative.
- We need to have an open mind about methods and learn when to use each one.
- We need to focus on principles not ways of doing things. What makes “good” research? reliable, credible, can bet the farm on it.
- We need to teach people the art of synthesis.
Statistics are boring. They’re hard. They’re useless. You’ll never use them in real life.
Oh, how wrong that is. I’ll agree that if you aren’t blessed with the genes that make math and statistics a piece of pie (mmmm, pie), then yes statistics are hard. But there are innumerable real-life examples to show just how important it is to be comfortable with statistics.
Sports: If you’re a fan of sports, you no doubt are bombarded with statistics throughout the season just like these interpretations of statistics shared by James Conley on the Pensburgh. The headlines are exciting but the reality of each headline is simple – they mislead and even outright lie. If you understood statistics, you’d immediately see for yourself what the numbers really said.
HEADLINE! Pittsburgh’s penalty kill is going to be an Achilles’ Heel this season!
(The team has killed 18 straight chances over its last five games.)
BREAKING! The team is going to walk away with the Metropolitan Division again!<
(They’re in second place, and half the division is within a point of catching them.)
THIS JUST IN! The Penguins just can’t put away teams late in the game!
(They’ve outscored their opponents 5-0 in the third period of two straight games, both wins.)
Medicine: How many commercials on TV and ads in magazines extol the virtues of amazing new drugs, perhaps even drugs that you are desperate to try to alleviate your own health issues? If you understood statistics, you would know right away when the ads were misleading. You’d spot when the sample sizes were too small to be reliable, when the effect size was too small to be meaningful, or when the lack of a test-retest design suggested insufficient testing.
Sometimes companies egregiously exaggerate how well their drugs work. In a brochure given to doctors and nurses last year, the Japanese drug company Eisai claimed that its Dacogen drug helped 38% of patients with a rare blood cell disorder in a clinical study. This figure was false, the FDA said in a November 2009 warning letter. In fact, the figure was taken from a tiny subgroup of patients who responded well to the drug. When all patients in the study were included, the real response rate was a much less impressive 20%, the FDA noted.
Read more on this and other misleading advertisements here.
Politics: Political polling is becoming more and more prominent in the news. If you had a better understanding of statistics, you would know when to trust the polls. You would know why percentages don’t always add to 100, why polls ‘weight’ data, or why the margin of error is ridiculously important (even if you don’t have a random sample).
Seven hundred randomly selected New York likely voters were interviewed by landline and cell telephone between October 1 and November 1, 2014. The margin of sampling error is +/- 3.6 percent. The data have been weighted to adjust for numbers of adults and telephone lines within households, sex, age, and region. Due to rounding, percentages may not sum to 100%. Responder numbers in each demographic may not equal the total respondent number due to respondents choosing not to answer some questions.
No matter how you look at it, statistics are among the most important classes you can take. It’s in your best interest to sign up for a class now.
- Data Tables: The scourge of falsely significant results #MRX (lovestats.wordpress.com)
- Proud to be a member of survey research #MRX (lovestats.wordpress.com)
Live blogging from the #MRIA national conference in Saskatoon, Saskatchewan. Any errors or bad jokes are my own.
Applying a Corporate Reputation Model to the Higher Education Sector
Sean Simpson, Vice President, Ipsos Reid Public Affairs
- when someone trusts your company, they are more likely to believe your communications
- are people aware, familiar, favourable, trusting, advocating your company, you need to get to the top of that, when people trust you enough they become your advocate
- trust and marketing efficiencies are related,
- higher education may not be trying to selling anything but the model works for education in addition to corporate
- ROI of reputation – apply to a school, attend a school, graduate from school, attract professors, influence government, create advocates for your school
- do trust scores get more donations? no, because donations are linked to the school you attended not really any other school
- more likely to donate to your own school if you have stronger trust with it
- drivers of trust include quality of programs, care of wellbeing of students, community involvement, job readiness
- two key drivers for schools overall – quality and wellbeing of students, but there are differences by size of school
- for large schools – quality of the program is the number one driver
- for small schools – care about wellbeing is the number one driver, quality of program is important but only via other routes
- rankings are produced but they aren’t the full picture, ranks don’t match up to macleans because they split their list into groups; colleges rankings show differences as well
- challenges – is trust the right proxy for reputation? it’s not an index because those are too intercorrelated that the index is too complicated
- model tells schools how to improve their reputation
- drivers are customizable
Welcome to me! It seems that I am the newest member of the Georgian College Research Analyst program advisory committee. I’m not completely sure yet what my role will entail but at least a portion of it will be to advise the college on the types of skills and knowledge their students should acquire as part of the program.
In my first meeting, I learned a number of interested things.
- Only about 20% of applicants are accepted into the program. Wow! That’s tougher than most university programs and many graduate degree programs!
- Their major research projects are often conducted as ‘freesearch.’ In other words, businesses and government offices take advantage of their students to conduct research for free. Given that the research projects I reviewed as part of the Education issue of Vue magazine (September 2013) were on par with a lot of paid-for work I’ve seen, whatever Georgian college is doing is top-notch and worth far more than free.
- Employers hiring RAPP students often write in the student evaluations that the students were productive on the very first day. I saw the quotes. I was impressed!
- While there are a number of awards for students conducting outstanding work during the program, there are currently no entrance scholarships for students who may be deserving of the program but simply cannot afford to apply to attend.
What did I take from this? The RAPP program finds great people and turns them into great researchers. It is to our advantage as market research employers to provide the students with internships as many of those interns will likely become our next awesome new hire. And think about whether your company can provide an entrance scholarship to a deserving student. There are a lot of organizations out there that can could easily make this their good deed of the day.
In grade 10, my math teacher showed up to class to snooze and babble. He took naps while we took tests, and he used chalk to write out problems on the board while we used chalk to whip at his head. Students enjoyed discreetly dropping inappropriate items into his pockets as he walked past them and they took full advantage of the test scoring keys that he left open on his desk. I didn’t learn much that year. In fact, the classroom was so distracting and intimidating that I managed to achieve my worst math grade ever. Worst teacher ever.
The next year, having realized just how far I had fallen behind in math, I took what, at the time, was an embarrassing step. I chose to enrol myself in the general math program. Not the advanced, headed off to university program. I prepared myself to join a class of losers being taught by a woefully inadequate football coach.
Find me in this photo!
Mr Verhoeven did not employ any discipline in his classroom, even though the students had a reputation as being unruly troublemakers. Truthfully, he didn’t need to. He was quiet, conscientious, and respectful of the students, and the students behaved the same way towards him. He paid attention to who was succeeding and gave special care to those who struggled. It was easy to learn in his classroom because he kept everyone relaxed and comfortable. I caught up quickly and ended up at the top of the class. And when I returned to the advanced math stream, I remained at the top of the class with a new-found respect for the teacher and the students who struggled in the class I left behind.
I fast forward once again to university, to the dreaded first year statistics course where the professor says, “two thirds of you will fail my course.” The first test came along and I managed to fail it along with nearly everyone else in the class. When I attempted to argue my unfair and undeserved grade with the professor, he told me that he knew I knew the answers but I wasn’t expressing myself clearly. For example, the median is not the middle item in a list, but rather the middle item in an ordered list. He was firm about not changing anyone’s grade. He worked hard to teach us that hard work and careful attention to detail would allow us to recover from failure.
I went into the final exam with a terrible grade but we were told if we achieved a better grade on that exam, it would become our final grade. I remember handing in my exam paper early. The professor took it, smiled at me, and asked if I had double checked it, was sure I was finished? I was sure. 🙂
[tweetmeme source=”lovestats” only_single=false]I’ve been asked on a number of occasions how I found a job in market research after completing a Phd in psychology. You’d think the opportunities are endless but, like any career path, there are always obstacles.
Interviewers have told me to my face that they refuse to hire or they don’t like Phds. They’ve even given me strange tests to determine whether I’m human or robot, like “What does this abstract painting on my wall mean to you?” (Honestly, that piece of crap is major ugly.)
So here is my advice. It’s free advice so the confidence intervals are wide. Please do ask questions and I promise I’ll answer all of them.
- The fact that you have a Phd means you know research and statistics. Don’t waste your cover letter proving this.
- The fact that you have a Phd means employers think you don’t know the real world and that you can’t speak casual english. Prove this wrong in your cover letter. Write in business dialect not in dissertation dialect. This is one case where fancy words do NOT impress.
- Forget the stats speak. When they ask you what a t-test is, don’t tell them it’s an analysis of mean scores and confidence intervals of a quantitative variable for a second qualitative variable. Speak english. Say it’s a way to determine if two groups of people differ on a measure like height or weight.
- Join a few online survey panels so you can get a lay of the land. What questions get asked? How are they asked? Do you want to shoot yourself during the survey because it’s so horrid? This will give you insight about the business you think you want to get into and…
- … something to talk about during interviews. In which you will speak like a human being not a professor.
- Go to a used bookstore and buy a Market Research 101 textbook. Learn it.
- If your field isn’t psychology, you would do well to take a course in social psychology or personality psychology. It will give you great insight into survey question design.
- Learn either SAS or SQL. Not the menu driven kind, the syntax programming type. Even if you don’t end up using it on the job, you will be better able to talk to the statisticians and get what you need in the time you need.
- Accept that in the business world, projects take 14 days not 14 months, with sample sizes of 200 not 2000, and conclusions that are final not proposals for 8 more years of research. Now is not the time to try to convince your gracious hosts otherwise. They aren’t stupid.
You are already qualified. You just need the right vocabulary and the right perspective on research for business. Any questions?
Read these too
In Canada, there are very few schools that specialize in teaching the skill of market research at the undergraduate level. In fact, I only know one.
Now, there are many, many programs that inclde a couple courses in statistics, or research design, or marketing. These at least provide some fundamental knowledge so that when you hear a term later on, you at least recognize that it is a term. If you can’t find an MR program, the next best thing is to do a degree in psychology, sociology, geography, or marketing. I may be biased but I think the best option is a major in psychology with a minor in marketing. You can see though, that even if you create an optimal program, none of these focus on the art and science of MR as its own academic area.
Even those folks who go on to earn graduate degrees fall into the same bucket. Psychology graduate students do their research on psychology topics and probably never take a marketing course. Their research skills are top notch but an internalized perspective on marketing is lacking. And, marketers do their research in marketing and don’t have the background in social psychology to better understand why people buy the way they buy.
What it means is that most new market researchers come to the table with serious gaps in knowledge. They must resort to learning on the job. If they’re lucky, the person who trains them is a wonderful mentor with many years of experience. But those folks are few and not always readily available to the junior folk. What is more likely the case is that someone barely senior to them tells them just enough to get the job done because they are still trying to learn the skills themselves. In my case, the only mentor I had was an intro marketing textbook that I picked up at a used book store.
We are fortunate that our MR societies have ongoing training courses and certification. Unfortunately, these cost money and new graduates just don’t have that kind of cash. Nor do their employers have money to invest in a newbie. Which means a lot of people in the MR industry are not as skilled as they should be.
Maybe this is partly why our industry is struggling through data quality issues. Not enough people understand the psychology behind survey answering. Not enough people understand the myriad precise techniques of writing survey questions. This lack of MR skills leads to bad surveys which leads to bad survey experiences and results in declining response rates.
So, here’s my idea. It’s not new. If you work with newbies, be that missing link. Be a mentor. Teach them everything you can. Send them to conferences. Make the time to set up lunch and learns not because you have to, but because its the right thing to do. Invest in your company by financing their CMRP certification. This will lead to a better research product, happier employees, and a stronger company.
I thank you, and your newbies thank you too!
There are many ways to terrify people. Put a spider on their shoulder, make them stand close to the edge of a cliff, tell them you’re going to visit the in-laws (ftr, mine are great). Different people are scared of different things. There does, however, seem to be one fear that transcends other fears – the fear of statistics and numbers.
How did this come to be? Were our math teachers horrible people? I doubt it (though one of mine was and that’s a whole seperate post). Were we threatened with having to do extra math if we didn’t finish our brussel sprouts? Doubt that too.
Here’s my theory. Remember english class where you wrote a beautiful essay and the teacher gave you an A? That A didn’t mean perfect, it meant great job. However, you never got an A in math. You got an 80%. In other words, you got 80% right, and 20% horribly, horribly wrong. You failed at 20%. You sucked for 20%. Even though you did a great job, you still managed to screw up a lot of answers.
Math insists on having a right answer. It’s right or its wrong. It’s not a teachers perception of your thoughts and ideas and its not even a measure of how much they hate you. For, even if your math teacher hates you, if your answer matches what’s in the teachers edition, you got the mark and the grade.
It seems to be that, even though nobody is perfect, we are scared of situations where there is no doubt we are wrong. We seem to forget that everyone is wrong at one point or another, and we all have strengths and weaknesses.
My advice to you is don’t be fearful. Expect to make mistakes. Expect to forget formulas. No one is perfect and no one gets every math problem right.
Statistics can actually be interesting if you really to listen to them. TV commercials and other marketing materials use lots of bad statistics and they are a source of great amusement, at least for me. And, you will find that people who are okay around numbers are in high demand in the job market. That’s good enough for me!