Plus c’est la même chose, The Future of Market Research Education Reg Baker, MRII #NetGain2015 #MRX
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
Plus c’est la même chose, The Future of Market Research Education
Reg Baker, Executive Director of MRII
- People don’t trust us with their data, we need to have this conversation with them
- Most of us are in the business by accident
- If we knew what we were doing, it wouldn’t be called research
- forces shaping research – from data scarcity to abundance, from asking to listening, from analyzing to synthesizing
The argument – there is a set of principles that distinguish good from bad regardless of method – science- Clients have expectations of accuracy, how bad they are willing to accept, how good to use for predictions
- MR education is way too focused on training to do a task and not enough on teaching principles to apply across technologies
- Most training is “how to write a survey” and “how to run a focus group”
- Training is not education. Training is acquiring knowledge for specific competencies. This is bringing new people into our organizations. People learn a tracking project and they populate the graphs.
- Education is knowledge, skills, habits, beliefs, formative beliefs, ideals
- How do people become educated in market research? There are 5 programs in the USA, a couple dozen people per year. Associations help with webinars that are sometimes sales pitches of one method, Continuing education people like Reg’s organization. Employers do the bulk of the training to make people productive – the smart ones will figure it out over time.
- What should a program teach its students? What about social listening, predictive analytics, management consulting, big data, consumer research?
- The firm of the future needs – specialists (data scientists, survey researchers, neuroscientists), business consultants, polymaths (he who knows much, generalists who understand how all the methods work together and assign the right method to the right objective)
- Joan Lewis – we need to be methodology agnostic. The answer to every business problem is NOT a survey [WHAT!?!?!]
- We need to teach people how to see noise.
- Market research is easy. There are just a few easy steps. Understand the business problem. Know the full range of methods and data sources that might be used. Gather the right set of data. Understands the strengths and weaknesses of the data and resolve the inconsistencies. Create an actionable narrative.
- We need to have an open mind about methods and learn when to use each one.
- We need to focus on principles not ways of doing things. What makes “good” research? reliable, credible, can bet the farm on it.
- We need to teach people the art of synthesis.
Related articles
What the CBC Learned About the Olympics Unifying TV and Digital Data, by Kristin Wozniak and Greg Dinsmore, CBC #NetGain2015 #MRX
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
Kristin Wozniak, Manager of Television Research at CBC and Greg Dinsmore, Manager of Digital Research at CBC
- SOchi/FIFA was a great opportunity for the corporation to focus on one thing at a time. Throw everything you have at understanding something.
- Wanted to help content producers with their shows.
- Every event was live streamed. Day went from midnight to 3pm and then needed an evening show.
- TV ratings come in a day late so used digital data as first look at TV data at second point. But that wasn’t quite enough.
- Mandate to serve all canadians meant all platforms – radio, television, online. How do you get single source data? Encoded digital streams with television signals. (There was very little radio coverage)
- 97% of canadians watched or listened to sochi content on any platform
- Web analytics don’t say anything about the people, but the panel does
- Adobe Sitecatalyst measures devices that interact with a digital offering – website, video, audio, apps
- Substantial difference between television and sitecatalyst
- Panel membership showed more content was viewed. But why? They way they count viewers is different. Device may count 1 person watching TV even if 3 people are watching. Group viewing could be the reason for the difference.
- Digital video was more attractive to younger people, much higher during school hours
- Digital data might underreport, particularly for large scale events
- Does digital ADD to time watched or is it at the same time as TV watching
- Sochi viewer watch far less television than average Canadian, even during the sochi weeks. They aren’t just adding content. Maybe they’re just fans of a certain sport not the Olympics.
- Sochi digital viewers were more likely to be fans of sports in general, they watch less TV in general, they are cord-cutters, less likely to consume TV
- Maybe digital viewers have replaced some TV with digital and they only add on when they are really interested in the sport
- Digital viewers are more volatile in their viewing profiles
Related articles
Prediction Science: How Well Can We Predict the Future? Jon Puleston, Lightspeed GMI #NetGain2015 #MRX
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
Prediction Science: How Well Can We Predict the Future?
Jon Puleston, VP of Innovation at Lightspeed GMI
- [Jon had the room take a quiz on a random series of questions]
- What has he learned about predictions? Better at predicting certain things, behaviors. Not so good at predicting prices.
You can isolate people who are good at predicting, perhaps find them and use their super power 🙂- Prediction isn’t dependent on sample size. One person can be sufficient. It’s more about sample diversity and the intelligence of that sample.
- 16 is a crowd if they are all well-informed. That’s all you need.
- You could ask 1000 people or 5 people about the weather next week, but you really only need to ask the 1 person who saw a weather forecast.
- How do you aggregate crowd wisdom – mean/median/mode
- Crowds mean that errors get distilled out
- A crowd of people predicted the price of the ipad within 1%
- But without any knowledge, the crowd is ignorant. People NOW can’t predict the weight of a cow.
- Prediction is littered with cognitive biases – 68% of people will say that a coin will toss heads because we always hear ‘heads’ first
- People’s preferences for wine depend on whether you ask “Who prefers red wine?” vs “Who prefers white wine?”
- People who check their emails before breakfast are more likely to say people check their email before breakfast
- Emotions get in the way of making valid predictions. We are more positive about our own teams vs other teams when predicting score counts.
- Do you you clean up after a meeting vs Do people clean up after meetings. People say they do but they don’t. [I do. Even when I wasn’t in the meeting.]
- Read: http://www.amazon.com/Expert-Political-Judgment-Good-Know/dp/0691128715
- only 48% of stock market gurus stock market predictions were correct
- People who ‘bet’ 1 unit not as good as 2, but betting 3 is a little better. People who bet against are really good.
- After 15 people predict and other people see that, they start to predict the same way, the answers don’t move.
- But in an independent voting method, larger surveys are better than 15 people
- Best predictive market situations allow sharing of information, let people discuss and debate
- e.g., in guess which mug is most popular, someone will suggest mugs are good gifts, someone will suggest lots of people garden, people decide that the gardening mug will be most popular
- Think of board room meetings where they didn’t discuss things before they vote on a decision. Stray comments are problematic.
- Try dividing up the herd and then recombine the three groups back into one. Helps improve accuracy just like how we run 3 focus groups not 1.
- Let people change their opinions in surveys [We NEVER let people do that!]
Related articles
Wake Up or Die. Research Automation – The Future of Market Research Corinne Sandler, Fresh Intelligence #NetGain2015 #MRX
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
Wake Up or Die. Research Automation – The Future of Market Research
Corinne Sandler, CEO of Fresh Intelligence
- No war in the world has ever been won without intelligence
we are needed more and more as competition among brands intensifies- Where is market research on the Montclare SaaS250 list? Salesforce is the closest one.
- We need to think outside the fishbowl
- We play a consultative role, we see a brief that goes from analyst to director and we consult all along the way. What if the brains were integrated into technology.
- We need to move from consulting on research to dispensing research.
- Progress depends on the unreasonable man – George Bernard Shaw
- Intelligence needs to be accessible to the world
- You have only two choices today – you can use a third party or you can do it yourself
What about combining brains with an automated component, fully automated turn-key platform with consultants- Buyers of research have specific needs, and automation suits those needs well. Quality is a cost of entry, must believe that is what consumer is truly saying, need to rely on that right data.
- Buyers need to prove value of every dollar they spend. Priorities don’t always allow them to share the wealth among several brands.
- These methods allow you to do all the work without reveal proprietary information
- For instance, what if you told the provider 5 brand names, and the system automatically plugged them into a pre-programmed survey with pre-programmed charts.
- How do you create an app that adds value for toilet paper, because EVERYONE has an app. “Charmin Sit or Squat App” – tells you where the closest washroom is and how clean it is. [My mom would LOVE THIS!]
- 90% of our decisions are based on intuition
Related articles
Brand Building in a Digital, Social and Mobile Age Joel Rubinson, Rubinson Partners Inc. #NetGain2015 #MRX
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
Brand Building in a Digital, Social and Mobile Age
Joel Rubinson, President and Founder of Rubinson Partners Inc.
- Picture of brand success has to change
- We are no longer in a push word, consumers pull information at their leisure
- We engage in shopping behaviours even when we aren’t really shopping, we are always IN the path to purchase
- Brands must become media
Starbucks is the best example of a marketer that gets it. 40 million fans on facebook. millions of website visits. millions have downloaded their app. Every interaction generates data they can use, can be used for personalization, to amplify brand communications. They no longer have to pay for every message.- The rise of math experts in advertising – lift from using math to place advertising is a repeatable success
- Programmatic messaging is key. Think about impressions that are served up one user at a time. marketers goal is serve the most relevant ad at the right price. And this needs to scale.
- Research is missing in action when it comes to math – we lack digital metrics, still rely on survey based tracking, we have a post-mortem mind set, we are failing to change how marketing works
- We must get serious about integrating digital – why isn’t this happening, why are we locked in a survey world
- Our comfort zone is surveys. We know how to construct 20 minute surveys. Our learning zone is the mobile area where we unpack our surveys into smaller pieces.
- The panic zone is digital, we don’t understand it. We must move digital into the comfort zone.
- lets start by just looking at the data, look at page views, look at themes in social media, how big is your brand audience, how many likes on facebook, how many twitter followers, how many newsletter signups. These are unambiguous measures. Look at clicking and sharing and conversions.
- Stop treating social media as a hobby, not specialty projects, not ancillary thing to look at. You must find ways to increase positive word of mouth.
- Do we really need feedback from consumers every single day on attributes they never consider? Can’t social media which is much more organic do this?
- Bring in data that you can’t get from a survey that has action value. Some online panel companies already use a social login called OAuth. Append all the Facebook data to your survey and use it for targeting.
- Data aggregators have lots of profiling information for targeting ads throughout the web which means different people get different ads based on cookies from their browser
- You can also link in frequent shopper data to your survey data.
- You don’t have to guess whether an ad is working. You can run an experiment and serve the ad to one group of people and see the change in group behaviour.
- MR needs to know that brand meaning is done completely different now. People seek out knowledge, digital delivers information in real time. But marketing research hasn’t changed.
- Think digital and do something big. Shift some money into datascience or integration. Conduct in the moment research with smartphones.
Related articles
Emerging Technologies – Are They Still Emerging? Lenny Murphy, GreenBook Blog and GRIT Report #NetGain2015 #MRX
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
Emerging Technologies – Are They Still Emerging?
Lenny Murphy, Editor-in-Chief of GreenBook Blog and GRIT Report
- Attitudinal, behavioural, and intrinsic data
- Foundational research is no longer taking months but hours
Moving from questioning to discussing, from asking to observing, from data to insight, from understanding to predicting, from the big survey to data streams, from rational to behavioural, from quarterly to real time, from siloed to converged- the traditional survey as the primary driver of information will decline
- Data science is not a hoity toity term for a statistician. It’s information technology and algorithms and languages and hadoop and R. It’s statistics on steroids.
- The future looks very different.
- Over the next five years, we are in the realm of DIY, non-conscious measurement is emerging such as facial scanning and automated emotion measuring, automation and AI in terms of very very smart devices, internet of things where all of your things will collect and share data from your shoes to your car, virtual and augmented reality will change our media habits
- DIY – there are many free DIY tools
- The ‘make it’ revolution – consumers can ‘print’ their own things, print some shoes, do an ideation session using a printer. cost of these devices can be as low as $100.
- Emotional measurement – facial scanning, shopping behaviour videos, eye tracking
- AI – tons of money going here, google has spent millions on quantum computers, these will just be part of everything we do
- Internet of Things – Internet as we know it might disappear. Daily lives are just always all connected. e.g., Microsoft’s hololense.
- Do a virtual shopping experience without a computer. But you still feel like you are in the store.
- Imagine a connect fridge [will it shop for me once it notices I’m out of BREAD AND MILK!!]
- Google Glass succeeded in every aspect they hoped. The real product will come out in the next couple of years.
- Gamification has never taken hold but many companies are working in this area. Game to map out neurons.
- Which companies will be our competitors for clients and budget? Google, IBM, Apple, facebook, AOL, Verizon, Comcast, Disney, at&t, GE, groupm, WPP, amazon
Related articles
What’s Hot and What’s Not Hot by Ray Poynter, Vision Critical University #NetGain2015 #MRX
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
What’s Hot and What’s Not Hot: Ray Poynter, Director of Vision Critical University
- Ray’s book are for sale at the back of the conference. Find him and he will sign your book! (Yes, please!)
- What is still hot?
- Mobile is really big and that’s why Ray has written a book on it [buy it 🙂 ]
- Why is CATI so big – in this room, most people do NOT answer the landline in their home. Mobile used to cost more. Not sure if the person will be driving when you call a mobile phone. Hard to geographically target mobile phones like you could RDD.
- PEW research is top notch CATI probability surveys. It is the majority of what they do and they have just recently bumped the percentage of their calls that is mobile.
- Online surveys – 30% are attempted by people on mobile. Some people KNOW they are doing mobile and others don’t. May be 50% in just a couple of years. But only 15% of surveys are suitable for mobile devices. Most surveys are not optimized for mobile. Not thought about wording or question types. Not even checking the data to see if mobile vs laptop data are different.
In 950 Tesco stores, they do surveys on tablets with geolocation, datestamp, etc.- Heineken did a beer audit in Africa. Recruited interviewers, gave them a smartphone. Phone made SURE every location was geotagged. Photos of every location. Quality of data was far superior.
- Communities
- Companies doing so are beginning to disappear because communities are more mainstream. Everyone has their own community.
- DIY is enormous in society. DIY travel, DIY bank machines, Uber, AirBNB, ZappiStore.
- DIY has spawned automation. If every idiot can write a survey, they will. So let’s make it safer.
- SurveyMOnkey is the biggest survey platform in the world.
- To be hot, it must be scalable and it must work – NPS doesn’t do this. 🙂
- DIY isn’t great with efficacy. There won’t be many neuroscience for dummies books in the near future.
- What is HOT right now
- In the moment – Ask the breakfast survey the very second you finish your breakfast. Survey about the hotel registration before they open their hotel room door.
- Location Based Research – Put a geofence around a starbucks so you know who walk in or out. This also attracts aggressive marketers, not just researchers. So the message on your phone could be a survey or a sales pitch – ShopKick. Do they turn on the microphone on your phone? Do they turn on your camera? Do they tell you they have done so?
- Microsurveys – RIWI, google consumer surveys. Usually 1 to 3 questions. Google is up to 10 questions. Won’t tackle your problems that have a high dollar value associated with them.
- Automation – Automate reports as well as research process. What do we add to this? What do we add to the trends? What canNOT be automated?
- Always choose the simplest tool – don’t need to take a picture of every window and find software to count those pictures. [sounds stupid but really think about it]
- What’s bubbling new and exciting
- Text analytics – sentiment analysis is getting better for all except twitter. much better for emails and letters to companies, comments on youtube, inbound call centers, which letters are genuine sales leads or complaints or bomb threats, reaction marketing.
- Web Messaging – Whatsapp, WeChat. People are doing less talking to everyone and more talking to individuals. In comparison, whatsapp grew WAY more quickly than facebook and twitter. This is massively scalable. Panel companies will go this way. [They already are!]
- ResearchBots – Processing time and moderators takes a lot of time. New things don’t work all the time and that’s why it’s bleeding edge. Not very scalable at this point
- NOT so hot –
- Facial coding – good with an extremely experienced trained person sitting in the same room. Via webcam isn’t quite so good. Fully automated is very clever but delivers almost nothing. Software can identify specific pictures but a human must still go and interpret all those pictures. Great for assessing people’s reactions to packages. Not a general purpose tool. Doesn’t suit most research problems.
- Webcam Qual – You don’t want to take video from home because you still have to brush your hair and change out of your pajamas. Webcam on the bus means everyone behind you on the bus sees the images too.
- Social media research – We thought it would destroy MR but it’s really a niche. Most research teams have scaled back on this. Maybe using tweets only. not used so much for insights but more for reactions to advertising campaigns. Social does answer questions not asked. Social usually doesn’t answer your specific research questions. Vendors often say “I agree it has under-delivered but my company is doing it right!”
- Social media 2.0 – integration with marketing, integration with survey research, integration with tracking, interrogative.
- BT Case Study – Net Easy – how easy is it to work with BT was a better measure than NPS. They looked online for people talking about ease or difficulty and responded with solutions. Achieved a 3.5million reduction in costs by doing this. 600 000 people who would have called a telephone were able to DIY from the website.
- What about passive data, gamification, biometrics, wearables, quantified self, Internet of Things, single source, neuroscience. There is too much stuff to register the quality of everything. You can’t learn it all.
- Gamification doesn’t solve a lot of problems but it HAS made us rethink what we’re doing it.
- Behavioural economics is really efficacious but it is incredibly specific.
- Passive data from phone recording everything you press and everywhere you go. Won’t see big movements here. It will be mostly qualitative.
- Big data is beginning to move but predictiveness is limited right now.
- Wearables – sharable is great but these people are not yet representative. Mostly qualitative and very targeted.
- Geotracking – very tiny right now, works well in qualitative. Can draw maps of where individual people went. Mapping ebola is a different story – limitations of cell phone towers in other countries makes it impossible to map journeys in small locations.
- Internet of things – only exists in minds and publishers right now.
- Single source – Means tying together many data sources, it’s a power battle, a methodology battle. WHO is the single source? The telcom? A research company? Privacy battles of combining data.
- Top 2 Things to think about.
- Mobile – traditional, in the moment, multimedia, passive
- Integrative and participative – 360 panels, databases, communities, social, mobile, qual, collaborative all together
- “We will always need faxes” “We will always need horse and buggies” ….. We will NOT always need surveys. Ray thinks no more surveys in 20 years – classic 20 or 30 minute surveys. Suspects only 33% of spend will be on surveys by 2019.
- We need to redesign our ethics – most of our ethics were established 60 years ago mostly by men, all of them white, and most of them dead
Related articles
Fad or Foible: MR Trends by Bernie Malinoff #NetGain8 #MRX
Live blogging from MRIA’s #NetGain8 conference in Toronto. Any errors or stupid jokes are my own.
“Fad or Foible” MR Trends Affecting the Industry, and Skill Set Needs To Delight Your Client, Bernie Malinoff, CMRP, President, Element54, Montreal
- remember “second life” – just because it’s a shiny new toy doesn’t mean it’s relevant
- researchers tend to be conservative, risk avoiders, it’s a strength to some degree, people trust that we will be disciplined about our work, but this can also hold us back
- we used to be data poor, now the problem is data obesity – Hal Varian, Google
- [Bernie has written out tweets on our slides that we can write into twitter, now isn’t that thoughtful 🙂 ]
- don’t be concerned about the person sitting next to you, worry about people who’ve never been to a market research conference and possibly never consider themselves market researchers
- the dirty dozen – are you afraid of gamification, online communities, social media, crowdsourcing, facial analysis
- many emerging technologies are now mainstream
- you can now capture emotions of 43 facial muscles and vocally detected intonations – add that to your basic film plus sound – now you have what i said and HOW i said it. these are off the shelf products you can buy now
- supplier selection is often based on more creative and energetic modes
- researcher of the future is a strategist, synthesizer, method agnostic, story tellers – now it’s use the right method for the research objective, not the tool you’re most familiar with
- blend technology with rigour, find a fit for purpose technology when it’s appropriate
- 3 cases do not make a norm, a new method will not and cannot replace all other methods
- replace fear of the unknown with curiosity
IVR Faster, Cheaper, Robuster by Frank Graves, Anne Crassweller #NetGain8 #MRX
Live blogging from MRIA’s #NetGain8 conference in Toronto. Any errors or stupid jokes are my own.
IVR: Faster, Cheaper and More Robust, Frank Graves, Owner and President, EKOS Research, Ottawa, and Anne Crassweller, President, NADbank Inc., Toronto
- counting counts
- the loss of the long form census was disappointing because that’s a loss of a lot of counting that serves a lot of different cultural and community organizations and groups
- “it’s all about the sample” – that’s what she wants on her tombstone 🙂
- 20% of Canadians live in cell only households – much research excludes these people from research
- clients want yesterday’s data today so they can react in time for this evening
- online panels are not a panacea [uh, does someone really think that?!?!?]
- opt-in non probability panels are not reflective of households, yield low and unknown response rates, subject to coverage, non response, and selection bias that can’t be corrected for, should not report margin of error [and yet we still do]
- need to do dual frame sampling of both landline and cell only
- need better randomization of response categories
- need experimentally optimized call backs with replacement
- to say you use mail says nothing about the quality and details of your processes – one mail study is completely different than every other mail study
- [frank was too passionate in his talk and I missed taking notes on some of it. sorry. you missed out. 🙂 ]
- HD-IVR gets really close to census results on demos and newspaper habits
- key reason is that all adults are recruited and surveyed on the same platform
- there are more differences between research method than between landlinehousehold and cellphoneonly
- modeling up to census numbers doesn’ t work, you need better methods [completely agree! get your sample right from the beginning and you won’t have to “fix” things afterwards]
Self Ethnography by Caroline Fletcher #NetGain8 #MRX
Live blogging from MRIA’s #NetGain8 conference in Toronto. Any errors or stupid jokes are my own.
Self Ethnography-From Push to Pull, Caroline Fletcher, Research Director, Sound Research, Toronto
- why does research get boiled down into boring slides?
- film creates an emotional engagement with research findings
- a 5 to 7 minute can go cross department and cross pollinate, it sings, longer shelf life expectancy, highly efficient
- maybe stop doing one pagers and do a film instead
- people forget to fill out their diaries and fill in all the days at the last minute
- how would you deal with “pretend we’re not here watching you behave”
- provides genuine behaviour without other people watching them, not moments staged for moderator
- respondents feel more relaxed on camera, and offer a little more creativity – homemade slideshows, filming commutes
- people might not open the door to strangers but they will do this
- people might bring you into their thanksgiving day as they carve the turkey or as they receive their letter of being accepted as adoptive parents [damn, that’s making me cry]
- film pain points like putting together ikea furniture, or hooking up the tv [imagine doing a survey “When you were attaching part A to part C, what difficulties did you experience? HA HA HA]
- film allows cultural barriers to be broken, can hear personal private conversations that would never happen with someone outside the culture
- good for documenting journeys where a film crew can’t go, great for pre and post tests, good for focus group conversation
- video is not just to sugar coat a bitter pill
The LoveStats Blog 