Questionning the questionnaire – using games to real self-report biases by Amber Brown and Joe Marks #CASRO #MRX
Live blogging from Nashville. Any errors or bad jokes are my own.
– surveys that aren’t well designed have social desirability bias, aspirational biases, demand characteristics, satisficing
– games can help with some of these if they are properly designed
– purchase/visit intent can have problems as people want to please you, are aspirational in their answers with little follow through, similar to charitable giving and exercise
– study asked about prior and future behaviour of behaviours
– people were offered either cash or theme park tickets and then asked whether they planned to visit the park – would they take the cash (they probably won’t go) or would they take the tickets (they probably will go) (Cash is always less)
– for a charity company – will you donate your incentive to a charity or take the cash (cash is always less)
– for an exercise company – will you take a sports authority gift card or a cash incentive (cash is always less)
– for readership – will you take a book store gift card or cash
– the incentive choice was a good predictor of the intent question
– games engage instinctual thinking. you’re just trying to win. people play games every day. it’s faster and gives less time for biases to creep in
– the test is actual choice behaviour which his similar to the marketplace
– would you be willing to donate to wikipedia? real case study – do you want $10 in cash or donate $50 to wikipedia. 14% chose the 10$ donation but 2% chose the $30 donation
– the game comes much closer to real behaviour
– can help to counter biases that poorly designed surveys may have
[i want to read the paper on this one. very cool!]
Emerging Technologies – Are They Still Emerging? Lenny Murphy, GreenBook Blog and GRIT Report #NetGain2015 #MRX
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
Emerging Technologies – Are They Still Emerging?
Lenny Murphy, Editor-in-Chief of GreenBook Blog and GRIT Report
- Attitudinal, behavioural, and intrinsic data
- Foundational research is no longer taking months but hours
- Moving from questioning to discussing, from asking to observing, from data to insight, from understanding to predicting, from the big survey to data streams, from rational to behavioural, from quarterly to real time, from siloed to converged
- the traditional survey as the primary driver of information will decline
- Data science is not a hoity toity term for a statistician. It’s information technology and algorithms and languages and hadoop and R. It’s statistics on steroids.
- The future looks very different.
- Over the next five years, we are in the realm of DIY, non-conscious measurement is emerging such as facial scanning and automated emotion measuring, automation and AI in terms of very very smart devices, internet of things where all of your things will collect and share data from your shoes to your car, virtual and augmented reality will change our media habits
- DIY – there are many free DIY tools
- The ‘make it’ revolution – consumers can ‘print’ their own things, print some shoes, do an ideation session using a printer. cost of these devices can be as low as $100.
- Emotional measurement – facial scanning, shopping behaviour videos, eye tracking
- AI – tons of money going here, google has spent millions on quantum computers, these will just be part of everything we do
- Internet of Things – Internet as we know it might disappear. Daily lives are just always all connected. e.g., Microsoft’s hololense.
- Do a virtual shopping experience without a computer. But you still feel like you are in the store.
- Imagine a connect fridge [will it shop for me once it notices I’m out of BREAD AND MILK!!]
- Google Glass succeeded in every aspect they hoped. The real product will come out in the next couple of years.
- Gamification has never taken hold but many companies are working in this area. Game to map out neurons.
- Which companies will be our competitors for clients and budget? Google, IBM, Apple, facebook, AOL, Verizon, Comcast, Disney, at&t, GE, groupm, WPP, amazon
I did it. Yes. I broke down and spent my Christmas money. Let’s put aside the fact that I still get Christmas money from the moms and move on to what I spent it on.
In just six to eight weeks, this pretty little plum coloured Fitbit will arrive at my door. (The “make it pink so girls will buy it” marketing scheme works on me but plum is just as good.)
Supposedly, it will monitor my heart rate all the time including when I am awake and asleep. It would have been cool to have it a few weeks ago when my four wisdom teeth were ripped out of my face but I’m sure some other quite unpleasant event will greet me soon enough.
I’m quite looking forward to learning:
– how consistent my sleep is, and how many times I wake up at night
– whether my heart rate speeds up or slows down when I get ready for work or leave work, or when I go toy awesomely fun ukulele class
– how incredibly nuts my heart rate is when I speak at conferences, show up at cocktail hour, plow through a crowded exhibit hall. Though I may seem calm and relaxed, it really takes a ton of mind games to turn quiet me into loud me.
And at the same time, I’ll be wondering… If someone gets their hands on my data, what will they do with it? What products will they develop as they learn about me? What heart rate medications will they need to sell to me? What fitness products will they need to sell to me? Will I need to buy the shirt version to measure electrical outputs? The sock version to measure sweat outputs? The earbud version to measure brainwaves? What will marketers and brand managers learn about me and my lifestyle?
Now that I think about it, this is MY form of gamification. I can’t wait to see charts, watch trends, and compare Norms. And now that I’m learning Python and rstats, I would love to get my hands on the dataset of millions of people and millions more records. With permission of course.
Epic Win! Successful Gamification in Marketing Research by Briana Brownell and Carl Gutwin #MRIA14 #MRX
Live blogging from the #MRIA national conference in Saskatoon, Saskatchewan. Any errors or bad jokes are my own.
Epic Win! Successful Gamification in Marketing Research
Briana Brownell, Manager, Analytics, Insightrix Research Inc.
Carl Gutwin, Professor of Computer Science, University of Saskatchewan
- 1- gamify to increase engagement, encourage people to do things they wouldn’t necessarily do otherwise
- 2 – get something new, get more information from each person, information that wasn’t available via a survey
- 3- reach the unreachable, people who might not take a survey
- spent half the time of the project just coming up with a good idea to turn into a game
- what is the core task – fundamental action you want the user to perform – their age, count the light bulbs in their house, need to match that action with the survey and the game
- add around that some game mechanics – the process of playing the game, and the game itself
- most game mechanics aren’t compatible with surveys – points and scores work. but survey needs a truth, a valid answer not an imagined outcome. you can’t let people choose which questions they want and in what order.
- Game 1: research game show for an ad test. could they identify the sponsor, identify slogans and products, did they understand the purpose of the ad [makes me wonder – if we’re encourage people to pay attention to an but they never actually pay attention to ads, what exactly are we measuring]
- added time pressure, cheers and jeers for correct and incorrect answers, music
- where is the barrier between survey and game?
- found two mistakes – in the full game, more people were getting it right, the feedback mechanism caused people to learn when they were getting right and wrong answers, refielded without the problematic feedback. smaller difference found for a time pressure section which made people give shorter responses, an undesirable outcome.
- it was motivating. people were asked if they wanted to quit or continue. Difference of 12 vs 21 “would you like to continue, traditional vs gamified version.
- no differences by gender, age, tenure, or gaming experience
- Game 2: shopkeeper for a choice based conjoint. Game was little people coming in your shop and you need to help them pick a product. Also tried to do it with a little alien man on another planet – spacemonsters.
- major questions – was the data different, was the game motivating, was the possibility of getting incorrect answers motivating, did the games realism affect the results. There ISN’T a correct answer to a survey question so need to work this out.
- The data was not different at all – when people pick the best value, it doesn’t matter if they pick for themselves, for other people, or for aliens
- The realism did not affect the results. equally motivated in all scenarios.
- The possibility of getting incorrect answers didn’t affect it.
- Was the game motivating? actually, it was demotivating. They did 47 extra games the traditional way but only 37 with the game version. It was too different from reality.
- is it worth it too add gamification? make sure you match the task with the game mechanism first.
- don’t make the game too gamey
- think about auxiliary data – what can i get that I can’t get elsewhere
Gamijoint: Improving Conjoint Data with Gamification”
|The research industry has begun to witness the benefits of gamification in different research environments, but not yet in conjoint analysis. In this research, the authors tested elements of game mechanics in a role-playing conjoint exercise and mapped its results to choice-based and adaptive choice-based conjoint. Do gaming elements improve respondent engagement and data quality? In addition to the presenters, this paper was authored by Mohit Shant (AbsolutData Research & Analytics Solutions).
- Market research is 90 years old and we know it pretty well. so do our responders and they know what to expect. We need to do something new to get engagement back.
- Need to add fun, adaptive, human element, motivation, and rewards to surveys – gamification. This is not new.
- We use gamification everywhere except conjoint. Conjoint, which is complex, can lead to fatigue, bad data, speeding, straightlining.
- They added a story & fun to the survey. Used language intelligently as if it was a real person talking. Rewards were instant. Had users create an avatar and pretend to make decisions for their organization. Received performance feedback along the way.
- US and India, 150 respondents in each cell – choice based conjoint, adaptive choice based conjoint, gamified conjoint, gamified timed conjoint. Considered CBC the benchmark as it is used most often.
- [makes me wonder if there is a subset of people who perform better on regular surveys compared to gamified surveys. I always tune out of “gamified” or “fun” tasks because “just get to the point and ask me!”]
- Model fit looked similar for the four techniques. So gamification did not distract.
- Hold out accuracy was good for all but better for ACBC
- Mean absolute error was similar for all
- Enjoyability and satisfaction were significantly better for gamijoint, ease of understanding was the same for all
- Similar results US vs India, but internet in India had a harder time supporting the requirements
- People appreciated the feedback, people took more than a minute to customize their avatar
- The timed vs non-timed didn’t work as the timed portion didn’t give people a chance to finish their task
- Responders thought it was different, good, interesting, liked, unique, delighted [as seen in wordcloud]
- [Another content guru presentation, as most at CASRO digital have been. awesome 🙂 ]
- Peanut Labs Ask-Me-Anything with special guest Tom Ewing
- Peanut Labs Ask-Me-Anything with special guest Kristin Luck
- What is a convenience sample?
- What does plus or minus 3% 19 times out of 20 mean?
- Short answer lists inflate endorsement rates
- What is Vue magazine? #MRX
- CASRO in San Antonio: The fun so far #MRX #CASRO
There are two kinds of annoying eaters: those who hate practically everything (that’s me) and those who like practically everything (that’s my SO). The problem with picky eaters is obvious – it’s nearly impossible to make meal that includes only things they like.
The problem with the other people is less obvious. No matter what you make, they like it. If you burn it, they like it. If you add no seasoning, they like it. If it looks like dirt and slime mixed nicely together, they like it. Which means that if you ask them what their favourite food is so you can make it for their birthday, they have no idea what to tell you. That‘s the problem.
To try and get around this problem, I used to ask about everything I made, “Do you like this?” Eventually, I realized that was a useless question. I got no variability with the answers. Everything was a yes. Everything was good, nothing was bad. Like I believe that. As if everything I cook is actually good. (Desserts yes, non-sugar foods no way!)
So I tried a different question. “On a scale from 1 to 10, how would you rate this.” Sadly, that well-designed question with decades of rigorous experimentation behind it also failed. Nearly everything I made was rated as a 7 or 8. Which we already know is inherently not valid.
But recently, with the gamification of surveys clearly in mind, I deliberately asked a different question. One that was designed to create an emotional responses. I asked, “If I never made this again, would you care?” (Try answering that question while thinking about asparagus, fudge, anchovies, and Twinkies.) And I started to get different answers. Differentiated answers. As in some foods would be missed if I never made them again while others would not be missed at all. I had found the holy grail of eliciting responses that better suited my research objective.
I know we’ve been hearing about the gamification of surveys a lot lately. I can’t say that I used a gamified survey, but a blatant rethink of my survey question surely helped solve a simple problem for me. Imagine how rethinking the questions on your surveys would better answer your important questions.
(n=1, ABC non-randomized design, No spouses were harmed while conducting the experiment.)
- Validity of Gamification: Sweeney, Goldstein, and Becker #CASRO #MRX (lovestats.wordpress.com)
- Shorter isn’t always better: Inna Burdein #CASRO #MRX (lovestats.wordpress.com)
- Do I have your attention? By Pete Cape #CASRO #MRX (lovestats.wordpress.com)
Will You Marry Me? – Exploring the Validity of the Gamification of Research”
By Terry Sweeney, Vice President, Operations and Client Services, Cross-Tab; Dan Goldstein, Chief Strategy Officer, DB5; Steve Becker, Vice President Brand Strategy, DB5
- Does a gamified approach provide greater insight [than what?]
- AIDA – Attention, Interest, Desire, Action (developed in 1898)
- Seduction model – Catching the eye, emotional involvement, courtship, consumation, relationship; Matches up with Brand fame, emotional connection, changed or reinforced belief, behavioural response, brand relationship
- (Don’t confuse habit with a relationship, e.g., as you purchase a brand of soap)
- Approach to this research – asking survey questions, engaging survey with flash to animate it, gaming approach to same questions (think of gaming as using a little basket to catch the brands you like or using a hammer to smash the brands you hate)
- The more engaging the survey, the more sustained interest from the responder
- People felt the gaming version was different, more interesting, and more fun
- dropout rate also lower for gaming survey even though the time spent on it was longer
- Adding game like elements can help responders engaged – a “mental sorbet”
- More engaging surveys show more elevation, e.g., more extreme opinions [is that a data quality issue?]
- Use gaming to separate out brands that are clustered in one category, interchangeable with similar trends, helps increase differences between brands
- More engaging means increased passion, e.g., yes I like something but if you ask when someone’s really excited, they’ll like the thing a lot more
- All of these differences are true, they’re just in a different context
- We must measure the differences that are relevant to the researcher, seek the answer to your real question
- Lower engagement of surveys create high cost of research
- Cyborgs vs Monsters in modularizing surveys: Edward Paul Johnson and Lynn Siluk #CASRO #MRX (lovestats.wordpress.com)
- Shorter isn’t always better: Inna Burdein #CASRO #MRX (lovestats.wordpress.com)
- DIY Panel: Gardlen, Ribeiro, Smith, Terhandian, Thomas #CASRO #MRX (lovestats.wordpress.com)
- Perfecting Social Media Segmentation: Margie Strickland #CASRO #MRX (lovestats.wordpress.com)
- Mobile and CAWI Parallel: Frank Kelly and Sherri Stevens #CASRO #MRX (lovestats.wordpress.com)
- Data Privacy: Gina Pingitore and Kristin Cavallaro #CASRO #MRX (lovestats.wordpress.com)
- Combining Mobile, Social and Survey: Carol Haney #CASRO #MRX (lovestats.wordpress.com)
- Do I have your attention? By Pete Cape #CASRO #MRX (lovestats.wordpress.com)
… Live blogging from downtown Toronto…
DataMining and Integration
Melanie Courtright, ResearchNow and Prince De, Conversition
Digital Remix: A well-rounded global story about consumer behaviour and the ‘Digital Olympics’
- Online surveys plus social media listening plus web behavioural data plus mobile survey
- Survey data gave us answers to what they knew they wanted information about
- Listening and web-behavioural methods gave us answers to questions we didn’t know we had
- Findings from one data set led them to look for similar kinds of information in other datasets
- You can’t change the question set of a survey everyday, if you forgot to ask a question you’re out of luck. But it’s not the same for listening and mobile.
- Mobile sample skewed slightly younger which is great because they are less likely to sign up for panels, BUT the end data provided the same results
- Start with a question at all times and don’t get bogged down by everything that’s available
- Social media data volumes matched exactly with the start and stop of the olympic games, similarly website hits of various athletics sites corresponded with the olympics
- In the online space, people are more likely to simply talk about “apps” instead of naming them. But the name of the app was available in the behavioural data. [makes me think people don’t care WHAT app they use]
- Nike had the highest sponsorship awareness even though they weren’t official sponsors. Discovered in surveys and confirmed in social media data.
- Also didn’t anticipate tape-delay issues in time for the survey yet this was picked up in social media
- One method answered questions and raised questions that were answered by the second or third method
- Social media meant that they didn’t have to fill survey with cumbersome open ends
- The four toughest questions you must ask #MRX (lovestats.wordpress.com)
Gamification in market research is becoming more and more popular. People like fun, people like games, it’s a great way to increase participation in market research. But is gamification really taking off or is the idea of gamification really taking off?
I suspect many people are interested in taking advantage of it, but it’s hard to actually implement. Turning a survey into an interactive game requires time, money, and a new sense of creativity that many of us wish we had but know we never will. Some forms of gamification turn quantitative surveys into large scale qualitative studies that require large scale qualitative analysis, another skill that many of us don’t have.
Now don’t get me wrong. Where a study should be conducted via gamification because that is the best way to get the best data, then neither time nor money should hold you back from conducting the study that way.
But for the rest of us, all is not lost. Quantitative survey writers CAN learn from the gamification paradigm. Think about the kinds of questions that normally appear on a survey and then think about what they could look like in a gamified study. One is boring and one is fun. More fun = better engagement. Better engagement = better quality data. Consider the following pairs of questions.
How likely are you to recommend the following brands to friends and family?
If you were having a dinner party, which of these brands would you serve to your friends and family?
How likely are you to purchase each of the following brands?
If you had only $10 in your wallet, which of the following brands would you buy?
How new and different are each of the following brands?
If a friend asked to try a food they’d never had before, which of the following brands would you give to them?
Have you purchased any of the following brands?
Which of the following brands are in your fridge right now?
Yes, the questions are very different. Yes, the questions will elicit different results. Yes, the questions still follow the same general format as a regular boring quant survey.
But, the questions will be new to responders, more interesting, more engaging, and they won’t elicit the rote thought that most questions do. They will elicit more careful thought, something I suspect you want. So the next time you’re writing a survey, consider doing something new. Consider putting a little thought into a brand new style of questions.
- 3 Problems with Gamification in Market Research #MRX (lovestats.wordpress.com)
- The Top 10 Things We Love About Social Media Research #MRX (lovestats.wordpress.com)
- Do Google Surveys use Probability Sampling? #MRX #MRMW (lovestats.wordpress.com)
- An open letter to Scotiabank and ING #MRX (lovestats.wordpress.com)