I have a favourite research joke. Have you heard it before? It has two possible punch lines and it goes like this.
What’s worse than a pie chart?
Answer 1: a 3d pie chart
Answer 2: several pie charts
I bet you’re rolling on the floor laughing aren’t you! Well, I have a treat for you because that is not the funniest marketing research joke I know. There are several formats of the funniest ever market research joke too so get ready…
We know 40 minute surveys are too long but no one’s ever going to stop writing them. HA HA HA HA HA HA
We know long grid questions are a bad idea but no one’s ever going to stop using them. HA HA HA HA HA HA
We know respondents hate multiple loops but no one’s ever going to stop writing them. HA HA HA HA HA HA
You think these aren’t jokes? I challenge you to prove otherwise. I’ve been in numerous situations where laughter is the standard response to these statements.
I find it infuriating to listen to smart and informed people overtly display a lack of effort to address the problem and laugh it off as silly. They generally feel they have no power. Clients feel they have zero alternatives in writing their surveys and vendors feel clients will take their business elsewhere if they refuse to run a bad survey.
Let me say this. Every single person out there has power to make surveys better. Imagine if we all worked together. Imagine if everyone spoke up against bad surveys. Imagine if everyone took quality surveys seriously. Imagine what would happen to our complete rates. Just imagine.
Live blogged from Nashville. Any errors or bad jokes are my own.
children have more internet access than adults. their homes are littered with devices. they start with a leap-pad and download games for it. have it in the car and it goes everywhere with them. then they get a nintendo. they are in-tune with mobile. they are the first generation to grow up with tech. today’s students are not the people our education system was designed to teach.
classrooms rely on tech early now. clickers for interaction. interactive reading solutions. reading apps. smart boards instead of chalk boards. many schools have some iPads as standard in the classroom.
designing surveys for kids. we are working on agnostic and respondent friendly surveys. but we rarely place focus on survey design for kids, especially when focused on mobile.
Do kids really go onto the computer for 30 minutes to answer a survey? [My response – HA HA HA HA HA HA HA. Oh sorry. No, i don’t think so. ]
They did qual and quant to figure out how kids think about and use surveys.
– parents are not concerned with parents using their phone
– kids prefer less than ten minutes
– age 11 to 17 say they rarely use computers!!!
– children read every single question and respond very carefully
– easy concepts may actually be difficult for them
– testing is critical
– responses need to be different to avoid confusion
– less wording is essential
– more engaging question types are easier for them to understand
– simplified scales are more easily processed, maybe using images
– use more imagery, bigger buttons
[this is funny – dear 4 year old – how likely are you to recommend this product to your friends, family, and colleagues?]
– kiddie fingers aren’t as precise with hitting buttons especially when survey buttons are close to phone buttons
– kids don’t understand our concepts of new, different, intent, believability
– kids up to age ten are much more likely to get help from their parent 60% or more, falls to 15% with older teens
– a pre-recruit is helpful, then send the official invite/portal, then again get parental permission
– response rates are higher on tablets, smartphones next, computers worst
– LOI is longer on smartphones, best on computers
– people on smartphones felt there were too many questions
– click rates vary by device but the end conclusions are the same [cool data here]
– ideal length is around 10 minutes
– 3 point scales may be enough [halleluja! Do we TRULY need ten or five point scales in marketing research? i think in many cases it’s a selfish use not a necessary use.]
I challenge you to rethink your behaviours. I challenge you to jump off that pedestal of marketing researchers are more ethical than other people in the marketing world and think about whether you’re being as ethical as you like to think you are. I challenge you to:
1) tell people that answering your survey or participating in your focus group might make them sad or uncomfortable or angry
2) recognize that benign questions about age, gender, income, brand loyalty, weather, and politics make people unhappy, uncomfortable, and angry
3) incentivize people when they quit a survey partway through especially when a question may have made them uncomfortable
4) allow people to not answer individual questions but still complete the entire survey
5) debrief people at the end if surveys by sharing some details about how the results will be used to make people happier via better products and services
Can you hold yourself to a higher standard? Can you start right now?
Day two of WAPOR has come and is nearly gone, but my brain continues to ponder and debate all that I heard today. I hope you enjoy a few of the ramblings from my macaron infested brain.
- People don’t lie on government surveys. Wow. That’s news to me! My presentation focused on how people don’t always provide exactly correct answers to surveys for various reasons – the answer isn’t there, they misread something, they deliberately gave a false answer. But, while people may feel more incentive to answer government surveys honestly, those surveys are certainly not immune to errors. Even the most carefully worded and carefully pre-tested survey will be misread and misinterpreted. And, some people will choose to answer incorrectly for a variety of reasons – privacy, anti-government sentiment, etc. There is no such thing as “immune to errors.” Don’t fool yourself.
- How do you measure non-internet users? Well, this was a fun one! One speaker described setting up a probability panel (i know, i know, those don’t really exist). In order to ensure that internet usage was not a confounding variable, they provided a 3G tablet to every single person on the panel. This would ensure that everyone used the same browser, had the same screen size, had the same internet connection, and more. Of course, as soon as you give a tablet to a non-internet user, they suddenly become….. an internet user. So how do you understand perceptions and opinions from non-internet users. Chicken and egg! Back to paper you go!
- Stop back translating. I don’t work much in non-English languages so it was interesting to hear this one. The authors are suggesting a few ideas: questionnaire writers should write definitions of each question, preliminary draft translations should be provided by skilled translators, and finally, those two sets of information should go to the final translator. This is how you avoid “military rule” being translated as “role of the military” or “rules the military has” or “leadership of the military.” Interesting concept, and I’d love to know whether it’s efficient in practice.
- Great presenter or great researcher: Pick one. I was reminded on many occasions today that, as a group, researchers are not great presenters. We face the screen instead of the audience, we mumble, we read slides, and we speak too quietly. We focus on sharing equations instead of sharing learnings, we spend two thirds of the time explaining the method instead of sharing our insights. Let’s make it a priority to become better speakers. I know it won’t happen over night but I’ve progressed from being absolutely terrible to reasonably ok in a short matter of just 15 years. You can do it too.
- Peanut Labs Ask-Me-Anything with special guest Jim Bryson (web.peanutlabs.com)
- WAPOR Day 1: Why are Google, Facebook, and Microsoft so far ahead of us in research? #MRX (lovestats.wordpress.com)
- The Conference Presenter Gender Gap #WAPOR #AAPOR #MRX (lovestats.wordpress.com)
- The Bakery Review: A Daily Blog of the Macarons of Nice, France (lovestats.wordpress.com)
- America: You aren’t representative of the world #MRX #WAPOR (lovestats.wordpress.com)
Innovation in Web Data Collection: How ‘Smart’ Can I Make My Web Survey?” by Melanie Courtright, Ted Saunders, Jonathan Tice #CASRO #MRX
“Innovation in Web Data Collection: How ‘Smart’ Can I Make My Web Survey?”
- The proportion of respondents taking surveys on tablets and mobile phones continues to increase
- Researchers are exploring ways to improve data accuracy and the respondent experience on mobile surveys
- Mobile or touch-screen devices enable different ways of interacting with the respondent to capture responses
- Programmers used to developing mobile applications may naturally want to extend such features to surveys, and researchers may see these features as new and inventive
- Want to use a randomized experimental design to see how data quality and respondent experience is affected
- Web-based survey fielded April 2014; Auto insurance satisfaction and attitudes; 3,600 respondents, 10-minute survey, 60 questions; Respondents self-selected into PC, mobile phone and tablet cells; Optimized for Tablet and Mobile
Slider start position influences “passive” responses
- Respondents instructed to click on the slider button
- Sliders were programmed to record a lack of use as missing
- People are generally satisfied with their auto insurance company, so saw a large amount of non-use of the Right slider starting position
- Slider start position matters more on touch devices
- The right starting position tended to bias mean scores upward
- PC users using a mouse were not as affected by the slider start position as much and had much less passive use of the slider on both 5- and 11-points
- Touch device users who used sliders liked them
- Respondents generally preferred the type of scale they used throughout the survey
- PC users preferred Standard and had No Preference more.
- Touch device users had slight preference for Sliders when they used them
- Sliders may be suitable for continuous objective measures
- Tests so far have been on attitudinal response scales.
- Sliders may be more appropriate for entering an objective value
- Prevents touch users from having to type
- Responses similar to those entered into a text box
- Length of the list matters more than style of list
- Respondents were asked to give a time from 3 randomly assigned list lengths (of different granularities), on either a Radio button list, or a Drop-down list.
- The longer the list, the more likely respondents chose a time earlier on the list, regardless of how the list was presented.
- The display of Drop-down lists varies by browser. The Safari browser is dominant on iPhones, but browsers vary more on Android phones.
Android browser differences result in primacy effects. Because the default Android browser only shows the first three choices on the list and doesn’t easily scroll, those choices where selected much more often when shown in a drop-down.
- Unprompted use of Voice-To-Text is very low. Even when asked, most respondents didn’t use it. About 90% of Tablet users opted to type in their response, either because they didn’t have the functionality, or didn’t want to use it. Mobile users were more evenly split between using it and not wanting to
- Don’t want to use it because of heavy accents for hispanic people or asian people, need to be quiet, won’t be as accurate, environment was too loud
- premature to recommend it on a wide basis but people are becoming more familiar with itRespondents using Voice-To-Text gave slightly longer answers
- Using Image or Video Capture on mobile devices – there are many differences by OS, browser, and screen size which will affect results
- request for a generic image of “where you are” led lots of feet, selfies, and friends – literal definition of “where you are” :)
- environments ranged from home, school, church, beach, office, bars, labs, gyms, cars, hospital, kitchens, bathrooms, airports, malls
- It’s in-experience data without any delay
- Picture quality instructions recommended: Blurry, bad lighting and more
- These are not professional photographers. Some, but very limited, importing of existing pictures. Question wording is critical. Photo size is a double-edged sword. Internet connection speed and latency is worth considering
- reason for not using was it seemed intrusive but it was all by full permission
- There were fewer “easy” ratings for tablets vs phones
- it is candid, personal, and open, it is in the moment and in-context, no geographic limitations, no real tech issues
- But, the data needs to be reviewed individually, and it doesn’t work on non-smartphones
- still need to test a lot and educate people on why and how to do it. Yet, consumers are still quite ahead of us when it comes to tech
- consider rating every survey on it’s mobile friendliness – open ends, length, LOI, scale lengths, grid lengths, use of flash, rich media, audio or video streaming which add bandwidth, responsive design – all contribute to whether a study will work well on a phone. consider incenting CLIENTS for mobile friendly surveys
- also consider designing every survey from stage 1 for mobile phones as opposed to adapting a web survey to phone
A “How-To” Session on Modularizing a Live Survey for Mobile Optimization by Chris Neal and Roddy Knowles #FOCI14 #MRX
A “How-To” Session on Modularizing a Live Survey for Mobile Optimization
Chris Neal, CHADWICK MARTIN BAILEY
& Roddy Knowles, RESEARCH NOW
- conducted a modularized survey for smartphone survey takers, studied hotels for personal travel and tablets for personal use, excluded tablet takers to keep the methodology clean
- people don’t want to answer a 20 minute survey on a phone but clients have projects that legitimately need 20 minutes of answers
- data balanced and weighted to census
- age was the biggest phone vs computer difference
- kept survey to 5 minutes, asked no open ended questions, minimize the word count, break grids into individual questions to avoid burden of scrolling and hitting a tiny button with a giant finger
- avoid using a brand logo even though you really want to. space is at a premium
- avoid flash on your surveys, avoid images and watermarks, avoid rich media even though it’s way cool – they don’t always work well on every phone
- data with more variability is easier to impute – continuous works great, scale variables work great, 3 ordinal groups doesn’t work so well, nominal doesn’t work so well at all
- long answer options lists are more challenging – vertical scrolling on a smartphone is difficult, affects how many options responders choose, ease of fewer clicks often wins out
- branching is not your friend. if you must branch, have the survey programmers account for the missing data ahead of time, impute all the top level variables and avoid imputing the bottom level branched variables
- Predictive mean matching works better than simply using a regression model to replace missing data
- hot decking (or data stitching which combines several people into one) replaces missing data with that from someone who looks the same, worked really well though answers to “other” or “none of the above” didn’t work as well
- hot decking works better if you have nominal data
- good to have a set of data that EVERYONE answers
- smartphone survey takers aren’t going away, we need to reach people on their own terms, we cannot force people into our terms
- we have lots of good tools and don’t need to reinvent the wheel. [i.e., write shorter surveys gosh darn it!!!]
Ah, yet another enjoyable set of sessions from #AAPOR, chock full of modeling, p-values, and the need to transition to R. Because hey, if you’re not using R, what old-fashioned, sissy statistical package are you using?
This session was all about satisficing, burden, and data quality and one of the presenters made a remark that really resonated with me – when is burden caused by responders. In this case, burden was measured as surveys that required people to extend a lot of cognitive ability, or when people weren’t motivated to pay full attention, or when people had difficulty with the questions.
Those who know me know that it always irks me when the faults of researchers and their surveys are ignored and passed on to people taking surveys. So let me flip this coin around.
- Why do surveys require people to extend a lot of cognitive ability?
- Why do surveys cause people to be less than fully motivated?
- Why do people have difficulty answering surveys?
We can’t, of course, write surveys that will appeal to everyone. Not everyone has the same reading skills, computer skills, hand-eye coordination, visual acuity, etc. Those problems cannot be overcome. But we absolutely can write survey that will appeal to most people. We can write surveys with plain and simple language that don’t have prerequisites of sixteen Dicken’s novels. We can write surveys that are interesting and pleasant and respective of how people think and feel, thereby helping them to feel motivated. We CAN write surveys that aren’t difficult to answer.
And yes, my presentation compared data quality in long vs short surveys. Assuming my survey was brilliantly written, then why were there any data quality issues at all? :)
This afternoon, I attended a session on how the number of survey contacts affects data quality and results equivalence this afternoon. I just loved the tables and stats and multicollinearity. Many of my hunches, and likely your hunches were confirmed and yes overly obvious.
But something bothered me. As cool as it is to confirm that people who are reluctant to participate give bad data and people who always participate give good data, it irked me to be reminded that our standard business practice is to recontact people ten times. 10. TEN. X.
Have we conveniently ignored various facts?
– people have call display. When they see the same name and number pop up ten times, they learn to hate that caller. And the name associated with that caller. And that makes our industry look terrible.
– half of people are introverts. A ton of them let every call go to voicemail which means we are pissing them off by calling them ten times in a few days. Seriously pissing them off. I know. I’m a certified, high order introvert.
– I like to listen to what people say about research companies online. People DO search out the numbers on their call display and identify survey companies. Even if you use local numbers to encourage participation. And yet again, this makes us look bad.
Why do we allow this? For the sake of data integrity? Hogwash. It’s easy for me.
Do we care about respondents or not?
Do Smartphones Really Produce Lower Scores? Understanding Device Effects on Survey Ratings by Jamie Baker-Prewitt #CASRO #MRX
“Do Smartphones Really Produce Lower Scores? Understanding Device Effects on Survey Ratings”
As the proliferation of mobile computing devices continues, some marketing researchers have taken steps to understand the impact of respondents opting to take surveys on smartphones. Research conducted to date suggests a pattern of lower evaluative ratings from smartphone respondents, yet the cause of this effect is not fully understood. Whether the observed differences truly are driven by the data collection device or by characteristics of smartphone survey respondents themselves requires further investigation. Leveraging the experimental control associated with a repeated measures research design, this research seeks to understand the implications of respondent-driven smartphone survey completion on the survey scores obtained.
- Jamie Baker-Prewitt, SVP/Director of Decision Science, Burke, Inc.
- Tested four devices for data quality and responses
- Brand awareness was not significantly different
- Brand engagement – trust, financially stable, value, popular, proud, socially responsible – did show differences. PC users had higher ratings. Smartphone takers had lower ratings.
- Customer engagement – purchase, recommend, loyalty, preference – half of tests showed significant differences. PC users had higher recommend scores and smartphone takers had lower recommend scores.
- Different topics and sources all suggested that devices cause lower ratings
- Did a nice repeated measures design with order controls
- Frequency of purchasing looked the same on both devices, average cell phone bill showed no differences [interesting data point!]
- No differences on brand engagement – 1 out of 30 was significant [i.e., the 5% error rate we expect due to chance]
- Purchase data looked very similar in many cases for PC vs phone, frequency distributions were quite similar
- Correlations between PC and phone scores were around .8, which is very high [recall people did the same survey twice, once on each device]
- Current research replicates original research, no significant device effect. Did not replicate lower scores from smartphones.
- Study lacked mundane realism, they were in a room with other people taking the survey, there weren’t ‘at home’ distractions but there were distractions – chatty people, people needed assistance, people might have simply remembered what they wrote in the first survey
- Ownership of mobile will continue to grow and mobile surveys will grow
- Business professionals are far more likely to answer surveys via mobile, fastfood customer are more likely to use smart phone for surveys
- Very few people turned the phone horizontally – they could see less screen but it was easier to read. Why not tell people they CAN turn their phone horizontally.
Total Survey Error is a relatively recent approach to understanding the errors that occur during the survey, or research, process. It incorporates both sampling errors, non-sampling errors, and measurement errors, including such issues as specification error, coverage errors, non-response errors, instrument error, respondent error, and pretty much every single other error that could possibly exist. It’s an approach focused on ensuring that the research we conduct is as valid and reliable as it can possibly be. That is a good thing.
Here’s the problem. Total Survey Error is simply a list. A list of research errors. A long list, yes, but a list of every error that every researcher has been trained to recognize and account for in every research project they conduct.
We have been trained to recognize a bad sample, improve a weak survey, conduct statistics properly, generalize appropriately, and not promise more than we can deliver. Is conducting research the old name of ‘total survey error?’ It is not a new, unique approach. It does not require new study nor new books.
Perhaps I’m missing something, but isn’t total survey error how highly skilled, top notch researchers have been trained to do their job?