“Do Smartphones Really Produce Lower Scores? Understanding Device Effects on Survey Ratings”
As the proliferation of mobile computing devices continues, some marketing researchers have taken steps to understand the impact of respondents opting to take surveys on smartphones. Research conducted to date suggests a pattern of lower evaluative ratings from smartphone respondents, yet the cause of this effect is not fully understood. Whether the observed differences truly are driven by the data collection device or by characteristics of smartphone survey respondents themselves requires further investigation. Leveraging the experimental control associated with a repeated measures research design, this research seeks to understand the implications of respondent-driven smartphone survey completion on the survey scores obtained.
- Jamie Baker-Prewitt, SVP/Director of Decision Science, Burke, Inc.
- Tested four devices for data quality and responses
- Brand awareness was not significantly different
- Brand engagement – trust, financially stable, value, popular, proud, socially responsible – did show differences. PC users had higher ratings. Smartphone takers had lower ratings.
- Customer engagement – purchase, recommend, loyalty, preference – half of tests showed significant differences. PC users had higher recommend scores and smartphone takers had lower recommend scores.
- Different topics and sources all suggested that devices cause lower ratings
- Did a nice repeated measures design with order controls
- Frequency of purchasing looked the same on both devices, average cell phone bill showed no differences [interesting data point!]
- No differences on brand engagement – 1 out of 30 was significant [i.e., the 5% error rate we expect due to chance]
- Purchase data looked very similar in many cases for PC vs phone, frequency distributions were quite similar
- Correlations between PC and phone scores were around .8, which is very high [recall people did the same survey twice, once on each device]
- Current research replicates original research, no significant device effect. Did not replicate lower scores from smartphones.
- Study lacked mundane realism, they were in a room with other people taking the survey, there weren’t ‘at home’ distractions but there were distractions – chatty people, people needed assistance, people might have simply remembered what they wrote in the first survey
- Ownership of mobile will continue to grow and mobile surveys will grow
- Business professionals are far more likely to answer surveys via mobile, fastfood customer are more likely to use smart phone for surveys
- Very few people turned the phone horizontally – they could see less screen but it was easier to read. Why not tell people they CAN turn their phone horizontally.