Best of #Esomar Canada: Jon Puleston Games a Better Survey #MRX

esomar logoThis is one of several (almost) live blogs from the ESOMAR Best of Canada event on May 9, 2012. Any errors, omissions, and ridiculous side comments are my own.

Jon Puleston, VP GMI

Creative survey design and gamification

  • We’ve taken a journey of massive pages of boring hard to read text to simple clear phrasing in 140 characters
  • Now we need to do the same for surveys and that doesn’t mean just throwing a thumbs up picture on your survey. (Massive giggling in my head. I know that tactic extremely well!)thumbs up
  • Survey writers are competing against Twitter and Farmville and that’s why we see a horrible decline in completion rates
  • We need to see surveys as a creative art form but we are still at the 1980’s whacky clip art and funky page transition stage
  • We need to let respondents give feedback in the way they want
  • A typical survey starts with a terribly boring long question of precisely what we want to know. We forget just how important foreplay is (ah yes, joke intended)
  • Think about survey design as a television interview with a celebrity. Would you ask George Clooney “On a scale from 1 to 10, how much did you enjoy making that last film?”
  • A first bad question means respondents will have no respect for the rest of the questions.
  • Imagery is very important and yet we just slap on a really dumb smiley face or thumbs up. Why haven’t you used a design firm to do this? It’s not about making it look nice. It’s about communicating effectively.
  • Must grids dominate every survey? They don’t capture the imagination of respondents.
  • The average person spend 4.3 seconds considering their answer to a question. For a grid, it drops down to 1.7 seconds.
  • We are researchers and yet we don’t use research to design better surveys. We rarely pilot surveys any more even though it’s extremely easy to pilot an online survey. You can get 15 completes and make tweaks all within an hour.
  • Now for a game. Jon took two volunteers who weren’t as familiar with gamification.
  • First women was asked a series of ordinary questions. Tell me about toronto. What’s your favourite meal. What’s in your fridge. What would you do with a brick. Jon responded with “um”, “ok”, and stared at his notes most of the time. Then he complained that she was too creative and ruined his demonstration. (Yup, that’s how live demos ALWAYS work. 🙂 )
  • Second person was then brought into the room and asked about his family, to write a postcard home to his family, to imagine he’s been convicted and is on death row and has to describe his last meal, to name his favourite foods within two minutes and anything not named can never be eaten again, who can think of more uses of a brick.
  • Game conclusion: you can get the same kind of data, but more of it and in more detail if you are creative.
  • The approach you use to ask a question can enthuse and excite people to give answers.
  • Book on gamification “Reality is broken”
  • Check out website
  • What defines a game: Anything that is thinking and that we do for fun. Games have rules, skill and effort.
  • Gardening is a popular game for older age groups. There are rules for planting seeds, skill and effort to get a wonderful garden. (heeeey, was I just called old?)
  • Twitter is a game that tries to get you to say your thought in a short space and then get the message out
  • Email is a game: do you get the feedback you wanted
  • Surveys are games…… just rather dull ones
  • In a face to face interview, people can’t just turn their backs and walk away. We’re still working on that heritage.
  • Don’t ask for my favorite colours – ask what colour you’d paint your room
  • Don’t ask what you want to wear – ask what you’d wear if you were going on telelvision
  • Don’t ask what do you think of this product – ask what would you value this company at
  • Describe yourself vs describe yourself in exaclty 7 words. It’s liberating and you’ll actually get more information.
  • How much do you like these music artists – If you owned a radio station, which of these artists would you put on your play list.
  • Add in a competitive element. what brand comes to mind – how many brands can you guess in two minutes.
  • Rewards. Give them points for answering questions correctly. Stake some ‘money’ on what brand reflects this logo. These allow people ot be more circumspect instead of overly positive about everything. Makes it more easy to give a negative answer.
  • The best games encourage us to think and we like thinking.
  • But remember, games can affect data. Greater thought and consideration changes answers. Point scoring can steer data badly as people try to cheat the system. (This is the same with any research. Once you improve the research, the historical norms, whether they are correct or incorrect, are no longer applicable. But… if the survey was horribly boring, just how valid were those results? Be honest with yourself!)

3 responses

  1. Refreshing and inspiring. However when it comes to finances, running such type of research requires plenty of time for the setup, resources and a client with an open mind that is willing to spend more money and time in order to get the just the same data but richer. Considering current market competition and tight budgets, the average customer would not want to opt for this option due to its financial implications. It is a brilliant option to offer but with the caveat of associated costs and set up time that Client’s on a tight budget might not be able to afford.

  2. Very good presentation but I was surprised to hear the speaker say that face-to-face interviews/questionnaires were not pre-tested and didn’t have visual exhibits!!!! Many/most of the suggestions to elicit rich data in online quant using gamification parallel exercises used in focus groups/workshops. What will happen to all that rich diagnostic data in analysis if there is no budget for coding and it is consigned to text analytics programmes?

    1. It was definitely interesting! Who will take up the challenge of more pre-testing….

%d bloggers like this: