Tag Archives: questionnaire design

Shhhh…. A Post in Which We Reveal the Midi-chlorians of Questionnaire Design

This post originally appeared on the Sklar Wilton & Associates blog.

There are no midi-chlorians when it comes to questionnaire design.

Sigh. I’m sad to start the post like that but it’s true. There are no Jedi mind tricks that will make people generate better questionnaire data. There are no sacred texts on the market research version Ahch-To containing that one single piece of advice that will allow someone who’s never written a questionnaire before to create an effective questionnaire that generates actionable outcomes. The only Force at our disposal is careful training as a Padawan and years of experience. Fortunately, as a questionnaire Jedi Knight myself, having years of experience does mean that I can share a few tidbits I’ve learned along the way, tidbits not necessarily found in an academic textbook. So here goes.

spaceQuestionnaires aren’t about grammatically perfect writing: After perhaps two decades of primary, secondary, undergraduate, and graduate school, many of us have learned an abundance of grammar and writing skills that we’ve been told are essential for clear communication. Don’t end sentences in a preposition. Don’t use sentence fragments. Don’t start sentences with ‘and.’ However, as questionnaire writers, we have a very specific goal: To write questions and answers that are understandable to as many people as possible. And sometimes, that means joining the Dark Side and ignoring the rules we’ve struggled to follow for years. With that in mind, when there isn’t a good alternative, it is indeed okay to write questions that end in prepositions!

  • Which country do you live in? [Or better, ask “Where do you live?”]
  • Which of these have you heard of?
  • Which of these have you seen before?

Questionnaires aren’t about professional and formal writing: Of course we want research participants to recognize that the questionnaire they’re completing is important and should be taken seriously. However, formal language can be a deterrent to questionnaire completion, particularly for people whose reading skills don’t match the writing skills of the researcher. Besides, participating in a research questionnaire ought to feel like entertainment, not like a 30-minute life skills exam. Banish that language to a life locked in carbonite and instead, choose a casual language style that people will feel comfortable with. (Oh, see what I did with that preposition!) You need to avoid slang, idioms, and inside jokes that are meaningless without context, but you can certainly inject a bit of casual but relevant humour along the way.

  • Are you ready to chat about carpet cleaners and vacuums? It might be a boring topic but we all need a clean home!

Questionnaires aren’t about comprehensive questions: Sometimes, in our attempts to be clear and focused, we end up writing questions that are long and complicated, subsequently making it difficult for people to deconstruct and comprehend the intention behind asking the question in the first place and causing the resulting data to be riddled with quality issues. The alternative is to break sentences apart. Short sentences make comprehension accessible to everyone. People who are reading in a second language can understand short sentences. People who have different reading skills can understand short sentences. Be part of the resistance when it comes to long questions and long answers. If our goal is comprehension, short sentences are always preferred.

  • In the last month, how many large bottles of detergent did you buy? (A large bottle is 1 litre or 1 kilogram or more. Please include liquid and powder detergent.)

Questionnaires aren’t about category comprehensiveness: When you start thinking all the questions that could be answered, it’s easy to stretch a 5-minute questionnaire into a 35-minute questionnaire. Use the force to avoid this inclination. Short questionnaires retain the interest and attention of participants and therefore generate much better data. Cut every question you know you won’t act on. Cut every question that won’t generate an actionable outcome. Cut all the ‘nice to know’ and ‘I wonder whether’ questions. If the questionnaire still requires more than 15 minutes to complete, then you need to move to step two – figure out whether it can be cut it into pieces. That could mean giving twice as many people half as many questions, or spreading the questionnaire out over multiple occasions.

Quality questionnaire writing is a rare skill: Whether it’s designing marketing strategies that double the business in one year, accurately translating mission statements into six languages, or writing effective questionnaires, everyone is a Jedi at something. Jedi Knights in the research industry have written entire textbooks on how to create a good questionnaire. They’ve witnessed thousands of fatal errors across many different categories and industries, and know many of the common and obscure mistakes. Even better, Jedi Masters have learned a plethora of techniques to counteract hundreds of cognitive biases that prevent people from answering truthfully. They’ve acquired a unique skill of ensuring questionnaires will meet specific needs and generate the best possible data quality. If your research outcomes are intended to feed into major decisions impacting the health of your business, it is essential that you seek out the advice of Jedi Master questionnaire writers.

And with these tips firmly entrenched, may the survey force be with you!

 

Annie Pettit, PhD, FMRIA, is a consultant for Sklar Wilton & Associates. She helps marketers build research tools that facilitate clear and direct answers to key questions and problems.

Sklar Wilton & Associates has worked for more than 30 years with some of Canada’s most iconic brands to help them solve tough business challenges to unlock growth and build stronger brands. SW&A was recognized as a Great Workplace for Women in 2018, and the Best Workplace in Canada for Small Companies in 2017 by the Great Place To Work® Institute. Recognized as the number one Employee Recommended Workplace among small private employers by the Globe and Mail and Morneau Shepell in 2017, SW&A achieved ERW certification again in 2018.

Like our posts? Sign up for our newsletter and enjoy insights from our associates in your mailbox every 4 to 6 weeks.

Advertisements

Is MyDemocracy.ca a Valid Survey?

Like many other Canadians, I received a card in the mail from the Government of Canada promoting a website named MyDemocracy.ca. Just a day before, I’d also come across a link for it on Twitter so with two hints at hand, I decided to read the documentation and find out what it was all about. Along the way, I noticed a lot of controversy about the survey so I thought I’d share a few of my own comments here. I have no vested interested in either party. I am simply a fan of surveys and have some experience in that regard.

First, let’s recognize that one of the main reasons researchers conduct surveys is to generate results which can be generalized to a specific population, for example the population of Canada. Having heard of numerous important elections around the world recently, we’ve become attuned to polling research which attempts to predict election and electoral winners. The polling industry has taken a lot of heat regarding perceived levels of low accuracy lately and people are paying close attention.

Sometimes, however, the purpose of a survey is not to generalize to a population, but rather to gather information so as to be more informed about a population. Thus, you may not intend to learn whether 10% of people believe A and 30% believe B, but rather that there is a significant proportion of people who believe A or B or C or D. These types of surveys don’t necessarily focus on probability or random sampling, but rather on gathering a broad spectrum of opinions and understanding how they relate to each other.  In other cases, the purpose of a survey to generate discussion and engagement, to allow people to better understand themselves and other people, and to think about important issues using a fair and balanced baseline that everyone can relate to.

The FAQ associated with MyDemocracy.ca explains the purpose of the survey in just this manner – to foster engagement. It explains that the experimental portion of the survey used a census balanced sample of Canadians, and that the current intention of the survey is  to help Canadians understand where they sit in relation to their fellow citizens. I didn’t see any intention for the online results to be used in a predictive way.

I saw some complaints that the questions are biased or unfair. Having completed the survey two and a half times myself, I do see that the questions are pointed and controversial. Some of the choices are extremely difficult to make. To me, however, the questions seem no different than what a constituent might be actually be asked to consider and there are no easy answers in politics. Every decision comes with side-effects, some bad, some horrid. So while I didn’t like the content of some of the questions and I didn’t like the bad outcomes associated with them, I could understand the complexity and the reasoning behind them. In fact, I even noticed a number of question design practices that could be used in analysis for data quality purposes. In my personal opinion, the questions are reasonable.

I’m positive you noticed that I answered the survey more than twice. Most surveys do not allow this but if the survey was launched purely for engagement and discussion rather than prediction purposes, then response duplication is not an issue. From what I see, the survey (assuming it was developed with psychometric precision as the FAQ and methodology describe) is a tool similar to any psychological tool whether personality test, intelligence test, reading test, or otherwise. You can respond to the questions as often as you wish and see whether your opinions or skills change over time. Given what is stated in the FAQ, duplication has little bearing on the intent of the survey.

One researcher’s opinion.

 

Since you’re here, let me plug my new book on questionnaire design! It makes a great gift for toddlers and grandmas who want to work with better survey data!
People Aren’t Robots: A practical guide to the psychology and technique of questionnaire design
http://itunes.apple.com/us/book/isbn9781370693108
https://www.amazon.ca/dp/1539730646/
https://www.smashwords.com/books/view/676159

People Aren’t Robots – New questionnaire design book by Annie Pettit

I’ve been busy writing again!

People Aren’t Robots: A practical guide to the psychology and technique of questionnaire is the best 2 bucks you’ll ever spend!

Questionnaire design is easy until you find yourself troubled with horrid data quality. The problem, as with most things, is that there is an art and science to designing a good quality and effective questionnaire and a bit of guidance is necessary. This book will give you that guidance in a short, easy to read, and easy to follow format. But how is it different from all the other questionnaire design books out there?

  • It gives practical advice from someone who has witnessed more than fifteen years of good and poor choices that experienced and inexperienced questionnaire writers make. Yes, even academic, professional researchers make plenty of poor questionnaire design choices.
  • It outlines how to design questions while keeping in mind that people are fallible, subjective, and emotional human beings. Not robots. It’s about time someone did this, don’t you think?

This book was written for marketers, brand managers, and advertising executives who may have less experience in the research industry.

It was also written to help academic and social researchers write questionnaires that are better suited for the general population, particularly when using research panels and customer lists.

I hope that once you understand and apply these techniques, you think this is the best $2 you’ve ever spent and that you hear your respondents say “this is the best questionnaire I’ve ever answered!”

Early reviews are coming in!

  • For the researchers and entrepreneurs out there, here’s a book from an expert. Pick it up (& read & implement). 👌
  • Congrats, Annie! A engagingly written and succinct book, with lots of great tips!
  • Congratulations! It’s a joy watching and learning from your many industry efforts.
  • It looks great!!! If I could, I would buy many copies and give to many people I know who need some of your advice.🙂

One size no longer fits all – beyond traditional surveys #ESOMAR #MRX 

Live blogged at Esomar in Dublin. Any errors or bad jokes are my own.

Modular surveys for agile research by Grant Miller and John Crockett

  • Survey with 150 questions and has been done for 30+ years – social values survey
  • Can we modularize this survey? How will that impact the results?
  • Used RIWI as the data provider, a mostly random URL bar sampler
  • Used chunking, broke survey into multiple sections – reshaped the survey based on sensical modules – 35 modules
  • fielded invidual modules, sometimes 1 or 2 or 3 or 4 randomly selected modules, let people answer as many as they wanted
  • problems with chunking – missing data in different areas, data from ten people might work out to five completes
  • people answered far more modules than expected, but a lot of data was coming from few respondents – 70% of data came from 30% of responders, didn’t see major demographic differences
  • opportunity to ask more questions because it didn’t seem to create bias demgraphically
  • There were differences versus panel data, skewed to younger population
  • [brand name drop :/]
  • RDIT has a positivity bias, more so than online panelists, this source uses scales differently, they were less likely to use the most negative response
  • we have to stop being uncomfortable working with partial data [in other words, stop forcing a response to every survey question!]
  • be open to blended data, partial data, be open to think differently
  • lesson learned – even though people like really short and researchers like really long, you can be inbetween 

When should we ask, we when should we measure by Melanie Revilla and German Loewe

  • How many times did you connect to your email last week?  Do you have access to this information? Can surveys collect this data?
  • Surveys have been used for subjective and objective data over the years, will we do this in the future?
  • What is the determinant of quality in survey data? memory affects it, but our memory is completely overwhelmed
  • we have so many distractions now, events are much quicker, so many products to think about, and why do we bother even trying to remember anything anymore since our phone remembers for us
  • used metering devices associated with a panel, compared stated versus actual passive device usage, is one more accurate and when?
  • asked people about the last five websites they visited, what was the match rate – 1% recalled 5 out of 5,  6% remembered 4, 9% remembered 3, 29% remembered none
  • ask people about ‘most often’ websites, spontaneous recall, 7 days or 2 months, people were far better with 2 months recall
  • with prompted recall,  trend wasn’t as expected but they don’t know why yet
  • there is always more over-reporting than under-reporting, acquiesence bias
  • people don’t remember their online activities
  • recall is even worse on a smartphone, so much marketing taking place [hello competely distracted! phone games, text messages, video watching, snapchatting]
  • think about when six blind men touch a different part of an elephant and they describe it differently, but together, they describe an elephant

The impact of questionnaire design on measurements in surveys #4 #ESRA15 #MRX 

Live blogged from #ESRA15 in Reykjavik. Any errors or bad jokes are my own.

Well, last night i managed to stay up until midnight. The lights at the church went on, lighting up the tower and the very top in an unusual way. They were quite pretty! The rest of the town enjoyed mood lighting as it didn’t really get dark at all. Tourists were still wandering in the streets since there’s no point going to bed in a delightful foreign city if you can still see where you’re going. And if you weren’t a fan of the mood lighting, have no fear! The sun ‘rose’ again just four hours later. If you’re scared of the dark, this is a great place to be – in summer!

Today’s program for me includes yet another sessions of question data quality, polling question design, and my second presentation on how non-native English speakers respond to English surveys. We may like to think that everyone answering our surveys is perfectly fluent but let’s be realistic. About 10% of Americans have difficulty reading/writing in English because it is not their native language. Add to that weakly and non-literate people, and there’s potential big trouble at hand.


the impact of answer format and item order on the quality of measurement

  • compared 2 point scale and 11 point scale, different order of questions and question can even be very widely apart, looked at perceived prestige of occupations
  • separated two pages of the surveys with a music game of guessing the artist and song, purely as distraction from the survey. the second page was the same questions in a completely different order, did the same thing numerous times changing the number of response options and question orders each time. whole experiment lasted one hour
  • assumed scale was uni-dimensional
  • no differences comparing 4 point to 9 point scale, none between 2 point and 9 point scale [so STOP USING HUGE SCALES!!!]
  •  prestige does not change depending on order in the survey [but this is to be expected with non-emotional, non-socially desirable items]
  • respondents confessed they tried to answer well but maybe not the best of their ability or maybe their answers would change the next time [glad to see people know their answers aren’t perfect. and i wouldn’t expect anything different. why SHOULD they put 100% effort into a silly task with no legitimate outcome for them.]

measuring attitudes towards immigration with direct questions – can we compare 4 answer categories with dichotomous responses

  • when sensitive questions are asked, social desirability affects response distributions
  • different groups are affected in different ways
  • asked questions about racial immigration – asked binary or as a 4 point scale
  • it’s not always clear that slightly is closer to none or that moderately is closer to strongly. can’t just assume the bottom two boxes are the same or the top two boxes are the same
  • education does have an effect, as well as age in some cases
  • expression of opposition for immigration depends on the response scale
  • binary responses leads to 30 to 50% more “allow none” responses than the 4 point scale
  • respondents with lower education have lower probability to choose middle scale point

cross cultural differences in the impact of number of response categories on response behaviour and data structure of a short scale for locus of control

  • locus of control scale, 4 items, 2 internal, 2 external
  • tested 5 point vs 9 point scale
  • do the means differ, does the factor structure differ
  • I’m  own boss; if i work hard, i’ll succeed; when at work or in m private life what I do is mainly determined by others; bad luck often gets in the way of m plans
  • labeled doesn’t apply at all, applies completely
  • didn’t see important demographic differences
  • saw one interaction but it didn’t really make sense [especially given sample size of 250 and lots of other tests happening]
  • [lots of chatter about significance and non-significance but little discussion of what that meant in real words]
  • there was no effect of item order, # of answer options mattered for external locus but not internal locus of control
  • [i’d say hard to draw any conclusions given the tiny number of items, small sample size. desperately needs a lot of replication]

the optimal number of categories in item specific scales

  • type of rating scale where the answer is specific to the scale and doesn’t necessarily apply to every other item – what is your health? excellent, good, poor
  • quality increased with the number of answer options comparing 11,7,5,3 point scales but not comparing 10,6,4 point scales
  • [not sure what quality means in this case, other audience members didn’t know either, lacking clear explanation of operationalization]

The impact of questionnaire design on measurements in surveys #3 #ESRA15 #MRX 

Live blogged from #ESRA15 in Reykjavik. Any errors or bad jokes are my own.

We had 90 minutes for lunch today which is far too long. Poor me.  I had pear skyr today to contrast yesterday’s coconut skyr. I can’t decide which one i like better. Oh, the hard decisions I have to make!  I went for a walk which was great since it drizzled all day yesterday. The downtown is tiny compared to my home so it’s quite fun to walk from one end to the other, including dawdling and eating, in less than half an hour. It’s so tiny that you don’t need a map. Just start walking and take any street that catches your fancy. I dare you to get lost. Or feel like you’re in an unsafe neighbourhood. It’s not possible.

I am in complete awe at the bird life here. There are a number of species i’ve never seen before which on its own is fun. It is also baby season so most of the ducks are paired off and escorting 2 to 8 tiny babies. They are utterly adorable as the babies float so well that they can barely swim underwater to eat. I haven’t seen any puffins along the shore line. I’m still hopeful that a random one will accidentally wander across my path.

By the way, exceptional beards really are a thing here. In case you were curious.

  the Who: experimental evidence on the effect of respondent selection on collecting individual asset ownership information

  • how do you choose who to interview?
  • “Most knowledgeble person”, random selection, the couple together, each individual adult by themself about themself, by themself about other people
  • research done in uganda so certainly not generalizable to north america
  • ask about dwelling, land, livestock, banking, bequeathing, selling, renting, collateral, investments
  • used CAPI, interviews matched on gender, average interview was 30 minutes
  • challenges included hard to find couples together as one person might be working in the field, hard to explain what assets were
  • asking couple together shows differences from ownership incidence but the rest is the same
  • [sorry, couldn’t determine what “signficant positive results” actually meant. would like to know. 😦 ]

Portuguese national health examination survey: questionnaire development

  • study includes physical measurements and a survey of health status, health behaviours, medication, income, expenses
  • pre-tested the survey for comprehension and complexity
  • found they were asking for things from decades ago and people couldn’t remember (eg when did you last smoke)
  • some mutually exclusive questions actually were not
  • you can’t just ask about ‘activity’ you have to ask about ‘physical activity that makes you sweat’
  • responses cards helped so that people didn’t have to say an embarrassing word
  • had to add instructions that “some questions may not apply to you but answer anyways” because people felt that if you saw them walking you shouldn’t ask whether they can walk
  • gave examples of what sitting on the job, or light activity on the job meant so that desk sitters don’t include walking to the bathroom as activity
  • pretest revealed a number of errors that could be corrected, language and recall problems can be overcome with better questions

an integrated household survey for Wales

  • “no change” is not a realistic option [i wish more people felt that way]
  • duplication among the various surveys, inefficient, survey costs are high
  • opportunity to build more flexibility into a new survey
  • annual sample size of 12000, randomly selected 16+ adults, 45 minutes
  • want to examine effects of offering incentives
  • survey is still in field
  • 40% lower cost compared to previous, significant gains in flexibility

undesired repsonse to sureys, wrong answers or poorly worded question? how respondents insist on reporting their situation despite unclear questionning

  • compared census information with family survey information
  • interested in open text answers
  • census has been completed since 1881
  • belle-mere can mean stepmother and mother in law in french
  • can’t tell if grandchildren in the house belong to which adult child in the house
  • ami can mean friend or boyfriend or partner or spouse, some people will also specify childhood friend or unemployed friend or family friend
  • can’t tell if an unknown location of child means they don’t know the address or the child has died
  • do people with an often changing address live in a camper, or travel for work?
  • if you only provide age in years for babies you won’t know if it’s stillborn or actually 1 year old

ask a positive question and get a positive answer: evidence on acquiesence bias from health care centers in nigeria

  • created two pairs of questions were one was positive and one was negative – avoided the word no [but the extremeness of the questions differed, eg., “Price was reasonable” vs “Price was too expensive” ]
  • some got all positive, all negative, or a random mix
  • pilot test was a disaster, in rural nigeria people weren’t familiar with this type of question
  • instead, started out asking a question about football so people could understand how the question worked. asked agree or disagree, then asked moderately or strongly – two stage likert scale
  • lab fees were reasonable generated very different result than lab fees were unreasonable [so what is reality?]
  • it didn’t matter if negatives were mixed in with positives
  • acquiescence bias affects both positive and negative questions, can’t say if it’s truly satisficing, real answer is probably somewhere in between [makes we wonder, can we develop an equation to tease out truth]
  •  large ceiling effects on default positive framing — clinics are satisfactory despite serious deficiencies
  • can’t increase scores with any intervention but you can easily decrease the scores
  • maybe patient satisfaction is the wrong measure
  • recommend using negative framing to avoid ceiling effects [I wonder if in north america, we’re so good at complaining that this isn’t relevant]

The impact of questionnaire design on measurements in surveys #2 #ESRA15 #MRX  

Live blogged at #ESRA15 in Reykjavik. Any errors or bad jokes are my own.

Breaktime treated us to fruit and croissants this morning. I was hoping for another unique to iceland treat but perhaps that was a sign to stop eating. No, just kidding! Apparently you’re not allowed to bring food or drink into the classrooms. The signs say so. The signs also say no Facebook in the classrooms. Shhhh…. I was on Facebook in the classroom!

The sun is out again and I took a quick walk outside. I am thankful my hotel is at the foot of the famous church. No matter where I am in this city, I can always, easily, and instantly find my hotel. No map needed when the church is several times higher than the next highest building!

I’ve noticed that the questions at this conference are far more nit-picky and critical than I’m used. I suspect that is because the audience includes many academics whose entire job is focused on these topics. They know every minute detail because they’ve done similar studies themselves. It makes for great comments and questions, though it does seem to put the speaker on the spot every time!

smart respondents: let’s keep it short.

  • do we really need scale instructions in the question stem? they add length, mobile screens have limited space, and respondents skip the instructions if the response scale is already labeled [isn’t this just an artifact of old fashioned face to face surveys, telephone surveys]
  • they tested instructions that matched and did not match what was actually in the scale [i can imagine some panelists emailing the company to complain that the survey had errors!]
  • used a probability survey [this is one case where a nonprobability sample would have been well served, easier cheaper to obtain with no need to generalize precisely to a population]
  • answer frequencies looked very similar for correct and incorrect instructions, no significant differences, she’s happy to have nonsignificant results, unaffected by mobile device or age
  • [more regression results shown, once again, speaker did not apologize and the audience did not have a heart attack]
  • it seems like responsents ignore instructions in the question, they reply on the words in the answer options, e.g., grid headers
  • you can omit instructions if the labeling is provided in the answer options
  • works better for experienced survey takers [hm, i doubt that. anyone seeing the answer options will understand. at least, thats my opinion.]

from web to paper: evaluation from data providers and data analysts. The case of annual survey finances of enterprises

  • we send out questionaires, something happens, we get data back – we don’t know what happens 🙂
  • wanted to keep question codes in the survey which seemed unnecessary to respondents, had really long instructions for some questions that didn’t fit on the page so they put them on a pdf
  • 64% of people evaluted the codes on the online questionnaire positively, 12% rated the codes negatively. people liked that they could communicate with statistics netherlands by using the codes
  • 74% negative responses to explanations of question which were intended to reduce calls from statistics netherlands, only 11% were positive
  • only 25% of people consulted the pdf with instructions
  • most people wanted to received a printed version of the questionnaire they filled out, people really wanted to print it and they screen capped it, people liked being able to return later, they could easily get an english version
  • data editors liked that they didn’t have to do data entry but now they needed more time to read and understand what was being said
  • they liked having the email address because they got more direct and precise answers, responses came back faster, they didn’t notice any changes in the time series data

is variation in perception of inequality and redistribution of earnings actual or artifactual. effects of wording, order, and number of items

  • opinions differ when you ask how much should people make vs how much should the top quintile of peopl emake
  • they asked people how much a number of occupations should earn, they also varied how specific the title was e.g., teacher vs math teacher in a public highschool
  • estimates for specific descriptions were higher, high status jobs got much higher estimates
  • adding more occupations to the list makes reliability in earnings decrease

exploring a new way to avoid errors in attitude measurements due to complexity of scientific terms: an example with the term biodiversity

  • how do people talk about complicated terms, their own words often differ from scientific definitions
  • “what comes to mind when you think of biodiversity?” – used text analysis for word frequencies, co-occurences, correspondence analysis, used the results to design items for the second study
  • found five classes of items – standard common definition, associated with human actions to protect it, human envionment relationship, global actions and consequences, scientific definition
  • turned each of the five types of defiintions into a common word definition
  • people gave more positive opinions about biodiversity when they were asked immediately after the definition
  • items based on representations of biodiversity were valid and reliable
  • [quite like this methodology, could be really useful in politics]

[if any of these papers interest you, i recomend finding the author on the ESRA program and asking for an official summary. Global speakers and weak microphones makes note taking more challenging. 🙂 ]



The impact of questionnaire design on measurements in surveys #1 #ESRA15  #MRX  

Live blogged from #ESRA15 in Reykjavik. Any errors or bad jokes are my own.

I tried to stay up until midnight last night but ended going to bed around 10:30pm. Naturally, it was still daylight outside. I woke up this morning at 6am in broad daylight again. I’m pretty sure it never gets dark here no matter what they say. I began my morning routine as usual. Banged my head on the slanted ceiling, stared out the window at the amazing church, made myself waffles in the kitchen, and then walked past the pond teaming with baby ducks. Does it get any better? I think no. Except of course knowing i had another day of great content rich sessions ahead of me!


designs and developments of the income measures in the european social surveys

  • tested different income questions. allowed people to use a weekly, monthly, or annual income scale as they wished. there was also no example response, and no example of what constitutes income. Provided about 30 answer options to choose from, shown in three columns. Provided same result as a very specific question in some countries but not others.
  • also tested every country getting the same number breaks, groups weren’t arranged to reflect each countries distribution. this resulted in some empty breaks [but that’s not necessarily a problem if the other breaks are all well and evenly used]
  • when countries are asked to set up number breaks in well defined deciles, high incomes are chosen more often – affected because people had different ideas of what is and isn’t taxable income
  • [apologies for incomplete notes, i couldn’t quite catch all the details, we did get a “buy the book” comment.]

item non-response and readability of survey questionnaire

  • any non-substantive outcome – missing values, refusals, don’t knows all count
  • non response can lower validity of survey results
  • semantic complexity measured by familiarity of words, length of words, abstract words that can’t be visualized, structural complexity
  • Measured – characters in an item, length of words, percent of abstract words, percent of lesser known words, percent of long words 12 or more characters
  • used the european social survey which is a highly standardized international survey, compared english and estonian, it is conducted face to face, 350 questions, 2422 uk respondents
  • less known and abstract words create more non-response
  • long words increase nonresponse in estonian but not in english, perhaps because english words are shorter anyways
  • percent of long words in english created more nonresponse
  • total length of an item didn’t affect nonresponse
  • [they used a list of uncommon words for measurement, such a book/list does exist in english. I used it in school to choose a list of swear words that had the same frequency levels as regular words.]
  • [audience comment – some languages join many words together which means their words are longer but then there are fewer words, makes comparisons more difficult]

helping respondents provide good answers in web surveys

  • some tasks are inherently difficult in surveys, often because people have to write in an answer, coding is expensive and error prone
  • this study focused on prescription drugs which are difficult to spell, many variations of the same thing, level of detail is unclear, but we have full lists of all these drugs available to us
  • tested text box, drop box to select from list, javascript (type ahead look up)
  • examined breakoff rates, missing data, response times, and codability of responses
  • asked people if they are taking drugs, tell us about three
  • study 1 – breakoffs higher from dropbox and javascript; median response times longer, but codability was better. LIsts didn’t work well at all.
  • study 2 – cleaned up the list, made all the capitalization the same. break off rates were now all the same. response times lower but still higher than the textbox version. codability still better for list versions.
  • study 3 – if they couldn’t find a drug in the list, they were allowed to type it out. unlike previous studies which proceeded with the missing data. dropbox had highest missing data. javascript had lowest missing data. median times highest for drop box. trends for more and more drugs as expected, effect is more but not as much more.
  • older browswers had trouble with dropdowns and javascript and had to be routed to the textbox options
  • if goal is to get codable answers, use a text box. if goal is to create skip patterns then javascript is the way to go.

rating scale labelling in web surveys – are numeric labels an advantage

  • you can use all words to label scales or just words on the end with numbers in between
  • research says there is less satisficing with verbal scales, they are more natural than numbers and there is no inherent meaning of numbers
  • means of the scales were different
  • less tie to completes the end labeled groups
  • people paid more attention to the five point labeled scale, and least to the end point labeled score
  • mean opinions did differ by scale, more positive on fully labeled scale
  • high cognitive burden to map responses of the numeric scales
  • lower reliability for the numeric labels

Opening plenary session: Trends, Mobile Surveys, and Hacking Data #IIeX

Live blogged at #IIeX Atlanta. Any errors or bad jokes are my own. 


Using future trends to change business outcomes by J. Walker Smith

  • we should focus more on the vanishing point, turning points that look like cliffs not hockey sticks
  • we always look for the next best thing, weak signals getting stronger, before things become mainstream
  • its a coding problem, how do you know what to look for if you’ve never seen it, we don’t know how to look for it, is it just luck

  • experts are worse than chance at predicting the future
  • start with things you know how to code for, the obvious big things that you can’t ignore, what are the strong signals that are weakening because these are where innovation opportunities are opening up
  • narrow field of view to zero in on one spot where we know change is going to occur, these are unrecognized things that are already eroding.
  • big things have to make room for smaller things to spread
  • single households/solo living is the next big thing approach, we need products and services to allow people to live solo livestyles. but this is wrong. single households are just one small piece of a major dynamic. Bigger dynamic is actually shift from marrying early to marrying later. Vanishing point is marrying early. Many reasons for this. Result is not no marriage or solo living. People still get married but it’s just later. Longer transition to a married lifestyle.
  • In USA, 7 year increase, China 1 year, Russia, 2 years, India 3 years, [sorry, can’t read the others but some countries look like marrying up to 10 years later!]
  • People are not learning how to live alone but how to connect. How do you replace the lost family connections around things like eating a meal. Zipbob is a website where single people make restaurant reservations to share a meal with strangers. Mokbang is a website where you livestream with foodporn stars eating meals.  Use technology to live together in a different way.
  • The Kinship economy – data is always a secondary consideration of technology. Its always been about connecting people.
  • We spend all our time trying to get people to connect with our brand. We brought this to social media in the last ten years. But people don’t want a relationship with your brand but rather with people.
  • we need to be in the business of social currency, give people a way to connect with other people.
  • the bigger opportunity space is togetherness not single-ness.
  • [interesting talk, never really thought about it like that]

Who needs to ask a question? Using social media data to predict behavior by Vesselin Popov

  • digital footprints are a new way of doing psychometrics [i LOVE psychometrics. that was my career goal that led me to MR]
  • started with a facebook app mypersonality, among many other test games. They let people take them for free. Asked people to opt in to share the data with them. 6 million people shared their data. They share this data with 18 universities. They also have people’s status updates and facebook demos likes etc.
  • Is there a relatinship between your likes and your personality profile. Massively.
  • Compared friend ratings to computer model. Computer is better than a human judge once you have 300 likes. Used the Big 5 Index.
  • Can predict intelligence, life satisfaction, and more
  • Can compare profiles of people who like tom cruise vs frank sinatra – sinatra is more liberal and artistic. Artistic people like oscar wilde, banhause, plato, jon waters, Leonard Cohen. Try targeting on these criteria
  • introverts use different words than extraverts who talk about parties, night, tonight, weekend, excited where introverts talk about computers, anime [seriously? anime suggests a very skewed sample of introverts participated. Or we’re looking at differences due to decimal places not reality]
  • you can reach the hard to reach through their digital footprint [if you have permission to do so, just because you have a facebook profile doesn’t mean you’ve opted in to research]

It’s not me, It’s you: Research participants and data speak on mobile design and data quality by Melanie Courtright

  • we take participants for granted, we make promises to them and disappoint them

  •  we promise that they can answer the surveys, that we’ll be device agnostic but we’re now. More than half of surveys fail when taken on a mobile device. Enrollment via mobile device as increased by 300%
  • Most often a grid is turned into a set of vertical scale single questions
  • PC style surveys on a smartphone take ridiculous amount of time. people speed through once they’ve had enough. Speeding or quiting – what’s your preference.
  • Enjoyability scores are massively lower. Ease scores are a lot lower as well. When it’s fun and easy, people use the variability that is present in the scale.
  • people stop using the end of scale because it’s too difficult to do all the scrolling
  • when all the data is living together, you only need to ask questions about what isn’t already in that data. we must ask less and track more. don’t ask everything you can think of. stop ‘just adding one more question’
  • right isn’t always easy.

Hacking insights to drive innovation and ROI by Rolfe Swinton

  • cracker is someone who hacks with malicious intent
  • playful solving of technical work that requires deep understanding especially of a computer system
  • do you want your parole board meeting at 9am or 4pm? It’s just data right? the best time is first thing in the morning, immediately after lunch, or immediately after break. Never just before break time or home time.
  • sensor technology is becoming ubiquitous and nearly free. first gps cost $120000, now it’s $50
  • are companies changing at the same rate? digital video growth is 3 times in the last 3 years, across all age groups
  • hacking reason – tackling big problems requires a lot of components coming together
  • hacking reason – needs to be an act of play, need to take risks and have fun
  • when should you reach car buyers? peeople think about hair cuts near the end of the week,  cars they think about at the beginning of the week and go at the end of the wee
%d bloggers like this: