Live blogged from #ESRA15 in Reykjavik. Any errors or bad jokes are my own.

Well, last night i managed to stay up until midnight. The lights at the church went on, lighting up the tower and the very top in an unusual way. They were quite pretty! The rest of the town enjoyed mood lighting as it didn’t really get dark at all. Tourists were still wandering in the streets since there’s no point going to bed in a delightful foreign city if you can still see where you’re going. And if you weren’t a fan of the mood lighting, have no fear! The sun ‘rose’ again just four hours later. If you’re scared of the dark, this is a great place to be – in summer!

Today’s program for me includes yet another sessions of question data quality, polling question design, and my second presentation on how non-native English speakers respond to English surveys. We may like to think that everyone answering our surveys is perfectly fluent but let’s be realistic. About 10% of Americans have difficulty reading/writing in English because it is not their native language. Add to that weakly and non-literate people, and there’s potential big trouble at hand.

the impact of answer format and item order on the quality of measurement

• compared 2 point scale and 11 point scale, different order of questions and question can even be very widely apart, looked at perceived prestige of occupations
• separated two pages of the surveys with a music game of guessing the artist and song, purely as distraction from the survey. the second page was the same questions in a completely different order, did the same thing numerous times changing the number of response options and question orders each time. whole experiment lasted one hour
• assumed scale was uni-dimensional
• no differences comparing 4 point to 9 point scale, none between 2 point and 9 point scale [so STOP USING HUGE SCALES!!!]
•  prestige does not change depending on order in the survey [but this is to be expected with non-emotional, non-socially desirable items]
• respondents confessed they tried to answer well but maybe not the best of their ability or maybe their answers would change the next time [glad to see people know their answers aren’t perfect. and i wouldn’t expect anything different. why SHOULD they put 100% effort into a silly task with no legitimate outcome for them.]

measuring attitudes towards immigration with direct questions – can we compare 4 answer categories with dichotomous responses

• when sensitive questions are asked, social desirability affects response distributions
• different groups are affected in different ways
• asked questions about racial immigration – asked binary or as a 4 point scale
• it’s not always clear that slightly is closer to none or that moderately is closer to strongly. can’t just assume the bottom two boxes are the same or the top two boxes are the same
• education does have an effect, as well as age in some cases
• expression of opposition for immigration depends on the response scale
• binary responses leads to 30 to 50% more “allow none” responses than the 4 point scale
• respondents with lower education have lower probability to choose middle scale point

cross cultural differences in the impact of number of response categories on response behaviour and data structure of a short scale for locus of control

• locus of control scale, 4 items, 2 internal, 2 external
• tested 5 point vs 9 point scale
• do the means differ, does the factor structure differ
• I’m  own boss; if i work hard, i’ll succeed; when at work or in m private life what I do is mainly determined by others; bad luck often gets in the way of m plans
• labeled doesn’t apply at all, applies completely
• didn’t see important demographic differences
• saw one interaction but it didn’t really make sense [especially given sample size of 250 and lots of other tests happening]
• [lots of chatter about significance and non-significance but little discussion of what that meant in real words]
• there was no effect of item order, # of answer options mattered for external locus but not internal locus of control
• [i’d say hard to draw any conclusions given the tiny number of items, small sample size. desperately needs a lot of replication]

the optimal number of categories in item specific scales

• type of rating scale where the answer is specific to the scale and doesn’t necessarily apply to every other item – what is your health? excellent, good, poor
• quality increased with the number of answer options comparing 11,7,5,3 point scales but not comparing 10,6,4 point scales
• [not sure what quality means in this case, other audience members didn’t know either, lacking clear explanation of operationalization]

Live blogged from #ESRA15 in Reykjavik. Any errors or bad jokes are my own.

I tried to stay up until midnight last night but ended going to bed around 10:30pm. Naturally, it was still daylight outside. I woke up this morning at 6am in broad daylight again. I’m pretty sure it never gets dark here no matter what they say. I began my morning routine as usual. Banged my head on the slanted ceiling, stared out the window at the amazing church, made myself waffles in the kitchen, and then walked past the pond teaming with baby ducks. Does it get any better? I think no. Except of course knowing i had another day of great content rich sessions ahead of me!

designs and developments of the income measures in the european social surveys

• tested different income questions. allowed people to use a weekly, monthly, or annual income scale as they wished. there was also no example response, and no example of what constitutes income. Provided about 30 answer options to choose from, shown in three columns. Provided same result as a very specific question in some countries but not others.
• also tested every country getting the same number breaks, groups weren’t arranged to reflect each countries distribution. this resulted in some empty breaks [but that’s not necessarily a problem if the other breaks are all well and evenly used]
• when countries are asked to set up number breaks in well defined deciles, high incomes are chosen more often – affected because people had different ideas of what is and isn’t taxable income
• [apologies for incomplete notes, i couldn’t quite catch all the details, we did get a “buy the book” comment.]

item non-response and readability of survey questionnaire

• any non-substantive outcome – missing values, refusals, don’t knows all count
• non response can lower validity of survey results
• semantic complexity measured by familiarity of words, length of words, abstract words that can’t be visualized, structural complexity
• Measured – characters in an item, length of words, percent of abstract words, percent of lesser known words, percent of long words 12 or more characters
• used the european social survey which is a highly standardized international survey, compared english and estonian, it is conducted face to face, 350 questions, 2422 uk respondents
• less known and abstract words create more non-response
• long words increase nonresponse in estonian but not in english, perhaps because english words are shorter anyways
• percent of long words in english created more nonresponse
• total length of an item didn’t affect nonresponse
• [they used a list of uncommon words for measurement, such a book/list does exist in english. I used it in school to choose a list of swear words that had the same frequency levels as regular words.]
• [audience comment – some languages join many words together which means their words are longer but then there are fewer words, makes comparisons more difficult]

helping respondents provide good answers in web surveys

• some tasks are inherently difficult in surveys, often because people have to write in an answer, coding is expensive and error prone
  
• this study focused on prescription drugs which are difficult to spell, many variations of the same thing, level of detail is unclear, but we have full lists of all these drugs available to us
• tested text box, drop box to select from list, javascript (type ahead look up)
• examined breakoff rates, missing data, response times, and codability of responses
• asked people if they are taking drugs, tell us about three
• study 1 – breakoffs higher from dropbox and javascript; median response times longer, but codability was better. LIsts didn’t work well at all.
• study 2 – cleaned up the list, made all the capitalization the same. break off rates were now all the same. response times lower but still higher than the textbox version. codability still better for list versions.
• study 3 – if they couldn’t find a drug in the list, they were allowed to type it out. unlike previous studies which proceeded with the missing data. dropbox had highest missing data. javascript had lowest missing data. median times highest for drop box. trends for more and more drugs as expected, effect is more but not as much more.
• older browswers had trouble with dropdowns and javascript and had to be routed to the textbox options
• if goal is to get codable answers, use a text box. if goal is to create skip patterns then javascript is the way to go.

rating scale labelling in web surveys – are numeric labels an advantage

• you can use all words to label scales or just words on the end with numbers in between
  
• research says there is less satisficing with verbal scales, they are more natural than numbers and there is no inherent meaning of numbers
• means of the scales were different
• less tie to completes the end labeled groups
• people paid more attention to the five point labeled scale, and least to the end point labeled score
• mean opinions did differ by scale, more positive on fully labeled scale
• high cognitive burden to map responses of the numeric scales
• lower reliability for the numeric labels

Related articles

• Missing Data: Whose problem is it anyways?
• 5 ways to turn a really boring survey into a less boring innovative survey
• Is your data mullock?
• The Exciting Future of Market Research, ESOMAR APAC 2015 Event Review