Keynote: Design and implementation of comparative surveys by Lars Lyberg #ESRA15 #MRX 


Live blogged from the #ESRA15 conference in Reykjavik. Any error or bad jokes in these notes are my own. Thank you ESRA for the free wifi that made this possible. Thank you Reykjavik for coconut Skyr and miles upon miles of beautiful lupins.

  • Introduction
  • Data quality starts with high response rates and also needs trust from the public in order to provide honest answers
  • Iceland is the grandmother of democracy, a nation of social trust, icelanders still give high reponse rates
  • Intro: Guni Johannesson
  • Settlement until 1262, decline until 19th century, rise again in 20th centure – traditional story
  • founded the first democratic parliament, vikings were traders but also murderous terrorists, there was no decline, struggle for independance – this is the revised story; 2008 economic collapse due to many factors including misuse of history – people thought of themselves as vikings and took irresponsible risks

  • Lars Lyberg
  • WIsh we had more interesting presentations like the previous
  • 3M – multi-national, regional, cultural surveys, reveal differences between countries and cultures
  • [i always wonder, is cross cultural comparison TRULY possible]
  • Some global surveys of happiness included the presence of lakes or strong unions which automatically excludes a number of countries
  • problems with 3m studies  normally emphasize minimum response rates, specifications are not always adhered to, sometimes fabricated data, translations not done well, lack of total survey error awareness, countries are very different
  • special features of these studies – concepts must have a uniform meaning across countries, risk management differs, financial resources differ, national interests are in conflict, scientific challenges, adminisrative challenges, national pride is at stake especially when the media gets a hold of results
  • basic design issues – conditions cannot vary from definitions to methods to data collection, sampling can and should vary, weighting and stats are grey zones, quality assurance is necessary
  • Must QC early interviews of each interviewer, specs are sometimes not understood, sometimes challeneged, not affordable, not in line with best practice, overwhelming
  • Common challenges – hard to reach respondents, differences in literacy levels, considerable non response
  • interviewers should be alone with respondent for privacy reasons but it is common to not be alone – india, iraq, brazil there are often extra people around which affects the results, this is particularly important re mental health
  • a fixed response rate goal can be almost impossible to achieve, 70% is just unreasonable in many places. spending so much money to achieve that one goal is in conflict with TSE and all the other errors that could be attended to instead. in this example, only a few of the countries achieved it and only barely [and I wonder to what unethical means they went to achieve those]
  • strategies – share national exeriences, training, site visits, revised contact forms, explore auxiliary data, monitor fieldwork, assess non response bias
  • data fabrication [still cant believe professionals do this😦 ] 10 of 70 countries in a recent study have questionnable data, in 3 cases they clearly showed some data was fabricated PISA 2009, they often copy paste data [sigh, what a dumb method of cheating, just asking to be caught. so i’m glad they were dumb]
  • [WHY do people fabricate? didn’t get the desired response rate? embarrassed about results? too lazy to collect data?]
  • Translation issues – translation used to be close transalation with back translation, focus on replication “are you feeling blue” doesnt have the same meaning in another language, this still happens
  • Team Translation Model – TRAPD – draft translations, review and refine, adjudicate for pretest
  • Social desireaility differs in conformist and individual societies, relative status between interviewer and respondents, response process is different, perceptual variation is magnified even within a country, questionnaires must be different across countries
  • workloads differ – countries use different validation methods, countries dont know how to calculate weights, interviewer workload differed
  • specifications are often dubious, all kinds of variations are permitted, proxy responses can range fro 0% to 50% which is really bad for embarrassing questions where people don’t want others to know (e.g., a spouse could say the other spouse is happy)
  • Quality management approach – descrease distance between user and producer, find root causes of problems, allocate resources based on risk assessment, coordinate team responsibilities, strive for real time interventions, build capacity
  • Roger Jowell – 10 golden rules for cross national studies [find and reach this, it’s really good]
  • don’t confuse respect for cultural variations with tolderance of methodological anarchy, don’t aim for as many countries as possible, never do a survey in a country you know little about, pay as much attention to aggregate level background information as the individual level variables, assume any new variation you discover is an artifact, resist the temptation to crosstab everything [smart dude, i like these!]

 

Related Posts

 

%d bloggers like this: