Tag Archives: data quality

Voxpopme 8: Two key tips or tricks for communicating insights that resonate with the C-Suite and drive real results

By LoveStats on

Along with a group of market resevoxpopme logoarchers from around the world, I was asked to participate in Voxpopme Perspectives – an initiative wherein insights industry experts share ideas about a variety of topics via video. You can read more about it here or watch the videos here. Viewers can then reach out over Twitter or upload their own video response. I’m more of a writer so you’ll catch me blogging rather than vlogging. 🙂

Episode 8: Share two key tips or tricks you have for communicating insights that resonate with the C-Suite and drive real results.

Alrighty, tip number one: Sample Sizes.

The reasons for choosing sample sizes are a foreign concept to many people, leaders included. Many people depend on you to provide helpful guidance when it comes understanding what an appropriate sample size is, the drawbacks of those sizes, and how results can be interpreted given those choices. One tip I’ve used is to give them specific examples of what might and might not be statistically significant when the results do come through. For instance, rather than sharing the margin of error around a specific sample size, instead I’ll say something like:

With this sample size, a result of 30% would be statistically different from 37% but statistically the same as 36%. Are you prepared to choose a winning concept that is preferred by 30% of people rather than by 36% of people?

Tip number two: actionability.

As someone who loves raw data, cleaned data, charted data, graphed data, and tabled data, sometimes it’s hard to take the next step and make the data useable and actionable. But business leaders don’t always care about individual data points. They may not even be concerned with summaries of the results. What they really want is your informed opinion about what the data actually mean, and the appropriate options that should be considered as a result of the data. So, beyond reporting that 30% of people like a certain thing, use your understanding of the rest of the results to indicate why they like a certain thing, why they might not like it, the implications of moving forward (or not) with that thing, and how that choice might affect other products on the market already. Take the data as far forward as you possibly can in order to give them fodder to spark further ideas.

Bonus tip!

Know your own weaknesses. I know that data visualization is not my strength. When I need data to be visualized well so that it is understandable by everyone, from junior to senior and expert to newbie, my only option is to find an expert. And here’s an example of how an expert would illustrate missing data. I would have never thought to do it like but look at how effective it is. It’s worth the extra cost.

http://www.thirdway.org/infographic/the-absurd-way-we-report-higher-ed-data

<<<<
gt;<<
p>

Posted in: marketing research | Tagged: , , , , ,

Voxpopme 7: How will automation impact the industry, and you personally, over the next twelve months?

By LoveStats on

Along with a group of market resevoxpopme logoarchers from around the world, I was asked to participate in Voxpopme Perspectives – an initiative wherein insights industry experts share ideas about a variety of topics via video. You can read more about it here or watch the videos here. Viewers can then reach out over Twitter or upload their own video response. I’m more of a writer so you’ll catch me blogging rather than vlogging. 🙂

Episode 7: How will automation impact the industry, and you personally, over the next twelve months?

I’m not concerned with the next 12 months whatsoever. If we aren’t planning for the next five and ten years, we’re going to be in a lot of trouble. With that in mind, I’d like to consider how automation and artificial intelligence will impact me over that time frame.

The reality is that my job will change a lot. No longer will I receive a dataset, clean out poor quality data, run statistics, write a report, and prepare a presentation. Every aspect of that will be handled automatically and with artificial intelligence. I will receive a report at my desk that is perfectly written, with the perfect charts, and perfectly aligned to my clients’ needs.

So why will I still be there? I’ll be the person who points out the illogical outcomes of the data. How errors enter during the data collection process via human cognitive biases. I’ll be the person who interprets the data in an odd way that wasn’t predicted by the data but is still a plausible outcome. I’ll help clients read between the lines and use the results wisely rather than by the book – or rather, by the AI.

So how will automation and artificial intelligence impact our industry? If your business sells repetitive tasks, from survey programming to data cleaning to statistics to chart preparation and report writing, you’d better have a long term plan. Figure out your unique method of selling WISE applications. Not just data, but wiser data and wiser charts and wiser reports. There are already hundreds of companies innovating in these areas right now and they are waiting to find their customers. I expect you don’t want to hand over your customers to them.

Posted in: marketing research | Tagged: , , , , , , ,

Is MyDemocracy.ca a Valid Survey?

By LoveStats on

Like many other Canadians, I received a card in the mail from the Government of Canada promoting a website named MyDemocracy.ca. Just a day before, I’d also come across a link for it on Twitter so with two hints at hand, I decided to read the documentation and find out what it was all about. Along the way, I noticed a lot of controversy about the survey so I thought I’d share a few of my own comments here. I have no vested interested in either party. I am simply a fan of surveys and have some experience in that regard.

First, let’s recognize that one of the main reasons researchers conduct surveys is to generate results which can be generalized to a specific population, for example the population of Canada. Having heard of numerous important elections around the world recently, we’ve become attuned to polling research which attempts to predict election and electoral winners. The polling industry has taken a lot of heat regarding perceived levels of low accuracy lately and people are paying close attention.

Sometimes, however, the purpose of a survey is not to generalize to a population, but rather to gather information so as to be more informed about a population. Thus, you may not intend to learn whether 10% of people believe A and 30% believe B, but rather that there is a significant proportion of people who believe A or B or C or D. These types of surveys don’t necessarily focus on probability or random sampling, but rather on gathering a broad spectrum of opinions and understanding how they relate to each other.  In other cases, the purpose of a survey to generate discussion and engagement, to allow people to better understand themselves and other people, and to think about important issues using a fair and balanced baseline that everyone can relate to.

The FAQ associated with MyDemocracy.ca explains the purpose of the survey in just this manner – to foster engagement. It explains that the experimental portion of the survey used a census balanced sample of Canadians, and that the current intention of the survey is  to help Canadians understand where they sit in relation to their fellow citizens. I didn’t see any intention for the online results to be used in a predictive way.

I saw some complaints that the questions are biased or unfair. Having completed the survey two and a half times myself, I do see that the questions are pointed and controversial. Some of the choices are extremely difficult to make. To me, however, the questions seem no different than what a constituent might be actually be asked to consider and there are no easy answers in politics. Every decision comes with side-effects, some bad, some horrid. So while I didn’t like the content of some of the questions and I didn’t like the bad outcomes associated with them, I could understand the complexity and the reasoning behind them. In fact, I even noticed a number of question design practices that could be used in analysis for data quality purposes. In my personal opinion, the questions are reasonable.

I’m positive you noticed that I answered the survey more than twice. Most surveys do not allow this but if the survey was launched purely for engagement and discussion rather than prediction purposes, then response duplication is not an issue. From what I see, the survey (assuming it was developed with psychometric precision as the FAQ and methodology describe) is a tool similar to any psychological tool whether personality test, intelligence test, reading test, or otherwise. You can respond to the questions as often as you wish and see whether your opinions or skills change over time. Given what is stated in the FAQ, duplication has little bearing on the intent of the survey.

One researcher’s opinion.

 

Since you’re here, let me plug my new book on questionnaire design! It makes a great gift for toddlers and grandmas who want to work with better survey data!
People Aren’t Robots: A practical guide to the psychology and technique of questionnaire design
http://itunes.apple.com/us/book/isbn9781370693108
https://www.amazon.ca/dp/1539730646/
https://www.smashwords.com/books/view/676159

Posted in: marketing research | Tagged: , ,

People Aren’t Robots – New questionnaire design book by Annie Pettit

By LoveStats on

I’ve been busy writing again!

People Aren’t Robots: A practical guide to the psychology and technique of questionnaire is the best 2 bucks you’ll ever spend!

Questionnaire design is easy until you find yourself troubled with horrid data quality. The problem, as with most things, is that there is an art and science to designing a good quality and effective questionnaire and a bit of guidance is necessary. This book will give you that guidance in a short, easy to read, and easy to follow format. But how is it different from all the other questionnaire design books out there?

  • It gives practical advice from someone who has witnessed more than fifteen years of good and poor choices that experienced and inexperienced questionnaire writers make. Yes, even academic, professional researchers make plenty of poor questionnaire design choices.
  • It outlines how to design questions while keeping in mind that people are fallible, subjective, and emotional human beings. Not robots. It’s about time someone did this, don’t you think?

This book was written for marketers, brand managers, and advertising executives who may have less experience in the research industry.

It was also written to help academic and social researchers write questionnaires that are better suited for the general population, particularly when using research panels and customer lists.

I hope that once you understand and apply these techniques, you think this is the best $2 you’ve ever spent and that you hear your respondents say “this is the best questionnaire I’ve ever answered!”

Early reviews are coming in!

  • For the researchers and entrepreneurs out there, here’s a book from an expert. Pick it up (& read & implement). 👌
  • Congrats, Annie! A engagingly written and succinct book, with lots of great tips!
  • Congratulations! It’s a joy watching and learning from your many industry efforts.
  • It looks great!!! If I could, I would buy many copies and give to many people I know who need some of your advice.🙂
Posted in: marketing research | Tagged: , , , , , , , ,

Question Order, Question Length, And Question Context #AAPOR

By LoveStats on

Moderator: Jessica L. Holzberg, U.S. Census Bureau
Satisfied or Dissatisfied? Does Order Matter?; Jolene D. Smyth, University of Nebraska-Lincoln Richard Hull, University of Nebraska-Lincoln

  • Best practice is to use a balanced question stem and keep response options in order
  • What order should it be in the question stem
  • Doesn’t seem to matter whether the scale is left to right or top to bottom
  • Visual Heurstic Theory – help make sense of questions, “left and top mean first” and “up means good”, people expect the positive answer to come first, maybe it’s harder to answer if good is a the bottom
  • Why should the question stem matter, we rarely look at this
  • “How satisfied or dissatisfied are you?  [I avoid this completely by saying what is your opinion about this and then use those words in the scale, why duplicate words and lengthen questions]
  •  Tested Sat first and Disat second in the stem, and then Sat top and Disat bottom in the answer list, and vice versa
  • What would the non repsonse look like in these four options – zero differences 
  • Order in question stem had practically no impact, zero if you think about random chance
  • Did find that you get more positive answers when positive answer is first
  • [i think we overthink this. If the question and answers are short and simple, people change no trouble and random chance takes its course. Also, as long as all your comparisons are within the test, it won’t affect your conclusions]
  • [She just presented negative results. No one would ever do that in a market research conference 🙂 ]

Question Context Effects on Subjective Well-being Measures; Sunghee Lee, University of Michigan Colleen McClain, University of Michigan

  • External effects – weather, uncomfortable chair, noise in the room
  • Internal effects – survey topic, image, instructions, response sale, question order
  • People don’t view questions in isolation, it’s a flow of questions
  • Tested with life satisfaction and self-rated health, how are the two related, does it matter which one you ask first; how will thinking about my health satisfaction affect my rating of life satisfaction
  • People change their behaviors when they are asked to think about mortality issues, how is it different for people whose parents are alive or deceased
  • High correlations in direction as expected
  • When primed, people whose parents are deceased expected a lesser lifespan 
  • Primed respondents said they considered their parents death and age at death
  • Recommend keeping the questions apart to minimize effects [but this is often/rarely possible]
  • Sometimes priming could be a good thing, make people think about the topic before answering

Instructions in Self-administered Survey Questions: Do They Improve Data Quality or Just Make the Questionnaire Longer?

Cleo Redline, National Center for Education Statistics Andrew Zukerberg, National Center for Education Statistics Chelsea Owens, National Center for Education Statistics Amy Ho, National Center for Education Statistics

  • For instance, if you say “how many shoes do you have not including sneakers”, and what if you have to define loafers
  • Instructions are burdensome and confusing, and they lengthen the questionnaire 
  • Does formatting of instructions matter
  • Put instructions in italics, put them in bullet points because there were several somewhat lengthy instructions
  • Created instructions that conflicted with natural interpretation of questions, eg assessment does not include quits or standardized tests
  • Tried using paragraph or list, before or after, with or without instructions
  • Adding instructions did not change mean responses 
  • Instructions intended to affect the results did actually do so, I.e., people read and interpreted the instructions
  • Instructions before the question are effective as a paragraph
  • Instructions after the question are more effective as lists
  • On average, instructions did not improve data question, problems are real bu they are small
  • Don’t spend a lot of time on it if there aren’t obvious gains
  • Consider not using instructions

Investigating Measurement Error through Survey Question Placement; Ashley R. Wilson, RTI International Jennifer Wine, RTI International Natasha Janson, RTI International John Conzelmann, RTI International Emilia Peytcheva, RTI International

  • Generally pool results from self administered and CATI results, but what about sensitive items, social desirability, open end questions, what is “truth”
  • Can evaluate error with fictitious issues – e.g., a policy that doesn’t exist [but keep in mind policy names sound the same and could be legitimately misconstrued ]
  • Test using reverse coded items, straight lining, check consistency of seeming contradictory items [of course, there are many cases where what SEEMS to contradict is actually correct, e.g., Yes, I have a dog, No I don’t buy dog food; this is one of the weakest data quality checks]
  • Can also check against administrative data
  • “AssistNow” loan program did not exist [I can see people saying they agree becuase they think any loan program is a good thing]
  • On the phone, there were more substantive answers on the phone, more people agreed with the fictitious program [but it’s  a very problematic questions to begin with]
  • Checked how much money they borrowed, $1000 average measurement error [that seems pretty small to me, borrow $9000 vs $10000 is a non-issue, even less important at $49000 and $50000]
  • Mode effects aren’t that big

Do Faster Respondents Give Better Answers? Analyzing Response Time in Various Question Scales; Daniel Goldstein, NYC Department of Housing Preservation and Development; Kristie Lucking, NYC Department of Housing Preservation and Development; Jack Jerome, NYC Department of Housing Preservation and Development; Madeleine Parker, NYC Department of Housing Preservation and Development; Anne Martin, National Center for Children and Families

  • 300 questions, complicated sections, administered by two interviewers, housing, finances, debt, health, safety, demographics; Variety of scales throughout
  • 96000 response times measured, left skewed with a really long tail
  • Less education take longer to answer questions, people who are employed take longer to answer, older people take longer to answer, and none glish speakers take the longest to answer
  • People answer more quickly as they go through the survey, become more familiar with how the survey works
  • Yes no are the fastest, check all that apply are next fast as they are viewed as yes no questions
  • Experienced interviewers are faster
  • Scales with more answer categories take longer

Posted in: marketing research | Tagged: , , , ,

Rights Of Respondents #AAPOR

By LoveStats on

Live note taking at #AAPOR in Austin Texas. Any errors or bad jokes are my own.


Moderator: Ronald Langley, University of Kentucky

Examining the Use of Privacy Language: Privacy from the Respondent’s View; Kay Ricci, Nielsen Lauren A. Walton, Nielsen Ally Glerum, Nielsen Robin Gentry, Nielsen

  • Respondents have concerns about the safety and security of their data, Want do know how data is stored, collected, We hear about breaches all the time now
  • In 2015, FCC issued TCPA rules re automatic telephone dialling systems, can’t use them for cell phones without consent
  • Existing page of legalize was terrifying and could affect key metrics
  • 3 steps – in depth interviews, implemented language into TV diary screener, analyzed key metrics to understand impact of new language
  • 30 English interviews, 16 Spanish interviews
  • Did people notice the language, read it, skim it, did they care? Did they understand the terms, was the language easy or difficult
  • Tested several versions, tested a longer version with softer language
  • Only one person understood what an autodialler was, people didn’t realize it was a live interviewer, people didn’t care how their number was dialled if they were going to talk to a live human anyways
  • 2/3 didn’t like the concept, thought they’d be phoned constantly, 1/3 didn’t mind because it’s easier to hang up on a machine
  • People liked we weren’t selling or marketing products, but many don’t see the difference
  • Many people don’t know who neilsen is
  • People liked being reminded that it was voluntary, extra length was fine for this
  • The after version was longer with more broken up sentences
  • Test group had lower return rate but very slightly, lower mail rate
  • Higher return rate for 50 plus, and Hispanic
  • Contact number provision was much lower, drop from 71% to 66%
  • It’s essential to protest so you know the impact
  • [simple language is always better even if it takes more space]

Allowing Access to Household Internet Traffic: Maximizing Acceptance of Internet Measurement; Megan Sever, Nielsen Sean Calvert, Nielsen

  • How do we measure what people watch and buy online in their home
  • How do we access third party data , but then how do we great true demographic information to go with it
  • 22 semi structured interviews – mock recruit into please share your password
  • Ranges from absolutely yes – they believe it’s already being collected anyways
  • Sceptics wanted more information – what are you actually recording, how is my data secure
  • Privacy – security – impact on Internet performance
  • People seemed to think they would Screencap everything they were doing, that they could see their bank account
  • Brought examples of real data that would be collected, what the research company will see, essentially lines of code, only see URL, not the contents of the page, start and stop times; at this point participants were no longer concerned
  • Gave a detailed description of encryption, storage and confidentially procedures
  • Explain we’re not marketing or selling and data is only stored as long as is necessary
  • Reputation of the research company builds trust, more familiar folks were okay with it
  • Script should describe purpose of measurement, what will and will not be measured, how it will be measured, data security privacy and protection policies, impact on Internet performance, reputation of company
  • Provide supplementary information is asked for – examples of data, policies that meet or exceed standards, examples of Internet performance, background and reputation of company 

Informed Consent: What Do Respondents Want to Know Before Survey Participation; Nicole R. Buttermore, GfK Custom Research Randall K. Thomas, GfK Custom Research Jordon Peugh, SSRS; Frances M. Barlas, GfK Custom Research Mansour Fahimi, GfK Custom Research

  • Recall the kerfuffle last year about what information should be told to respondents re sponsor or person conducting the research 
  • Recognized participants should be told enough information to give informed consent – but also if we are concerned about bias, then we can tell people they won’t be debriefed until afterwards; but companies said sometimes they could NEVER review the sponsor and they’d have to drop out of #AAPOR if this happened
  • We worry about bias and how knowing the sponsors affects the results
  • Sponsor information is a less important feature to respondents
  • Do respondents view taking sureys as risky? What information to respondents want prior to computing surveys.
  • Topic, my time, and my incentive are thought to be most important
  • People were asked about surveys in general, not just this one or just this company
  • 6% thought an online survey could have a negative impact 
  • Most worried about breaks of privacy, confidentially; less worried is survey is waste of time or boring, or might upset them
  • 70% said no risk to mental health, 2% said high risk to mental health
  • 23% said stopped taking a survey because it made them uncomfortable – made think more deeply about life, made them angry, made them feel worse about themselves, made them feel sad, or increased their stress
  • Politics made them angry, bad questions made them angry, biased survey and too long survey made them angry [That’s OUR fault]
  • Same for feeling stressed, but also add in finance topics
  • Feel worse about self is the finance topic or health, or about things they can’t have
  • Feel sad related to animal mistreatment
  • People want to know how personal information will be protected, surely length, risks, topic, how results will be used, incentives, purpose of survey – at least one third of people [1/3 might not seem like a lot but when you’re sample is 300 people that’s 100 people who want to know this stuff]
  • Similar online vs phone, incentives more important for online, one the phone people wan to know what types of questions will be asked

Communicating Data Use and Privacy: In-person versus Web Based Methods for Message Testing; Aleia Clark Fobia, U.S. Census Bureau Jennifer Hunter Childs, U.S. Census Bureau

  • Concern about different messages in different place and they weren’t consistent
  • Is there a difference between “will only be used for statistically purpose” and “will never be used for non statistical purposes”
  • Tested who will see data, identification, sharing with other departments, burden of message
  • Tested it on a panel of census nerds :), people who want to be involved in research, 4000 invites, 300 completes
  • People were asked to explain what each message means, broke it down by demographics
  • 30 cognitive interviews, think aloud protocol, reads sets of messages and react, tested high and low performing messages [great idea to test the low messages as well]
  • FOcused on lower education and people of colour
  • Understanding is higher for in person testing, more misunderstanding in online responses, “You are required by law to respond to the census (technical reference)” was better understood than listing everything in a statement
  • People want to know what ‘sometimes’ means. And want to know which federal agencies – they don’t like the IRS
  • People don’t believe the word never because they know breaches happen
  • More negative on the web
  • Less misunderstanding in person
  • Easier to say negatives things online
  • In person was spontaneous and conversation
  • Focus on small words, avoid unfamiliar concepts, don’t know what tabulate means, don’t know what statistical means [they aren’t stupid, it’s just that we use it in a context that makes no sense to how they know the word]

    Respondent Burden & the Impact of Respondent Interest, Item Sensitivity and Perceived Length; Morgan Earp, U.S. Bureau of Labor Statistics Erica C. YuWright, U.S. Bureau of Labor Statistics

    • 20 knowledge questions, 10 burden items, 5 demographic questions, ten minute survey
    • Some questions were simple, others were long and technical
    • Respondents asked to complete a follow up survey a week later
    • Asked people how hard the survey was related to taking an exam at school or reading a newspaper or completing another survey – given only one of these comparisons 
    • Anchor of school exam had a noticeable effect size but not significant 
    • Burden items – length, difficulty, effort, importance, helpfulness, interest, sensitivity, intrusive, private, burden
    • Main effects – only sensitivity was significant, effect size is noticeable
    • Didn’t really see any demographic interactions
    • Burden length difficult; effort importance helpfulness interesting; sensitive intrusive private – these are the three factors 
    • Only first factor related to whether they would answer the second survey
    • Females more likely to respond a second time
    • More sensitive less likely to be answered again, more interestnig in would attract more women the second dime

      Posted in: marketing research | Tagged: , , , ,

      Data Quality Issues For Online Surveys #AAPOR

      By LoveStats on

      Moderator: Doug Currivan, RTI International Location: Meeting Room 410, Fourth Floor
      Impact of ‘Don’t Know’ Options on Attitudinal and Demographic Questions; Larry Osborn, GfK Custom Research; Nicole R. Buttermore, GfK Custom Research Frances M. Barlas, GfK Custom Research Abigail Giles, GfK Custom Research

      • Telephone and in person rarely offer a don’t know option but they will record it, offering it doesn’t improve data
      • May not be the case with online surveys
      • They offered a prompt following nonresponse to see how it changed results
      • Got 4000 completes
      • Tested attitudinal items – with, without, and with a prompt
      • don’t know data was reduced after a prompt, people did choose an opinion, it was effective and didn’t affect data validity
      • Tested it on a factual item as well, income, which is often missing up to 25% of data
      • Branching income often helps to minimize nonresponse (e.g., start with three income groups and then each group is split into three more groups)
      • 1900 completes for this question – 35k or below, > 35k, DK, and then branched each break; DK was only offered for people who skipped the question
      • Checked validity by correlations with income related variables (e.g., education, employment)
      • Lower rates of missing when DK is offered after nonresponse, it seems most missing data is satisficing 

      Assessing Changes in Coverage Bias of Web Surveys as Internet Access Increases in the United States; David Sterrett, NORC at the University of Chicago Dan Malato, NORC at the University of Chicago Jennifer Benz, NORC at the University of Chicago Trevor Tompson, NORC at the University of Chicago Ned English, NORC at the University of Chicago

      • Many people don’t have Internet access but nowadays it’s much better, can we feel safe with a web only survey
      • Is coverage bias minimal enough to not be worried – people who don’t have access to Internet 
      • It can be question by question, not just for the survey overall
      • Coverage bias is a problem if there are major differences between those with coverage and without, if they are the same kinds of people it won’t matter as much
      • Even when you weight data, it might not be a representative sample, weights don’t fix everything
      • ABS – address based sampling design – as opposed to telephone number or email address based
      • General social survey has information about whether people have Internet access and it has many health, social, political, economic issues; can see where coverage error happens
      • Income, education, ethnicity, age are major predictors of Internet access as predicted
      • What issues are beyond weighting on demographics
      • For many issues, there was a less than 1% point coverage error
      • For health, same sex marriage, and education, the differences were up to 7% point different
      • Over time – bias decreased for voting, support for assistance of blacks; but error increased for spend on welfare, marijuana, getting ahead in life 
      • Saw many differences when they looked into subgroups
      • [so many tests happening, definitely need to see replication to rule out which are random error]
      • As people who don’t have Internet access become more different from people who have it, we need to be cognizant of how that skews which subset of results
      • You can’t know whether the you are research is a safe one or not

      Squeaky Clean: Data Cleaning and Bias Reduction; Frances M. Barlas, GfK Custom Research Randall K. Thomas, GfK Custom Research Mansour Fahimi, GfK Custom Research Nicole R. Buttermore, GfK Custom Research

      • Do you need to learn your data [yes, because you don’t know if errors sit within a specific group of people, you need to at least be aware of the quality]
      • Some errors are intentional, others accidental, or they couldn’t find the best answer
      • Results did not change if no more than 5% of the data was removed
      • Is there such a thing as too much data cleaning
      • Cleaned out incremental percentages of data and then weighted to census, matched to census data as the benchmark
      • Saw no effect with cleaning up to 50% of the data with one of the panels, similar with the second almost no effect of cleaning
      • [given that different demographic groups have different data quality, it could matter by subsample]

      Trap Questions in Online Surveys; Laura Wronski, SurveyMonkey Mingnan Liu, SurveyMonkey

      • Tested a variety of trap questions, easy or hard, beginning or end – used the format of selecting an answer they specify
      • 80% were trapped with the hard question
      • [saddens me that we talk about ‘trapping’ respondents.. They are volunteering their time for us. We must treat them respectfully. Trap questions tell respondents we don’t trust them.]
      • Tested follow ups and captcha 
      • Announcements didn’t result in any differences, picture verification question trapped about ten percent of people
      • Captcha trapped about 1% [probably they couldn’t read it]
      • Prefered the picture trap
      • [I personally prefer using many questions because everyone makes errors somewhere. Someone who makes MANY errors is the problem, not someone who misses one question.]
      • At the end of the survey, asked people if they remembered the data quality question – many people didn’t notice it
      • One trap question is sufficient [wow, I disagree so much with this conclusion]

      Identifying Psychosocial Correlates of Response in Panel Research: Evidence from the Health and Retirement Study; Colleen McClain, University of Michigan – MAPOR student paper winner

      • People who are more agreeable are less likely to participate (big 5 traits)
      • More conscientious are more likely to participate 
      • More agreeable people took longer to respond to the survey
      • Conscientious  people respondent more quickly
      • More distrustful are less likely to check their records
      • Effects were very small
      • We need to consider more than demographics when it comes to data quality
      Posted in: marketing research | Tagged: , , ,

      Questionnaire Design #AAPOR 

      By LoveStats on | 2 Comments

      Live note taking at #AAPOR in Austin Texas. Any errors or bad jokes are my own.

      The feedback of respondent committment and tailored feedback on response quality in an online survey; Kristin Cibelli, U of Michigan

      • People can be unwilling or unable to provide high quality data, will informing them of the importance and asking for committment help to improve data quality [I assume this means the survey intent is honourable and the survey itself is well written, not always the case]
      • Used administrative records as the gold standard
      • People were told their answers would help with social issues in the community [would similar statements help in CPG, “to help choose a pleasant design for this cereal box”]
      • 95% of people agreed to the committment statement, 2.5% did not agree but still continued; thus, we could assume that the control group might be very similar in committment had they been asked
      • Reported income was more accurate for committed respondents, marginally significant
      • Overall item nonresponse was marginally better for committed respondents, not committed people skipped more
      • Not committed were more likely to straightlining 
      • Reports of volunteering, social desirability were possibly lower in the committed group, people confessed it was important for the resume
      • Committed respondents were more likely to consent to reviewing records
      • Committment led to more responses to income question, and improved the accuracy, more likely to check their records to confirm income
      • Should try asking control group to commit at the very end of the survey to see who might have committed 

      Best Practice Instrument design and communications evaluation: An examination of the NSCH redesign by William Bryan Higgins, ICF International

      • National and state estimates of child well-being 
      • Why redesign the survey? To shift from landline and cell phone numbers to household address based sampling design because kids were answering the survey, to combine two instruments into one, to provide more timely data
      • Moe to self completion mail or web surveys with telephone follow-up as necessary
      • Evaluated communications about the survey, household screener, the survey itself
      • Looked at whether people could actually respond to questions and understand all of the questions
      • Noticed they need to highlight who is supposed to answered the survey, e.g., only for households that have children, or even if you do NOT have children. Make requirments bold, high up on the page. 
      • The wording assumed people had read or received previous mailings. “Since we last asked you, how many…”
      • Needed to personalize the people, name the children during the survey so people know who is being referred to 
      • Wanted to include less legalese

      Web survey experiments on fully balanced, minimally balanced, and unbalanced rating scales by Sarah Cho, SurveyMonkey

      • Is now a good time or a bad time to buy a house. Or, is now a good time to buy a house or not? Or, is now a good time to buy a house?
      • Literature shows a moderating effect for education
      • Research showed very little difference among the formats, no need to balance question online
      • Minimal differences by education though lower education does show some differences
      • Conclusion, if you’re online you don’t need to balance your results

      How much can we ask? Assessing the effect of questionnaire length on survey quality by Rebecca Medway, American Insitute for research

      • Adult education and training survey, paper version
      • Wanted to redesign the survey  but the redesign was really long
      • 2 version were 20 pages and 28 pages, 138 questions or 98 questions
      • Response rate slightly higher for shorter questionnaire
      • No significant differences in demographics [but I would assume there is some kind of psychographic difference]
      • Slightly more non-response in longer questionnaire
      • Longer surveys had more skips over the open end questions
      • Skip errors had no differences between long and short surveys
      • Generally longer had lower repsonse rate but no extra problems over the short 
      • [they should have tested four short surveys versus the one long survey 98 is just as long as 138 questions in my mind]
      Posted in: marketing research | Tagged: , , , , ,

      Social noise – cacophony or symphony #MRSlive @TweetMRS #MRX 

      By LoveStats on

      Live blogged at MRS in London. Any errors or bad jokes are my own.

      The art of noise by Tom Ewing

      • #IPAsocialworksinitiative – to guide effective use of social media data
      • It’s all about measuring not counting, Ray Poynter wrote this part
      • The insight guide  is being written by Tom Ewing and he is interviewing people for it
      • Five basic questions
      1. What questions do you need to answer, what decisions will betaken because of it
      2. What do you know already, existing research, behavioural
      3. What social behaviora are you looking for, what data sources, do you need aggregate data or granular level
      4. What is the relevant timescale, ad hoc or continuous, how do you know yyou’re finished
      5. Do you need quant metrics, what are they, it’s more than likes follows
      • We have reached the peak of social tech
      • Seth Grimes says the state of social insight is “confused,” it’s not really a mature technology, widely adopted, but not fully integrated
      • It’s the early majority now, tactical, silos, unstructured text, the future is here but it is unevenly distributed
      • We need to clear out the cliches, integrate with methodologies, integrate with insights and business and people 
      • Different poll results would arise if you use Twitter vs Facebook [i see this as a sampling problem, you don’t just take any church or school or company and say it is representative]
      • UK Facebook data is closer to a census than a sample [but it is still very much a sample!]
      • You can join insights with other methodologies [data fusion is ALWAYS wise, every method reveals different insights]
      • Provide more file and predictive tracking systems and an ability to get granular and specific
      • Who owns social insight? Is it marketing? Analytics? Insights? PR? Sales? It’s more about collaborating  [why must one group OWN it, everyone can benefit hugely from it]
      • Finding common ground – we’re all working toward the same goal, emotional understanding of people
      • Limits of real time means it’s more tactical but there is indeed strategic value
      • How do you get insight about a topic when you know nothing about it [oh goodness, I could go on about this. I was asked to draw insights about cars and trucks. I know NOTHING about them and couldn’t understand anything people were talking about. Expertise in the topic is a MUST! Put me on the food/outdoors/music research.]
      • AI is already doing so much work that people would have done – collecting data, sorting and coding, analyzing, prepping, reporting
      • Social insight begins with social practice, use new platforms like Whatsapp and snapchat, look beyond the brand level to the topic level, consider pictures and video, look for patterns at the network level
      • Different things come out via text and images but you wno’t know unless you plan and look for differences in different types of content
      • We are working towards general use, stratégie, cooperation, seamless integration among people, rich media as well as unstructured text
      • Tom asks people to submit their social media case studies to him for publication in the book

      Finding insight in 140 characters by Jake Steadman

      • Jake [not from Statefarm] speaks
      • Story about Star Wars, in 1977, need to tap into mainstream media, tap into core group of fans, went on TV and publications. FOr second movie he went to com icon. Fans went in the hundreds to see them in person for Q and A.
      • Today, mobile is eating other media time.everyone has a mobile phone. December of last year, new Star Wars came out. Still need media and interest and fan base. THey launched on Twitter. That tweet drove all the other media. Drove coverage in print. Access to the stars still applies but they did it more directly and personal on Twitter. Stars did this on the red carpet, backstage, White House visits, all on Twitter. 1.2 billion tweets on opening weekend.
      • All social data can drive insight
      • There are culture issues and there are tools. 
      • Rise of the machines, machines allow you to become storytellers and consultants, it allows you to be a leader not a methodological policeman
      • Let’s you do same day insights and be agility, we need to get over our obsession with precision, forget statistical precision [agreed, don’t create precision with decimal places, know when your data is directional and don’t make it more than it is ]
      • Social is the democratization of data, client side is more like to be using social, agencies are more likely to be using big data, agencies need to be more for less and quicker
      • Look for your soggy fries – customers were starting be less loyal with a restaurant, you could run a large research project, or you could look at social data immediately, found many uses of the word ‘soggy fries’, it didn’t need precision but rather it needed recognition

      Panel with Jenny Burns, Christopher Wellbelove, Jess Owens, James Devon

      • Some people jump at social media because it’s cheaper than things like focus groups [be cautious, cheaper means you need to spend more time thinking and analyzing it]
      • Lots of people are analyzing social media without ever using it, you need to use the tools to really understand them [the speaker has now gone through three pairs of google glasses, and might be tweeting as we speak 🙂 ]
      • Social media is still treated as a slightly dangerous cousin
      • 67% of people using social media puts it really into late adoption, it is not new nor early adopter territory 
      • It says more about the market research industry being years behind to be so late in using this technology
      • Privacy and ethics still matter, some companies get individual permission to review Twitter accounts
      • People need to understand how their data is being used, whether it’s aggregated, Facebook data is fully anonymized and aggregated and great for ethics and privacy

      [I kind of like have a couple presentations followed by a separate panel]

        

        Posted in: marketing research | Tagged: , , , , ,

        Analysis, design, and sampling methods #PAPOR #MRX 

        By LoveStats on

        Live blogged at #PAPOR in San Francisco. Any errors or bad jokes are my own.

        Enhancing the use of Qualitative Research to Understand Public Opinion, Paul J. Lavrakas, Independent Consultant; Margaret R. Roller, Roller Research

        • thinks research has become to quantitative because qual is typically not as rigorous but this should and can change
        • public opinion in not a number generated from polls, polls are imperfect and limited
        • aapor has lost sight of this [you’re a brave person to say this! very glad to hear it at a conference]
        • we need more balance, we aren’t a survey research organization, we are a public opinion organization, our conference programs are extremely biased quantitative
        • there should be criteria to judge the trustworthyness of research – was it fit for purpose
        • credible, transferable, dependability, confirmability
        • all qual research should be credible, analyzable, transparent, useful
        • credible – sample representation and data collection
        • do qual researchers seriously consider non-response bias?
        • credibility – scope deals with coverage design and nonresponse, data gathering – information obtained, researcher effects, participant effects
        • analyzability – intercoder reliability, transcription quaity
        • transparency – thick descriptions of details in final documents

        Comparisons of Fully Balanced, Minimally Balanced, and Unbalanced Rating Scales, Mingnan Liu, Sarah Cho, and Noble Kuriakose, SurveyMonkey

        • there are many ways to ask the same question
        • is it a good time or a bad time? – fully balanced
        • is it a good time or not? – minimally balanced
        • do you or do you not think it is getting better?
        • are things headed in the right direction?
        • [my preference – avoid introducing any balancing in the question, only put it in the answer. For instance: What do you think about buying  a house? Good time, Bad time]
        • results – effect sizes are very small, no differences between the groups
        • in many different questions tested, there was no difference in the formats

        Conflicting Thoughts: The Effect of Information on Support for an Increase in the Federal Minimum Wage Level, Joshua Cooper & Alejandra Gimenez, Brigham Young University, First Place Student Paper Competition Winner

        • Used paper surveys for the experiment, 13000 respondents, 25 forms
        • Would you favor or oppose raising the minimum wage.
        • Some were told how many people would increase their income, some were told how many jobs would be lost, some were told both
        • Negative info opposed a wage increase, positive info in favor of wage increase, people who were told both opposed a wage increase
        • independents were more likely to say don’t know
        • negatively strongly outweighs the good across all types of respondents regardless of gnder, income, religion, partyID
        • jobs matter, more than anything
        Posted in: marketing research | Tagged: , , , , , ,