What started out as a summary of the conference turned into an entirely different post – DIY surveys. You’ll just have to wait for my summary then!
My understanding is that this was the first time SurveyMonkey spoke at an #AMSRS conference. It resulted in what seemed to be perceived by the audience as a controversial question and it was asked in an antagonistic way – what does SurveyMonkey intend to do about the quality of surveys prepared by nonprofessionals. This is a question with a multi-faceted answer.
First of all, let me begin by reminding everyone that out of all the surveys prepared by professional, fully-trained survey researchers, most of those surveys incorporate at least a couple of bad questions. Positively keyed grids abound, long grids abound, poorly worded and leading questions abound, overly lengthly surveys abound. For all of our concerns about amateurs writing surveys, I sometimes feel as though the pot is calling the kettle black.
But really, this isn’t a SurveyMonkey question at all. This is a DIY question. And it isn’t a controversial question at all. The DIY issue has been raised for a few years at North American conferences. It’s an issue with which every industry must deal. Taxis are dealing with Uber. Hotels are dealing with AirBnB. Electricians, painters, and lawn care services in my neighbourhood are dealing with me. Naturally, my electrical and painting work isn’t up to snuff with the professionals and I’m okay with that. But my lawn care services go above and beyond what the professionals can do. I am better than the so-called experts in this area. Basically, I am the master of my own domain – I decide for myself who will do the jobs I need doing. I won’t tell you who will do the jobs at your home and you won’t tell me who will do my jobs. Let me reassure you, I don’t plan to do any home surgery.
You can look at this from another point of view as well. If the electricians and painters did their job extremely well, extremely conveniently, and at a fair price, I would most certainly hire the pros. And the same goes for survey companies. If we worked within our potential clients’ schedules, with excellent quality, with excellent outcomes, and with excellent prices, potential clients who didn’t have solid research skills wouldn’t bother to do the research themselves. We, survey researchers, have created an environment where potential clients do not see the value in what we do. Perhaps we’ve let them down in the past, perhaps our colleagues have let them down in the past.
And of course, there’s another aspect to the DIY industry. For every client who does their own research work, no matter how skilled and experienced they are, that’s one less job you will get hired to do. I often wonder how much concern over DIY is simply the fear of lost business. In this sense, I see it as a re-organization of jobs. If research companies lose jobs to companies using DIY, then those DIY company will need to hire more researchers. The jobs are still there, they’re just in different places.
But to get back to the heart of the question, what should DIY companies do to protect the quality of the work, to protect their industry, when do-it-yourselfers insist on DIY? Well, DIY companies can offer help in many forms. Webinars, blog posts, and white papers are great ways to share knowledge about survey writing and analysis. Question and survey templates make it really easy for newbies to write better surveys. And why not offer personalized survey advice from a professional. There are many things that DIY companies can do and already do.
Better yet, what should non-DIY companies do? A better job, that’s what. Write awesome surveys, not satisfactory surveys. Write awesome reports, not sufficient reports. Give awesome presentations, not acceptable presentations. Be prompt, quick, and flexible, and don’t drag clients from person to person over days and weeks. When potential clients see the value that professional services provide, DIY won’t even come to mind.
And of course, what should research associations do? Advocate for the industry. Show Joe nonresearcher what they miss out on by not hiring a professional. Create guidelines and standards to which DIY companies can aspire and prove themselves.
It’s a DIY world out there. Get on board or be very, very worried.
Wake Up or Die. Research Automation – The Future of Market Research Corinne Sandler, Fresh Intelligence #NetGain2015 #MRX
Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.
Wake Up or Die. Research Automation – The Future of Market Research
Corinne Sandler, CEO of Fresh Intelligence
- No war in the world has ever been won without intelligence
- we are needed more and more as competition among brands intensifies
- Where is market research on the Montclare SaaS250 list? Salesforce is the closest one.
- We need to think outside the fishbowl
- We play a consultative role, we see a brief that goes from analyst to director and we consult all along the way. What if the brains were integrated into technology.
- We need to move from consulting on research to dispensing research.
- Progress depends on the unreasonable man – George Bernard Shaw
- Intelligence needs to be accessible to the world
- You have only two choices today – you can use a third party or you can do it yourself
- What about combining brains with an automated component, fully automated turn-key platform with consultants
- Buyers of research have specific needs, and automation suits those needs well. Quality is a cost of entry, must believe that is what consumer is truly saying, need to rely on that right data.
- Buyers need to prove value of every dollar they spend. Priorities don’t always allow them to share the wealth among several brands.
- These methods allow you to do all the work without reveal proprietary information
- For instance, what if you told the provider 5 brand names, and the system automatically plugged them into a pre-programmed survey with pre-programmed charts.
- How do you create an app that adds value for toilet paper, because EVERYONE has an app. “Charmin Sit or Squat App” – tells you where the closest washroom is and how clean it is. [My mom would LOVE THIS!]
- 90% of our decisions are based on intuition
Jack Be Nimble: Faster & Richer Insights Through Insourcing
Richard Shakarchi, E*TRADE FINANCIAL
- you can’t have faster, cheaper, better – one will always suffer. it’s a zero sum game
- There is a pyramid of services
- do sample providers provide direct value? not terribly [HEY! disagree!] easily outsourced.
- Research operations gets work programmed. do they add direct value?
- What about project management who makes sure things are done on time? Could be outsourced
- Design and reporting translate business need into research design. sometimes outsourced.
- Then comes insight at the very top. frequently omitted.
- Project information and design and reporting people give you tons and tons of data. but where is the insight. are we training anyone to become insights professionals?
- budgetary constraints are driving industry change [do you think budget would be an issue if researchers proved they were essential? no]
- many innovative techniques actually cost more
- budget constrained researchers can use omnibus surveys, google consumer surveys, good for generic simply research but not the deep dives
- if money determines which method you are using, then you are sacrificing something
- DIY research has a really bad reputation. OPen access to non-professional less to less than stellar control, quality of research deteriorated, limited capabilities for better design. Yes, cost went down but so did quality.
- Since early days, DIY has dramatically evolved, it’s not just surveys any more. we can do user testing of websites now, face-to-face interviewing, online communities
- I know I CAN DIY, but should I?
- Read “The Innovator’s Dilemma” As an industry evolves, new technologies will emerge. In the early days, the tech won’t be that great and people will ignore those tools. But just you wait. They will eat at the heels of the higher quality providers. But those “new” technologies eventually become just as good and now they are cheaper.
- “Guns don’t kill, people do” Don’t blame the DIY tools for the problems of those who use the tools.
- Break the paradigm. Among skilled researchers DIY is faster, better, cheaper.
- Own the insights. Clients are the category expert. They know all the products inside and out, the financials, and how the survey was programmed. No one is better qualified to explain the insights.
- If you do it long enough, you will have your own benchmarks for all the same types of measures your research supplier had.
- make sure the DIY team has all the expertise they need from consulting to project management, research design, question writing, survey programming, database skills, data processing, basic and intermediate statistics, storytelling -[okay, one person can’t know it all but your team can]
At the CASRO online research conference, one of the panels focused on DIY research and included a couple panel providers and a couple DIY companies. This panel was of particular interest to me because I’ve watched how DIY has taken quite a pounding in the last few years but for the wrong reasons.
I asked the panel about when qualified researchers should use Do It Yourself research and the answers included when you need results better, faster, and cheaper. Ok, a very generic and unhelpful response.
Then, another audience member asked when should DIY research NOT be used. Sadly, the panel could not offer a single idea about when DIY research was not appropriate. Given that one of my mantras is “Every research method has pros and cons,” this was a completely disatisfactory answer. And misleading.
So here is my opinion on when DIY should and should NOT be used.
DO use DIY research when:
- A qualified researcher has written the survey and designed the methodology
- The survey is very simple, short, and contains no complicated skip patterns
- You need results extremely quickly
- You have the population of target responders and ‘random’ sampling is not necessarily required
- You understand statistics well enough to know when sample sizes are too small, when to use a t-test or a chi-sqare, and why MOE is under hot debate
- You need to test a simple methodological issue prior to launching the full study (e.g., will the distribution of responses be better served with a 5 point or 3 point scale)
Do NOT use DIY research when:
- You don’t need anyone to proof read your survey because you never make mistakes
- An expert in survey design has not created the survey
- An expert in sampling/weighting has not developed and implemented the sampling plan
- An expert in data analysis will not be analyzing the results
- (so assuming that a qualified researcher is managing all aspects of the research…)
- You are running a complicated tracker – weekly/monthly, complex balancing, multi-country
- Your survey incorporates multiple and varied skip and logic patterns
- You require complicated census/target balancing and weighting
It’s a pretty easy answer.
- DIY Panel: Gardlen, Ribeiro, Smith, Terhandian, Thomas #CASRO #MRX (lovestats.wordpress.com)
- Bringing Colour into our Digital Lives: Piet Hein van Dam #CASRO #MRX (lovestats.wordpress.com)
- Cyborgs vs Monsters in modularizing surveys: Edward Paul Johnson and Lynn Siluk #CASRO #MRX (lovestats.wordpress.com)
Do It Yourself (DIY) Research Panel Discussion
- John Bremer, session moderator
- Bob Fawson, session moderator
- Phillip Garland, Vice President, Methodology, SurveyMonkey
- Efrain Ribeiro, Chief Research Officer, Lightspeed Research
- Ryan Smith, Co-founder and CEO, Qualtrics
- George Terhanian, N. A. President and Group Chief Strategy Officer, Toluna
- Randall Thomas, Vice President – Online Research Methods, GfK
- DIY is a source for innovation
- Think about DIY checkout lines vs DIY research. There are risks with it but the experts are still available is need be
- Data is data but DIY doesn’t solve the insight and analysis. You still need the researcher for that.
- You can get numbers out of a machine but do the numbers have any meaning
- Concern with DIY is are people writing good surveys, are they using samples properly, are they weight appropriately, are they analyzing data appropriately. DIY and Do-It-Alone are different. You always need the expertise around you.
- Current clients of DIY include people who didn’t have access to research before as well as many of the major research companies
- DIY can often get work done much more quickly
- When to use DIY – when you don’t want to do the work yourself, when you need results extremely quickly, when it’s not a major/serious issue, when you don’t have the staff for it, goor for an organization, good when you have standardized tools
- DIY is simply part of the assembly line
- The researcher of tomorrow will be comfortable with DIY tools
- When is DIY NOT appropriate – [folks didn’t answer this audience question 🙂 how about DIY shouldn’t be used for census rep weekly/monthly tracking over 12 months. it’s far too complicated to just throw in a tool.]
- Keynote: Reinventing Online Markets by Gayle Fuguitt #CASRO #MRX (lovestats.wordpress.com)
- Do I have your attention? By Pete Cape #CASRO #MRX (lovestats.wordpress.com)
- Does Sample Size still matter? By David Bakken #CASRO #MRX (lovestats.wordpress.com)
Welcome to my #Netgain6 MRIA live blogs. What happens at St. Andrews Conference Centre, gets blogged for all to read about. Each posting is published immediately after the speaker finishes. Any inaccuracies are my own. Any silly comments in [ ] are my own. Enjoy!
Bernie Malinoff, President, Element54
The Yin and Yang of Gamification
- Grids are toxic for consumers and addictive for researchers
- Bernie accepts he has a problem 🙂
- Survey “Engagement” space race
- Just because you have a flash question doesn’t mean you should use it – a yes no question is still a yes no question
- DIY is beneficial to all and makes us better consultants
- You could do the same question with a radio button vs slide vs facial expressions vs visual skiier game – Why not first decide WHY you would choose one of the methods, take a purpose based approach
- Issue of straightlining – Grid was 10%, flash question was 6%, flash question plus “bet $20” was 0% straightlining
- Data cha0s – 40% variance in answers due to different methodologies
- Gamification can be visual or linguistic
- Do you put a little bit in a survey when it’s boring? only at the beginning? only at the end? pepper it throughout? THINK what makes sense
- Flash surveys are 20% longer. Game version is 73% longer. But these options are much more enjoyable.
- Adoption hurdles: norms (blow on a tracker and the results will change), scalability (efficiency of programming all surveys the same way, surveys go off to Santa’s workshop and come back all done)
- I Love Lucy’s factory is an example of efficiency and creativity…. maybe. Is this how our survey programmers feel today? We treat it as just a production process.
- You can’t replace the fundamentals with software
- Most researchers are analytical but maybe not spending much time being creative about our approach to research
- Big or small supplier/client, we share the same goals of better research and better data
I got in a fight at the Esomar 3D conference, a near death, weapons drawn, lay your stats on the line fight. About what? About DIY research. I was accused of bashing DIY on the stage in front of the entire Esomar audience. In my effort to outline the serious consequences of failing to sample correctly, clean data correctly, analyze data correctly, I bashed research that is conducted with inappropriate and insufficient tools. So to be clear, here is my stance regarding DIY research.
I am against researchers doing their own plumbing, doctors doing their own wiring, lawyers doing their own carpet laying, locksmiths doing their own lawyering. I am against inexperienced, unqualified people doing any task that endangers the quality of the work and the reputation of the industry. I am against Joe Smith writing bad questions, applying statistics incorrectly, and generalizing results improperly.
I’m not against DIY. I’m against poor quality. How about you?
- A Decade of Netnography #Eso3D #esomar (lovestats.wordpress.com)
- The Lives of Cool Digital Natives #Eso3D #esomar (lovestats.wordpress.com)
- The Researchification of Games by Peter Harrison #Eso3D #esomar (lovestats.wordpress.com)
- How Far is Too Far by Bernie Malinoff #Eso3D #esomar (lovestats.wordpress.com)
- Facial Imaging #Eso3D #esomar (lovestats.wordpress.com)
- Inspiring Online Dialogue by Rodriguez and Wong #Eso3D #MRX (lovestats.wordpress.com)
- Digital Matrimonies that Transform the Face of Research by Kristin Luck #Eso3D #MRX (lovestats.wordpress.com)
- The Earthquake and Tsunami in Japan #Eso3D #esomar (lovestats.wordpress.com)
- How to Engage Consumers in Multinational Communities by Austin and Lerman #Eso3D #MRX (lovestats.wordpress.com)
[tweetmeme source=”lovestats” only_single=false]
G’day all and welcome to this months issue of 1 topic, 5 blogs. Todays topic is DIY surveys. Links to my fellow bloggers Bernie Malinoff, Joel Rubinson, Josh Mendelsohn and Brandon Bertelsen can be found below.
I will admit. From a client point of view, DIY surveys have a ton of advantages. You can get the survey out quickly without any middle-man holding you up. Without a middle-man, the cost can be greatly reduced. And, you get exactly what you want without pleading and whining and complaining. If you have some solid experience or training in survey design, I am quite in favour of DIY for short 5 minute surveys.
But, when it comes to substantial surveys, I have many, many issues. The issues revolves mostly around “getting exactly what you want.” Any time I review survey drafts from survey newbies, and EVEN from people experienced in survey design, I find many of these terrible problems.
1) Leading questions: The writer usually has no intention of doing this, but they almost always give away the answer they want to receive. And whether consciously or unconsciously, the responder recognizes that and happily provides it.
2) Missing options: The writer is so focused on the options they are interested in that they forget there are options that are far more popular. They make sure that Brand A and Brand B are represented, but completely forget about Brand C, Store brand, Never buy, and Don’t know. This is another way to get the exact answer you want purely by bad survey design.
3) I can usually find a bunch of category jargon including words I don’t even recognize. On numerous occasions, I have seen surveys ask something like “Do you plan to purchase the X6000?” I am left trying to remember if the previous question was about televisions or motorcycles in hopes that X6000 will suddenly mean something.
4) I can usually find survey jargon. People don’t purchase, they buy. People don’t use gum, they chew gum. People don’t purchase the 75g bar soap package, they buy the two pack.
Even if you much prefer DIY, I 100% believe that everyone needs a middle man. Whether that middle-man is a reputable survey company or another person in a completely different department, you absolutely must have fresh eyes. Fresh eyes find the annoying mistakes that you can no longer see because you’re tired of reading your survey. Fresh eyes find logic errors, spelling mistakes, and unclear questions. Market researchers will criticize far more harshly resulting in a far better survey, but fresh eyes of any sort are always in your favour.
In the end, it doesn’t matter how much money you saved by going the DIY route if your survey results are useless.
Links to the other 4 blogs coming shortly: