Along with a group of market researchers from around the world, I was asked to participate in Voxpopme Perspectives – an initiative wherein insights industry experts share ideas about a variety of topics via video. You can read more about it here or watch the videos here. Viewers can then reach out over Twitter or upload their own video response. I’m more of a writer so you’ll catch me blogging rather than vlogging. 🙂
Episode 6: How does market research maintain trust and authority in modern times where fake news and misinformation are perceived to be rife?
There are a few things we can do.
First, despite how expertise is being discredited more and more these days, let’s be more open and transparent about our credentials. More than simply degrees and experience, let’s talk about our membership in recognized industry associations such as Insights Association, MRIA, MRS, AMSRS, and Esomar, as well as ISO certifications. Let’s do more than simply mention we’re members, and instead start our conversations with that fact. Let’s describe what it means to be a member in good standing in terms of the code of standards and ethics we abide by. Let’s put those logos on the first page of our reports, and even include with them some of the ethics and standards statements that are most relevant to the specific project. Let’s use these as reminders for our clients that we always act in their best interest, and in the best interest of the research project, even if the results don’t work out the way we had hoped.
Second, let’s be more transparent with clients. Let’s tell clients about all of the strengths and weaknesses of our research processes, about the things that changed unexpectedly along the way, even if it means disappointing them. When we can’t achieve the response rate, sample size, or cost per complete that they require, let’s tell them right from the beginning and be clear about why it can’t be done. When the results we generate are completely unexpected and don’t line up with our hypotheses or norms, let’s be open and honest about what might have happened and whether there might be a problem. Let’s worry less about not winning a job, and more about demonstrating our commitment to the integrity of results. The secondary bonus of this transparency is that we can educate less experienced buyers on how research can be positively and negatively impacted by a variety of known and unknown variables so that they will be more informed buyers in the future.
Third, let’s be better public advocates. When we see our research in the media, let’s ensure the results, conclusions, and recommendations are clearly properly represented. And when they aren’t, let’s get in touch with the media to help them understand what the issue is, including telling them why margin of error or making a certain generalization isn’t appropriate. And if they refuse to correct the misinterpretation, let’s make a public statement to right the wrong, perhaps with a note on your website sharing details about how the information should be properly interpreted. And along the way, if we learn that certain media channels regularly misinterpret results, let’s reconsider working with those channels and even the clients that work with those channels. Every one of us has a part to play in helping to ensure our research results are properly portrayed.
Annie Pettit, Canadian Chair of ISO TC225
Debrah Harding, UK Chair
Elissa Molloy, Australian Chair
In the seven years since the creation of the quality standard ISO 26362, the use of online panels for market, opinion and social research has experienced massive growth and evolution. The standard was extremely useful in helping both clients and vendors explain and understand the technical aspects of what is now a ‘traditional’ online panel. And while online panels are now default sample sources for many researchers, new options that must also be considered have been developed since then.
In the online world, we have seen the introduction of panels that use not ‘traditional’ email invitations but rather options such as pop-up intercepts, or requiring people to visit a specific website and select from available research opportunities, or offering opportunities from pre-roll webpages. We now have to consider whether automated inventory and survey routing is appropriate for our needs. And of course, we now have the option to engage panel and sample brokers who will find sample providers for us.
The great success of online sample led to the decline of offline sample in rich areas of the world. But don’t let that fool you. There still exist large communities of people around the world where access to online services, or financial resources, means that advanced online surveys are simply not feasible. Offline panels are still very necessary and important in many communities and for many types of research.
And, what may seem surprising to some is that, now, in both offline and online environments, we must consider whether the sample or panel has probability or nonprobability characteristics.
In the time that our sector has greatly advanced researchers’ capabilities, people have also advanced in their responses to surveys. For some, answering surveys is now a normal activity for people, many of whom participate in one or more panels, in addition to innumerable surveys from ad hoc outreach programs and end-client research studies. Participants are more familiar than ever with techniques for increasing their chances of qualifying for incentives as well as techniques for completing surveys as quickly as possible, sometimes with less than good intentions and sometimes as a reaction to poor quality research tools and services.
It is clear that we have reached a new stage with samples where both offline and online sample have been accepted as valid and reliable techniques, each with a host of new intricate technical requirements.
On March 11 and 12, representatives from around the world, including Canada, UK, USA, The Netherlands, Australia, Japan, Austria, and more, will gather in London, England. There, we will discuss and debate the advancements our industry has made and how we can incorporate those advancements into the ISO standards. Our goal will be to update the online panel standard to better reflect the current and future state of sampling for market, opinion and social research. Also high on the agenda will be the new draft ISO standard for digital analytics and web analyses, which aims to develop the service requirements for digital research services. These leaders will also bring to light the global differences in research requirements and practice, to help solidify the wider issue of how the ISO research standards can best serve the research sector well into the future.
Quality: It’s What’s for Breakfast”
Kristine Francois, Director, Quality Planning, Market Strategies International; Cathy Kneidl, Training & Audit Advisor, CASRO Institute for Research Quality (CIRQ); Alexander Shashkin, Chief Executive Officer, Online Market Intelligence (OMI)
- Quality is the cost of entry for market research
- ISO is a quality management tool
- 20252 began in 2006, revised in 2012
- 26362 added in 2009
- Require annual 3rd party audits to maintain certification, both recognized around the globe as quality control standards
- Require overall quality system, about 70% overlap between the two standards
- 20252 required by many associations as a criteria for membership
- Panel: Standards relate to recruitment, panel structure and size, management, panel usage, client reporting, rules of conduct
- ISO creates standards of process beyong standards of practice and ethics offered by the industry associations themselves
- Very simply, ISO requires that you say what you do, close any ISO gaps, do what you say, consistently demonstrate you do what you say
- To begin certification – get the standards and appoint a quality manager or team which is a requirement, document your research process and outline a plan to close any gaps
- Allow system to run for about 3 months, internally audit and ensure compliance and traceability
- Complete self-assessment and then submit to CIRQ
- Two companies now speak about their reasons for becoming ISO
- Gave them a clear frame work for what they already wanted to build, already had the mindset to do it. Gave them a competitive edge. Clients wanted them to speak at their companies about quality, not necessarily a job.
- The templates are a great way to get your thought process in order, the workshop was several days and it helped them get ahead, get done more quickly, makes it actually seem possible. You can do it without the workshop but it will take a lot longer.
- The CEO and COO were drivers and believed in it which ensured success. Quality is everyone’s responsibility.
- Use translators in countries where English is not the language of business. May take more time but it still works well.
- The CIRQ auditors understand the research processes which makes things go faster. Re-certification time is cut in half, just 2.5 or 3 days to do. They only highlight different areas on the re-certification.
- Challenge was improving things as much as they felt was needed, and then wanting to improve them every year. Minimum requirement never seemed enough but the timing never seemed right to make things better, but they had to do it anyways.
- In the beginning, felt like people were doing it because they had to. Now it feels like top of mind and important to everyone. They’ve always been quality focused but now it seems much more important.
- It’s easier to show quality to their clients now.
- It’s hard to measure quality when there are a lot of other changes happening in a business.
- Clients trust the rigor that comes from ISO certification
- It’s a good onboarding, training tool
- ISO can be difficult for clients because the vendor will ask for certain things, sign-offs, approvals, by email and clients may not feel like doing it
- ISO doesn’t inhibit innovation and creativity, ISO is table steaks. It’s how you implement things consistently. It’s documentation.
- Has seen no requirements that are bad for business, help you improve your organization.
[tweetmeme source=”lovestats” only_single=false]I really hate standards.
I am far too independent and far too concerned about doing the right thing to like it when someone tries to tell me what to do. When it comes to carrying out market research, I’m irked when I’m told to follow standards and annoyed when I’m required to meet specifications. So if standards aren’t for me, then who are they for?
Market research standards are for
• people who believe in the letter of the law, not the spirit of the law
• new researchers to learn about basic and broad important features
• research buyers who don’t yet understand the nuances of every market research method and don’t always know if they’re buying fluff or substance
• small research companies that need backing to grow in the marketplace
• big research companies that might gradually ride the slippery slope
• every researcher and research company to learn from and ponder about
I guess that means research standards are for me.
Read these too
Day 2 brought lots of interesting topics as well. We started off with updates on the ESOMAR 26 document, the ARF quality initiative, and the ISO process. I did not know that the first ESOMAR code was developed in 1948! I did not know that people from 26 countries were involved in the ISO process. It was not a surprise, however, to hear the findings of the ARF quality study .
- data quality is not the serious issue that was previously thought
- panel duplication is really only about 16%
- heavy responders provide good data
- response rates are not the best indicators of quality data.
After the ARF presentation indicating a complete lack of ability to identify a weighting system that would equalize panels, one unlucky speaker had the unfortunate time slot of presenting his work on how the CART system can indeed weight data sources to be equal. Personally, I’m still on the side that panels cannot be weighted to be equal. I’ve participated in so many parallel studies for so many different types of surveys and categories of products and have yet to seen a perfectly successful case. Sure, you can always weight a few variables into comparability, but you lose out on a bunch more. Maybe CART is the magic solution. Time will tell.
Once again, survey panels took the hit on not being representative of the general population. Really, come on folks, what method of marketing research IS representative of the general population. None. End of story. In the end, the only kind of representativity that matters is making sure your sample suits the purpose. And, if your results never predict the marketplace, get out of the business.
Kim Dedeker was kind enough to share a few thoughts on the state of the industry since she first caused a good storm. Her advice – the industry can only be sustained through quality, we need to continue having and creating leadership through getting involved.
The last major topic of the day was mobile research. I must confess that I am STILL not a believer. I just don’t see how the ability to answer a 20 minute survey on a 2 inch screen as you ride the bus or pay for your new purchase is going to bring the survey industry back from the deadly response rate dive.
Feel free to also read a few other thoughts I had about ESOMAR Chicago in a piece I wrote for research-live.
Below is Kim speaking and a presentation about social media that mentioned the tweet-up at Lux Bar.