Along with a group of market researchers from around the world, I was asked to participate in Voxpopme Perspectives – an initiative wherein insights industry experts share ideas about a variety of topics via video. You can read more about it here or watch the videos here. Viewers can then reach out over Twitter or upload their own video response. I’m more of a writer so you’ll catch me blogging rather than vlogging. 🙂
Episode 6: How does market research maintain trust and authority in modern times where fake news and misinformation are perceived to be rife?
There are a few things we can do.
First, despite how expertise is being discredited more and more these days, let’s be more open and transparent about our credentials. More than simply degrees and experience, let’s talk about our membership in recognized industry associations such as Insights Association, MRIA, MRS, AMSRS, and Esomar, as well as ISO certifications. Let’s do more than simply mention we’re members, and instead start our conversations with that fact. Let’s describe what it means to be a member in good standing in terms of the code of standards and ethics we abide by. Let’s put those logos on the first page of our reports, and even include with them some of the ethics and standards statements that are most relevant to the specific project. Let’s use these as reminders for our clients that we always act in their best interest, and in the best interest of the research project, even if the results don’t work out the way we had hoped.
Second, let’s be more transparent with clients. Let’s tell clients about all of the strengths and weaknesses of our research processes, about the things that changed unexpectedly along the way, even if it means disappointing them. When we can’t achieve the response rate, sample size, or cost per complete that they require, let’s tell them right from the beginning and be clear about why it can’t be done. When the results we generate are completely unexpected and don’t line up with our hypotheses or norms, let’s be open and honest about what might have happened and whether there might be a problem. Let’s worry less about not winning a job, and more about demonstrating our commitment to the integrity of results. The secondary bonus of this transparency is that we can educate less experienced buyers on how research can be positively and negatively impacted by a variety of known and unknown variables so that they will be more informed buyers in the future.
Third, let’s be better public advocates. When we see our research in the media, let’s ensure the results, conclusions, and recommendations are clearly properly represented. And when they aren’t, let’s get in touch with the media to help them understand what the issue is, including telling them why margin of error or making a certain generalization isn’t appropriate. And if they refuse to correct the misinterpretation, let’s make a public statement to right the wrong, perhaps with a note on your website sharing details about how the information should be properly interpreted. And along the way, if we learn that certain media channels regularly misinterpret results, let’s reconsider working with those channels and even the clients that work with those channels. Every one of us has a part to play in helping to ensure our research results are properly portrayed.
Annie Pettit, Canadian Chair of ISO TC225
Debrah Harding, UK Chair
Elissa Molloy, Australian Chair
In the seven years since the creation of the quality standard ISO 26362, the use of online panels for market, opinion and social research has experienced massive growth and evolution. The standard was extremely useful in helping both clients and vendors explain and understand the technical aspects of what is now a ‘traditional’ online panel. And while online panels are now default sample sources for many researchers, new options that must also be considered have been developed since then.
In the online world, we have seen the introduction of panels that use not ‘traditional’ email invitations but rather options such as pop-up intercepts, or requiring people to visit a specific website and select from available research opportunities, or offering opportunities from pre-roll webpages. We now have to consider whether automated inventory and survey routing is appropriate for our needs. And of course, we now have the option to engage panel and sample brokers who will find sample providers for us.
The great success of online sample led to the decline of offline sample in rich areas of the world. But don’t let that fool you. There still exist large communities of people around the world where access to online services, or financial resources, means that advanced online surveys are simply not feasible. Offline panels are still very necessary and important in many communities and for many types of research.
And, what may seem surprising to some is that, now, in both offline and online environments, we must consider whether the sample or panel has probability or nonprobability characteristics.
In the time that our sector has greatly advanced researchers’ capabilities, people have also advanced in their responses to surveys. For some, answering surveys is now a normal activity for people, many of whom participate in one or more panels, in addition to innumerable surveys from ad hoc outreach programs and end-client research studies. Participants are more familiar than ever with techniques for increasing their chances of qualifying for incentives as well as techniques for completing surveys as quickly as possible, sometimes with less than good intentions and sometimes as a reaction to poor quality research tools and services.
It is clear that we have reached a new stage with samples where both offline and online sample have been accepted as valid and reliable techniques, each with a host of new intricate technical requirements.
On March 11 and 12, representatives from around the world, including Canada, UK, USA, The Netherlands, Australia, Japan, Austria, and more, will gather in London, England. There, we will discuss and debate the advancements our industry has made and how we can incorporate those advancements into the ISO standards. Our goal will be to update the online panel standard to better reflect the current and future state of sampling for market, opinion and social research. Also high on the agenda will be the new draft ISO standard for digital analytics and web analyses, which aims to develop the service requirements for digital research services. These leaders will also bring to light the global differences in research requirements and practice, to help solidify the wider issue of how the ISO research standards can best serve the research sector well into the future.
Live blogged in Nashville. Any errors or bad jokes are my own. Any typos are purely the fault of the iPad.
by Peter Milla and Dave Christiansen
CASRO has seen an increase in requests from clients and regulators for data privacy and security compliance
– code of standards
– safe harbor program
COmpliance means confirming to a rule, like a policy or law. CLients want operational transparency.
COmpanies will require 50 percent less business process workers and 500 percent more digital business jobs. especially regulatory analysts and risk professionals. These jobs are generally only in larger companies. This includes privacy officers.
Privacy and security are symbiotic. This can be a crisis for MR. Privacy is appropriate use of the data. security is the confidentiality and integrity of data.
– you cant just destroy data. what about all the backups. the saved copies that everyone has from their piece of the work.
– availability of data could impact life or death in some cases
What drives compliance
– client wants it [i hope vendors want it too. why is because clients want it?]
– legislation or regulation like HIPPA GLB COPAA FTC PIPIDA. you could be accused of unfair trade practice for discontinuing a poor responder.
– gain a competitive advantage
[wow, typing on an iPad keyboard is quiet and completely unobtrusive when you lay it flat! But i cant put pictures or links easily. Sorry.]
ISO 27002 – you cant be certified, you can be compliant
HIPAA compliance case study
– business associates now face liability. Uses not in accordance with BAA. failure to limit PHI. failure to provide breach notification. failure to provide HHS access when required. failure to comply with security rule.
– many companies state one year but they keep it forever
– Protected Health Information PHI.
– employees don’t usually intend to make errors, they just don’t know
– no easy checklist of requirements
– does offer a set of principles. instruction is to take necessary steps to disclose minimum necessary information
– much is process based
HIPAA security rule compliance
– risk analysis – evaluate likelihood of risks, implement appropriate security measures, document those measures, maintaining continuous review and assessment, ensure access control and integrity control, ensure transmission security, keep documentation up to date
BLUE CROSS – just had a breach that affected 80 million US citizens, 25% of the population. names, SIN, birthdays. be sure to use your free annual credit report. Take advantage of free credit monitoring. monitor your children as well. be alert when filing your income taxes.
Top security trends
– cybercrime, privacy and regulation, third party provider threats and breaches, BYOx in the workplace – Bring Your Own Device [like i’m doing right now. are my office security systems on my personal tablet?]
[note to self and everyone. turn the GPS off all of your devices. it is not necessary that every software program knows where you are, where you live, where you work, where your kids live]
Advanced Persistent Threat – APT
– china and Russia and Iran have active cyber espionage, aligned in every industry to take whatever they can, causing information security bar to be raised
CLients expect all their information is safe. need a dedicated person or team. CISSP, CISM, CISA, ISO, SDLC. [we have this person. they went to every single office in every country over the last couple weeks to remind every single person just how serious security issues are.]
[everyone should have come to this session. i don’t care if you think you’re doing fine. you need persistent reminders of just how worried you really ought to be.]
Information security is not IT security. spans people processes and technology. its digital written and spoken. it’s being proactive. it’s an organizational discipline.
– best practice for information security, NIST, CSF, COBIT. can be audited and certified. Earth’s ‘best practice’ its the policies procedures and controls and training.
– it is not industry specific. it is federal, state, industry, contractual relevant.
– vulnerability assessment annually or quarterly, penetration testing, gap assessment, awareness training, internal audit, risk assessment.
[Annie’s free public service announcement – do an internal audit today. if it looks like spam, it probably is. if it doesn’t look like what I usually email to you, i probably didn’t email it to you.]
With nearly 50 speakers taking the floor to discuss mobile research, every single attendee at today’s Market Research in the Mobile World conference was sure to leave with at least one new idea or a rethink of an old idea.
Mobile research has been around for at least 15 years even though only the last few years have actually presented us with the capabilities that researchers have been dreaming of for the last 40 years. Mobile research used to mean slapping a telephone or online survey onto a mobile phone regardless of whether the survey suited the phone. Long grids, long questions, long surveys and more made users and responders dread the mobile survey. Now, thanks to massive advances in smartphones, mobile research means not just the stand-by text surveys, but also photos of in-store shopping, videos of trying on jeans that don’t fit well, gps tracking as people drive convoluted routes from store to store to store to avoid left turns and tricky roads. And of course, don’t forget about research games that people can play on their phones while they wait for the bus, games that might never be played if they were sitting at the office. This is the new mobile. Well, not new, but this is the now doable mobile research.
Given my background on standards committees for CASRO, ESOMAR, MRA, and MRIA, I was extremely eager to listen to the panel on mobile research standards. What better opportunity than a conference whose entire focus is mobile research. Sadly, I left disappointed. After 45 minutes into the 60 minute session, I realized that the panel hadn’t really discussed standards other than to say that standards stifle innovation. Ouch. Even when the question was directly posed to the panel “Does this mean you can’t and won’t create mobile research standards,” the answer we got was no. While one panel member was quick to say that standards are important and used (Thank you @KristinLuck), that attitude just didn’t permeate across the entire panel.
I wholeheartedly believe that there must be mobile research standards. We must have standards for people whose attitude is “If it’s not illegal, it’s fair game.” We must have standards for people who are new to the game and don’t truly understand the strengths and weaknesses of the methodology. We must have standards for users to fairly compare competitors.
So I offer this challenge to my mobile research colleagues. Open up your research filing cabinet. Pull out your research on research results. Use the lessons you learned in Kindergarten and share those results with your mobile colleagues. Develop mobile research standards. And keep on innovating the way we know you will.
- Google surveys, and oh, some other people too #MRMW #MRX (lovestats.wordpress.com)
- Mobile Perspectives: Africa, Privacy, ROI, and Beyond #MRMW #MRX (lovestats.wordpress.com)
- What Market Research in the Mobile World means to me #MRX #MRMW (lovestats.wordpress.com)
- Emotions and Client Priorities #MRMW #MRX (lovestats.wordpress.com)
- Mixing up your survey data for better results #MRMW #MRX (lovestats.wordpress.com)
- Business Value Through Smarter Research #MRMW #MRX (lovestats.wordpress.com)
Welcome to this series of live blogs from the Market Research in the Mobile World Conference in Cincinnati. With so many sessions, I’m only blogging about a few sessions each day. All posts appear within minutes after the speaker has finished. Any errors, omissions, or silly side comments are my own. I’ll also be providing end of day summary blog posts for Esomar so keep your eyes peeled for those as well.
Establishing A Global Benchmark for Mobile Research
Tom Van Aman, Nick Nyhan, Ande Lees, Seamus McAteer, Guy Rolfe, Kristin Luck, Jeff Sellinger
- Can we even establish benchmarks? Yes, there are already some best practices. Question types, survey design. Don’t go into mobile space unless you’re willing to make sacrifices [hmm… i don’t think its about sacrifices. it’s about doing it properly.]
- Let’s get together to build a meter that apple will accept #CHALLENGERAISED
- Innovation often demands that you put aside standardization
- Standards are always relevent. Can you ignore 6 billion people on mobile while you’re considering standards? You can’t please everyone.
- Today’s experience will be very different from one and two years from now. How do you benchmark that?
- The ideal state is to let people do research that doesn’t feel like research
- “We are the world’s largest focus group” first AOL, then Yahoo, now Facebook. Who’s next…
- “For every insight you have, you double the number of confused people.”
- The old stuff is the foundation and puts the new stuff in context [i love the technical words]
- If you tie yourself to a benchmark, you can miss the unpredicted outcomes
- [sounds to me like benchmarks aren’t possible but i KNOW they are. we must benchmark quality and validity and minimum requirements]
- Mobile is the Glue Media. It is the only media you carry with you as you consume every other piece of media [oooh wait for the google glasses!]
- Biggest barrier is questionnaire design. We’re far away from this. Lots of reluctance to revise questionnaires.
- [I AM comfortable saying this is the way you must do it. That’s where the basic data quality rules come into play. e.g, NO 2 hour surveys. NO 50 item grids. I have no problem with that.]
- [Just a thought, if we write a survey intended to be answered on a laptop, then we ought not to let it be viewed on a small device]
- More data is not better data. “Something will pop”
- A Plea to the Canadian Government #MRX (lovestats.wordpress.com)
- That’s Me, The New Editor-In-Chief of MRIA’s Vue Magazine #MRX (lovestats.wordpress.com)
- The Top 10 Things We Love About Social Media Research #MRX (lovestats.wordpress.com)
- I hate social media research because: there are no demographics #1 #MRX (lovestats.wordpress.com)
Because Conversition has played an ongoing role in the social media space, Iwas asked by the MRIA to speak with them when they addressed the Standing Committee on Access to Information, Privacy and Ethics in Ottawa.
Brendan Wycks, the Executive Director of the MRIA, opened our talk with a discussion of MRIA’s views and desires and I followed with a discussion more focused on social media research. Below is my speech.
Address to the Standing Committee on Access to Information, Privacy and Ethics
House of Commons, Ottawa, June 5, 2012
Presented with Brendan Wycks, Executive Director of the MRIA
Thank you everyone for taking the time to meet with us.
As Brendan said, my name is Annie Pettit and I am the Vice President of Research Standards as well as the Chief Research Officer at Conversition, a Canadian market research start-up specializing in social media research. I am an avid social media research tweeter, blogger, and conference presenter, and have recently published a book about social media research which includes a chapter on social media research ethics. Because I am seen as a global thought leader in the social media research space, ESOMAR in Europe, and the Council of American Survey Research Organizations (CASRO) and the Marketing Research Association (MRA), MRIA’s counterparts in the United States, each invited me to be a contributing member of their social media research committees.
To give you a sense of the role that social media research is playing in the market research industry, I’d like to share with you a few results from the Spring 2012 Greenbook Research Industry Trends Report, a survey of over 800 market researchers around the world. Twenty-eight percent of those researchers had used social media research. Fifty-nine percent planned to use social media research in the next year. And more than 10% said that social media research is one of the greatest opportunities for researchers in the future.
Social media research is defined as the application of traditional market research principles to the collection and analysis of social media data for the purpose of better understanding policies and opinions. Just as survey researchers use survey data, social media researchers use social media data. And, we apply the same strict methodological practices to that data. For instance, just as with traditional survey research or focus group research, social media research begins by collecting the right data. Where survey researchers decide which people are best suited to participate in a survey, social media researchers decide which websites or other online forums are best suited for better understanding opinions. We incorporate traditional aspects of market research including sampling, weighting, scaling, norms, and box scores to ensure we measure opinions as accurately as possible.
The main purpose of social media research is to better understand the opinions people have towards policy issues, products and services, celebrities and politicians, social issues and cultural activities. Social media research helps us learn what people like and don’t like so that we can improve the services people receive, create better products, and better serve our constituents.
Most importantly, social media research is not a kinder, gentler word for social media marketing. We do not market products. We do not sell products. We, like our counterparts working in the traditional side of the industry, conduct market research. We abide by and respect the same methodological and ethical guidelines and standards as traditional researchers do.
I’d like to share with you just a few examples of how we abide by those principles.
First of all, we take great care to only collect public data. Some websites, like Facebook and Linkedin, use passwords to completely hide portions of data from outsiders, including Google. If you were to do a Google search, this data would not be found. Social media researchers do not and in fact, cannot collect this data. Many social media research users expect to see more data coming from Facebook but because much of it is programmed as private, very little is actually released. In some cases, we could just create a password and collect the data. But we don’t. We respect this privacy.
Other websites allow anyone to read the entries. Comments left on YouTube, Flickr, or WordPress are written for strangers to read and enjoy, and can be found via a Google search. The purpose of passwords in these cases is to allow readers to follow the conversation among many different people. This is the type of data that social media researchers collect.
In addition, we depersonalize data that is shared in reports, we do not engage with social media users without their consent, and we do not knowingly collect data from minors.
The internet has evolved rapidly in recent years. Ten years ago, it seemed incomprehensible that the average person would share intimate details of their life online. Today, bloggers are regular people who get excited when strangers, not their friends and family, read their thoughts and share them widely. Public forums are open social networks where strangers from around the world find and share opinions with each other. Twitter is a newer entrant into the social media space, and for many people using it, the ultimate goal is to write a tweet that millions of people around the world will read. We have reached a stage where social media has become so engrained in our lives that social media users expect companies to respond to social media comments written in an obscure corners of the internet. People expect their social media complaints to be met with letters of apology.
Right now, Canada is one of the global thought leaders in social media research and I’m proud to represent Canada in that role. But, I worry that if we lose this position, if we are unable to compete in the social media research space because our privacy standards restrict us rather than let us self-regulate, that our clients will have to use social media research conducted in countries with less than high ethical standards. That scares me.
Let us be thought leaders. Let us continue to lead in the social media research space. Let’s demonstrate to other countries that social media research can be conducted in a way that is beneficial to the government and corporate decision-makers who seek actionable insights from it; to research companies; and, most of all, to Canadians.
[tweetmeme source=”lovestats” only_single=false]I really hate standards.
I am far too independent and far too concerned about doing the right thing to like it when someone tries to tell me what to do. When it comes to carrying out market research, I’m irked when I’m told to follow standards and annoyed when I’m required to meet specifications. So if standards aren’t for me, then who are they for?
Market research standards are for
• people who believe in the letter of the law, not the spirit of the law
• new researchers to learn about basic and broad important features
• research buyers who don’t yet understand the nuances of every market research method and don’t always know if they’re buying fluff or substance
• small research companies that need backing to grow in the marketplace
• big research companies that might gradually ride the slippery slope
• every researcher and research company to learn from and ponder about
I guess that means research standards are for me.
Read these too