Tag Archives: focus group

Radical Market Research Idea #7: Participate in an untrusted methodology #MRX

If you get right down to it, I’m a quant. My history is with surveys and quantitative social media research. I have little experience with focus groups or neuroscience or eye tracking or many other respected and mistrusted methodologies. I can criticize the heck out of any of them but then, it really wouldn’t be fair.

But market researchers love to criticize. That’s what the test control design is set up to do. Prove and disprove based on logic and facts. So we criticize methodologies we aren’t familiar with even when we don’t have the facts. Are you one of the people who’s lambasted focus groups for their lack of generalizability? Have you laughed neursocience studies off the stage for their hocus pocus?

If you’ve never participated in a focus group, commission one now. Participate in the sampling, help write the discussion guide, help lead a group, help write up the results. See for yourself the good and bad that can come from it. Compare the results with those that come from the good and bad of the method you’re most familiar with. Learn something new. Try something new for once. Radical?


Big Data? Big Deal. #MRX

Recently, a new beast has begun to terrorize the market research industry. Big Data. Say it with a booming echoey voice and it sounds even scarier. For a while, I thought I was missing something. Big Data must be a specific term for something newly discovered, a new book, a new piece of software. But alas no. It simply means what it sounds like. Really big data sets.

Because I’m in the market research industry, I’ve worked with online survey panel datasets of hundreds of millions of records, transactional datasets with hundreds of millions of records, and social media datasets with hundreds of millions of records. If we allow for individual data points, then I’ve worked with datasets containing billions of data points.

So why are folks getting all excited big data? Have they never worked with big datasets before? Are they just now realizing that big datasets exist? Believe me, big datasets are no big deal to market researchers. This is what we do everyday. We already get it. Just ask us.

Will Communities Kill the 6 Group Project by Nick Priestley #SoMeMR #li #mrx

15.30 PANEL: Will communities kill the 6 group project?

  • The future for research communities in generating insight
  • How online communities fit into the overall market research mix
  • Evaluating cost efficiencies and quality trade-offs

Nick Priestley, Managing Director, Tuned In

  • Are communities mainstream now? We are approaching mass market in terms of awareness. People still have concerns about blurring boundaries but there are more and more success stories. Barrier is internal stake holders.
  • Is it research? should we call it research? It’s more about collaboration and getting input in at the beginning. Forward thinking clients are open to it. Does it replace traditional focus group?
  • is 6 group project under threat? It will never die. It is still relevant. But there are many benefits to online approach. Start with hypotheses, develop over time. You can do this in two hour. Vast geography, you can’t do focus groups in hard to reach areas.
  • What kind of people work best in communities? 1% are super users, 9% are active, 90% don’t do anything. We want more people to be active. Now, people are more familiar with the idea. We need to show the process more to interest more people.
  • Segment communities so you can identify the creative people and the strict people and use each group to their strengths. Find tasks that are creative as possible to encourage people. People often don’t realize they were capable of expressing themselves in that way.
  • Audience – it’s important to listen to people who know nothing and these people aren’ t in communities.
  • Online doesn’t necessarily mean cheaper. You get more value from community. More people, less geography issues, less time of day issues, more natural setting, technology gives you more creative options.
  • Is it the shiny object effect? Do groups need a fresh spark? There is absolutely room for both. [Ah yes, the dichotomy must always be there, qual vs quant, online vs offline, MY favourite must always win. Can we just all use one big tool box with all the shiny tools in it?]
  • Are communities a safe haven with fewer ethics issues? Are communities fake social media? Communities are no different from people in real life.
  • Audience – Will facebook put an end to parties? Same as will communities put an end to groups. The two must work together. [ha, yeah. no more parties for me.]
  • Communities when used properly will replace parts of panel and parts of focus groups. [Because it is a better tool for various objectives. Use the RIGHT tool from your toolbox.]
  • Audience – Are focus groups more boring than they need to be? Can we incorporate fun of communities back into focus group.
  • Audience – What about finding pre-existing communities, that aren’t created by MR. This area is littered with huge fails because of lack of transparency, researchers failing to identify themselves. Perhaps use those areas for recruitment. [remember patientslikeme]
  • Audience – 6 groups aren’t under threat, the creative brainstorming groups are under threat. Communities let you do it over a longer period of time.
  • Audience – Are communities at risk because anyone can do them, even if you aren’t trained? Well, the client can read the entire transcript and confirm it all.

In Search of Horribly Low Response Rates #MRX

Ask anyone what the response rate to their last research project was and they’ll hold their head in shame if the answer is a number under 10%. As researchers, we work really hard to generate response rates that are as high as we can possibly get them. In the competitive world of market research, the survey panel or focus group recruiter with the highest response rate just might win the job.

But wait. Why do some sources have higher response rates than others?

  1. Active rules: Sources that only invite people to research if they have completed a research study in the last month have much higher response rates.
  2. Incentives: Sources that provide more valuable incentives have higher response rates.
  3. Recruitment: Sources that recruit participants from research sources have higher response rates (e.g., “Thank you for answering our Purchase Satisfaction Survey. Would you like to join our panel?”

skinner box

In each of these three situations, the research panels have essentially pre-selected people based on their propensity to participate in research. And, as we all know, the propensity to participate in research is not a randomly distributed characteristic. Certain personality types are just more or less likely to want to participate in research. And this brings me to my point.

Shouldn’t we actually be seeking out the lowest response rate possible?  Instead of focusing on gathering opinions from people who are MOST likely to want incentives or who always participate in research, shouldn’t we keep the pipe lines open to accept opinions from research keeners as well as those who hardly ever want to participate in research and who couldn’t care less about incentives? Wouldn’t a really low response rate reflect a research participant pool that is awash with both keeners and frequent abstainers, a pool that is more reflective of the real population?

Perhaps we should actually be seeking out low response rates. Perhaps we shouldn’t judge sample providers simply on response rates.  Perhaps we should consider that the quality of a research sample goes far beyond response rates. What a strange thought.

What is market research? #MRX

Market Research

Image by Tom Hirt via Flickr

Ray Poynter recently asked what is market research so I figured why not share my thoughts as well.

Let’s start by understanding what market research is not. It is not surveys nor focus groups nor social media research. It is not sensory testing nor neuroscience. Market research is not a set of tools nor a style of work nor a laboratory setting. It is not qualitative or quantitative.

Market research, rather, is a system of learning and discovering information about the market place. It takes advantage of any and all, old and new tools and can be conducted under the umbrella of various styles of work. And, I’ll add to that a caveat that MR is a scientific method of learning about the market place.

Tools, like surveys and focus groups, can be used and abused by anyone but when they are applied in a scientific way, such as with precise sampling and questioning techniques, they help researchers properly understand the market. Whether the chosen tools are qual, quant, or quali-quant is of little consequence. It is the scientific process for turning people-watching and polls into valid and reliable measurements.

Your turn! What is market research?

Malinoff: New Face of MR #MRIA

Welcome to the virtual MRIA 2011 annual conference! This post reflects my personal musings and interpretations of this presentation. It was written during the presentation and posted minutes afterward. Any inaccuracies and silliness are my own.

Trends and challenges – the new Face of Marketing Research and Intelligence
With Don Mills (Corporate Research Associates), Gary Bennewies (Ipsos), Tom Anderson (Anderson Analytics), and Jean Marc Leger (Leger Marketing0 Moderated by Bernie Malinoff (element54).

  • Bernie is apparently the most interesting man alive. 🙂 (Janine Keogh from Kraft said so.)
  • MRIA assumes no legal responsibility for the content of this session. 🙂
  • Please tweet and blog the session so folks who couldn’t attend can join in. 🙂
  • (FYI panelists knew the general topics but did not know the questions.)

Client needs: How have client needs changed and what do we need to do better or different moving forward?

  • Jean Marc: No client, no problem. Clients are difficult. Some clients know nothing about research. Some clients know what they want and don’t know how to get it. Others THINK they know what they want. No matter what, no time, no budget. We need to be smarter, strategic conclusions.
  • Don: Global brands have different needs than national or regional. Faster, better, cheaper is desired. Demands on supplier is burdensome. Clients want guidance dealing with new world of research. It looks sexy but they don’t know how to benefit.
  • Tom: Clients want faster, do it yourself. More reason for control. Don’t fight the trend. LOOK for different kinds of clients. MR is more than just focus groups and surveys. Leverage big data, text mining.
  • Gary: BQC, better, quicker, cheaper. New demand coming. We must understand client’s business so we can provide real insight and direction. Not just practitioners but business solvers.

Technology: How does a traditional company avoid being market research roadkill?

  • Gary: Be involved in testing, experimenting, partnering in new things. Explore new ways of researching. Try biometrics. Maybe it won’t work but play with it. Take some risk. Past investments will become obsolete. 80% of marketing challenges end up in a study. In ten years, it will be fishing in the river instead. Harvest what already exists.
  • Jean Marc: Only change is stable. Online is now traditional. 70’s was what. 80’s was why. 90’s was how. 2000’s is what if. Now the question is when. Do all that is short period of time. Listen to clients and they will decide what they will need. We still do the old but we do it differently. Data collection is less and less important. It’s HOW you analyze. Line between research and marketing? Clients requesting different services. Mid-size companies will find this difficult.
  • Tom: MR is not an industry you want to get into now. MR needs to pick a direction. Join DIY trend? Build your knowledge into software. Become more of a consultant? Become a text mining service? Pick one.
  • Don: You must move or you will be left behind. be in digital data collection. Old world vs new world. New world is not in the room. There is a social research industry that is not in this room. Brand new entrants. Different language – sentiment, content analysis. (That’s me! In this room!) If Kraft hasn’t figured out social media, where are we? How do we get value from SM? (Ask me! Hello?) Do political and media polls enhance or detract market research industry credibility?
  • Gary: Use polls to boost profile. One of few ways to get a direct measurement of what you do, how close were you? Used all forms of research, not just surveys.
  • Don: Industry shot itself in the foot in last election. Half of companies were wrong. Why? Never happened this much before. Nobody called for the result that we got. IVR was wrong, online did well, traditional telephone was a mix. Too many polls. Regional data was so far off and it complicated the problem. Don’t release small sample size data or you will be burned. It’s not OK when people get it wrong. Every company that gets it wrong affects the industry. It hurt our industry.
  • Jean: Online was accurate last time. It’s our annual exam. You must demonstrate your accuracy. We aren’t trying to predict seats but the pollsters were good. Quebec was not easy, voting intention moved by 15 points. That’s crazy anywhere else but normal here. They did social media tracking as well. Polls are in the papers way too much. 350 polls published two years ago.
  • Tom: Help politicians get instant feedback with social media, become part of the process.

What is the future role for individual (e.g., CMRP) and Corporate (e.g., ISO) certification?

  • Don: Great supporter of certification. Most people who wrote and pass exam. Support it because we want professionalization of industry. It is a career. It is for their people not for clients. Supports ISO but is it affordable. If you are concerned about standards, DO something about it. make it look like a career choice, not an accidental job – professionalize it.
  • Tom: Following a recipe of MR, under the guise of quality, does not behoove most of us. Are trade organizations relevant. It stifles creativity. No value in certification. Hire based on skills. If they had a methods class, they are at the top anyways. Never look for CMRP title. Experience matters. Do MR feel inferior to Doctors? I don’t need letters after my name.
  • Gary: Is their value. Does it promote industry? Help clients select supplier? Standards are rooted around traditional research, maybe not so good. It stifles creativity. It can’t be a limiting factor. Certification must be current. Probability samples are out of date and yet it’s in the certification.
  • Jean Marc: Concerned about ease, length of certification. Manager your system better. Invest in people. Where are all the other companies that aren’t at this conference? Young people are better than older researchers. They are way more efficient and they leave the office at 4:30 and then they might work at 1am. They are the future. (Let’s sing. 🙂

How are we doing as an industry to fuse these skill sets at an individual level? Business, methods,tech.

  • Tom: If industry is bigger than surveys and it’s actually about knowledge, get employees and education. Get clients do a bit more and help them look smart. Pull in all types of data. Reverse mentoring – we should talk to younger people, mentor other industries. Read tech publications not just MR publications.
  • Gary: Don’t want them all in one person. Have experts, different people. Skill sets change over time though. People behind the scenes need to be better at tech. Senior people are not road-bumps or barriers to clients accessing new tech.
  • Don: Teach the employees your techniques. Hard to find people with good strategic thinking skills. Busy savvy comes with experience. Development environment is important. Autonomy is important. Let people grow at own speed. Young folk are aggressive about getting ahead and they will stay if you let them.
  • Jean Marc: Challenge is how to build a team, generate innovation. People have weakness but the team has all the skills. Clients talk more problems, less methods now.


Why certify if everyone disagrees and it’s out of date?

  • Don: ESOMAR is releasing new standards in SMR. Certification needs to keep pace. Be patient. (Young padewan)
  • Tom: Burke has great courses. Skills useful beyond our industry. That’s the certification you need.

Canada’s role in the MR world?

  • Gary: Canada is a resource, a talent, sabbaticals, moves. We have a lot of talent we can export.
  • Jean Mar: We need strong competitors in Canada. Don’t have a lot of Canadian clients doing foreign business.Let Canadian companies buy, not be acquired.

Should we go back to random sampling?

  • Jean Marc: Problem is voters don’t know who they are going to vote for. (That’s what we’re here for!)
  • Gary: Is random sampling possible? Not everyone has a land-line. In perfect world, random sampling would be great but it can’t happen.

Skillen: Respondents Headed for Extinction #MRIA

Welcome to the virtual MRIA 2011 annual conference! This post reflects my personal musings and interpretations of this presentation. It was written during the presentation and posted minutes afterward. Any inaccuracies and silliness are my own.

“The survey respondent – headed for extinction?”
With Janine Keogh (Kraft Canada), Kristin Luck (Decipher), Norman Baillie-David (TNS Canadian Facts), and Rasheeda Qureshi (Research Now). Moderated by Shane Skillen (Hotspex).

  • Panelists are water in a bathtub but they are going down the drain. How do we plug the drain? How do we not dirty the water?
  • 0.25% of survey responders do 32% of all surveys, 5% are doing 50% of all surveys. (Sigh, this again.)
  • Is this tiny group engaged? Is it quality data? All that matters is quality. (Bang on)
  • How do we stop people from disengaging? It’s not the number of respondents, it’s the quality. Decrease length and perhaps increase frequency. Don’t cram everything into one survey.
  • Length is not the only issue. Respondents generally want to assist but we are cramming respondents into our format. We never give them feedback. We never tell them how the info is used. They have no  idea how important their information is?
  • An actual responder is here –> How does she educate her friends about participating? She’s told to tell her friends to join. (I think that’s insufficient) What do her friends get out of it? She gets to think about something else other than work or chores.
  • We need to adapt incentives to respondent desires and it depends on the project.
  • Norman thinks we need to find more responders. (Nope, I think we need to do a better job on surveys.)
  • Some surveys are darn right painful. Who is responsible for this? Clients? Providers? Shared? (Seems to me providers aren’t given the chance to influence.)
  • “We’ve been doing this survey for 20 years, we can’t change it.” (I don’t buy this.)
  • If you turn down a bad survey, someone else will do it anyways. (No reason that you can’t put your foot down. Take a stand for quality.)
  • Ten years ago, telephone was it. Online was sexy. Now RR declined and we created that problem, it’s our fault. We need to reinvest in the raw material.
  • We can focus on getting more people in but we really need to focus on the experience or they’ll just leave.
  • Consumers want to talk and engage and when they enjoy the experience, they do it again. Why not try to be more social, make it two way. Think about focus groups and research communities. Take what’s great from them and integrate them into survey research.
  • Technology has done the industry a disservice. There is less direct connection now. Phone, face to face, respondents are no longer real people. They are respondent id. Technology needs to be about improving the experience.
  • Younger respondents are hugely important and their RR is terrible. The only reason we do research is to grow our business
  • Here is what we should all do right NOW.
  • Researchers should be in the schools, speak to classrooms, about research. You can do this NOW!
  • Researchers should share results from the surveys. (Yeah baby!)
  • Answer your own surveys – people don’t because they know it’s long and tedious. Then why ask your respondent to do it? Boring is boring no matter how much the incentive is. You still want to blow your brains out. 🙂 Take a video of the torture session. Take the torturous survey yourself.
  • Incentivize for good survey design (Great idea)
  • Uptake of innovative survey tools has been miserable because of worry about survey trends. Please please please consider it for panelists who are hating grids. Panels are at stake. They are free to use so do so.
  • What about survey experience surveys? That 4 question survey at the end. Why not share it with the client? Show them how their surveys compare.
  • Thank you screens are pathetic. Tell people why their participation matters.
  • Don’t legislate survey length. Create a great survey and make it the right length.

Scott Cho: Confessions of a Number Cruncher #MRIA

Welcome to the virtual MRIA 2011 annual conference! This post reflects my personal musings and interpretations of this presentation. It was written during the presentation and posted minutes afterward. Any inaccuracies and silliness are my own.

Confessions of a number cruncher…

Scott Cho, Vice-President – Consumer and Technology Research Practice Leger Marketing

  • Started the session with the smartphone focus group spoof, how toddlers, ginos, seniors, and hot babes use a smartphone. “I want a button on my phone that will give me a shower.” Love it!
  • We often create extremely complicated instead of just asking the question. Do we really need all those crosstabs and piles of paper on our desk? 3 statistical sins: People who make things complicated, People who run things without knowing why, People who run things so it looks like they’re busy.
  • Scott transitioned from a few groups a year to one year with 100+ focus groups, 100+ indepth interview, and 68 flights. His training was 30 minutes talking to a colleague and living near his boss. 🙂
  • One client thought the focus group attendees were too dumb, they were not his target group. Hello reality!
  • We need to show attendees the materials but often they are too sensitive to share. We can prevent screen caps but people take photos of the screen. This is why some clients can’t do anything online.
  • What is a real environment for store testing? Are security guards and metal detectors part of the real store experience? Makes no sense. If everyone knows it’s a test, does it have to be realistic?
  • We test future products and compare them to products available right now. Makes no sense. Competitors won’t share their concept products so it’s an unfair comparison.
  • Some things are too complicated to talk about, particularly for innovative devices: Mobile internet device – -> computer tablet. “augmented reality product”  Point a camera on a book and extra information is displayed on a computer screen. It’s a device intended for schools. Do you have a sense of the product from that description? Maybe not, so how can you respond to hypothetical research about it.
  • Experience with a product is more important than what the product actually does. Every phone can make
  • a phone call. It’s not about the phone call anymore. it’s the experience of making a call.
  • He hates the overhead camera. Do we really want to stare at his head in the video? (No, thank you)
  • It’s frustrating to watch other people moderate. You really want to bud in. 🙂
  • Moderators get the best chair in the room every single time. But… this puts the moderator on a different level. The moderator should be presented in the same way, a relatable way. (Good call, agree.)
  • Moderators can put on a persona and be whoever they need to be in a group. Be a young person. Be a dad. Be a dumb single guy. Be a rockin’ cool guy. Play the role that suits the occasion. But you must maintain the sincerity.
  • Energy as a moderator is contagious. Don’t talk dull and boring (like he’s doing this minute). Use personality, some edge, and you’ll get that in return. You want a focus group where people don’t want to leave at the end.
  • Why do we write discussion guides in powerpoint not word? Word forces you to think that it’s a script.
  • “Note passing” isn’t good. You’re messing around, interfering with the moderators connection to the responders. It disolves relationships. But, moderator does need some category knowledge. This is why you need moderators who have content area specialties.
  • Culture Code by Clotair Rapaille. Wealthiest researcher alive.:) He does 3 hour focus groups. Hour 1 he talks about anything. He says he doesn’t believe what people say. He wants to give people a chanc e to feel smart. Hour 2, they make up kiddie stories. Hour 3, they sleep.
  • What is consumer insight? It’s not just cool words like knowledge and discovery. Scott says it’s…. calculated discovery, about the aha moment and turning it into insight through knowledge of all the other surroundings.
  • It’s hard to tell what is right or wrong in qual research.
  • He no longer sees people as segments and numbers. They are people.
  • People are still surprising. Loser trackpants man is actually a brilliant person.
  • Watching insights in front of your eyes is incredible.
  • You can actually ask someone “Why won’t you buy that product?” (Put the fear into them. HA HA)

Ask a Simple Question, Get an Encyclopedia

Broadwater Focus Group

Image by Nebraska Library Commission via Flickr

If you want to be an excellent market researcher, you need to know a lot about many different topics. You need to know what makes a good survey question or focus group discussion guide and how to avoid writing a horrid one. You need to know about research methods, sampling, weighting, and sample size determination. Knowledge of statistics is essential and it must go beyond t-tests, chi-squares, and p-values. There is a ton of very detailed, complicated information you must know to do your job well.

But here is the problem. When people ask for research advice, they don’t always want an essay on the pros and cons of various options and techniques.  They know they’re asking a complicated question with a complicated answer but sometimes they just want a quick and simple answer. They want to know that they’re pointing in the right direction, that they’re generally thinking the right thing.

So what do we do? We don’t try to understand whether it’s a request for a simple answer or an in-depth consultation. No matter what they’re looking for, we give people a three hour lecture about the intricacies of research and make everything far more complicated than it needs to be. Our strange technical languages serves to scare off some people and bore others to tears.

Isn’t it time we considered what people really want? Perhaps just a simple answer to a simple question?

This is why your research sucks #MRX

You might not want to admit it, but at one time or another, you were probably on a team responsible for some research that sucked. Wonder why? Let me help you out.

1) You didn’t have a trained, experienced researcher at the helm.
Researchers are not a luxury component of research projects. Researchers know what makes a quality, unbiased, nonleading, useful questionnaire and focus group. They know what the most appropriate sample sizes are and WHY those are the most appropriate sample sizes. They know which statistics are the right ones and WHY those are the right ones. Researchers know how to take a problem and funnel it into a measurable, valid, and generalizable project.

2) You failed to identify and follow through on specific objectives.
There are two places where it is essential to focus on your objectives. First, when designing your research, you need to have a problem to solve or a reason to do the research. Without a problem, you could write a 400 question survey and still be trying to add more. Second, you need to focus once you get your data. Most surveys result in 300 page data-tables which are completely overwhelming, even for great researchers. Without focus and silo-vision, you will never find an answer. You can search but you will not find.

3) You focused on price and speed rather than quality and quantity.
Sure, you can choose the research with the best price and speed. But validity and reliability depend on sample sizes, and data cleaning, and appropriate statistical testing, and quality research design. These things are not quick nor cheap but when you need to accurately predict future sales or which TV show will be canceled or which product test will succeed, this is how you must do it.

4) You don’t follow through on the results.
Lots of really great research actually does get done, in large and small companies, via surveys and focus groups and social media research. But research is just crap if you don’t follow through on the results. If you KNOW you aren’t going to follow through on a set of findings, don’t bloat your survey and fatigue your respondents with it. If you KNOW you won’t have the time to follow through on the results for six months, DON’T do the research for six months. Research in a drawer is money in the toilet. Or, you could just give that money directly to me. Paypal accepted.

%d bloggers like this: