The when and why of DIY #MRX


334 - Mr. Fixit's

At the CASRO online research conference, one of the panels focused on DIY research and included a couple panel providers and a couple DIY companies. This panel was of particular interest to me because I’ve watched how DIY has taken quite a pounding in the last few years but for the wrong reasons.

I asked the panel about when qualified researchers should use Do It Yourself research and the answers included when you need results better, faster, and cheaper. Ok, a very generic and unhelpful response.

Then, another audience member asked when should DIY research NOT be used. Sadly, the panel could not offer a single idea about when DIY research was not appropriate. Given that one of my mantras is “Every research method has pros and cons,” this was a completely disatisfactory answer. And misleading.

So here is my opinion on when DIY should and should NOT be used.

DO use DIY research when:

  • A qualified researcher has written the survey and designed the methodology
  • The survey is very simple, short, and contains no complicated skip patterns 
  • You need results extremely quickly
  • You have the population of target responders and ‘random’ sampling is not necessarily required
  • You understand statistics well enough to know when sample sizes are too small, when to use a t-test or a chi-sqare, and why MOE is under hot debate
  • You need to test a simple methodological issue prior to launching the full study (e.g., will the distribution of responses be better served with a 5 point or 3 point scale)

Do NOT use DIY research when:

  • You don’t need anyone to proof read your survey because you never make mistakes
  • An expert in survey design has not created the survey
  • An expert in sampling/weighting has not developed and implemented the sampling plan
  • An expert in data analysis will not be analyzing the results
  • (so assuming that a qualified researcher is managing all aspects of the research…)
  • You are running a complicated  tracker – weekly/monthly, complex balancing, multi-country
  • Your survey incorporates multiple and varied skip and logic patterns
  • You require complicated census/target balancing and weighting

It’s a pretty easy answer.

Plumbers plumb.

Engineers engineer.

Researchers research.

8 responses

  1. Great post Annie,
    One key aspect of the DIY discussion that often gets overlooked is the lack of transparency / understanding as to where respondents are being sourced. Even if the interface might be “fast and cheap”, if the data is biased because respondents are drawn from non-representative panels of survey takers, and incorrect business decisions are made, it can actually create negative value.

    1. It’s always important to know where your sample comes from. Given that there is no such thing as a representative sample, it’s impossible to draw good conclusions without know what your sample is.

  2. I think all of the points above are useful and true. But think about the problem of research if you are a small non-profit or business. That is, one with a budget of at most a few thousand dollars that might be spent on research in a year and who has few connections with our industry. How many market research firms will help them in any form? How many firms market to them so the person has a chance to learn about how to get advice?
    It seems to me that until we as an industry figure out ways to help these folk (and make money doing it of course) that for us to complain about DIY isn’t totally fair to these firms. And to be fair to us, it isn’t easy to figure out how to do this if we spend most of our lives dealing with larger firms.

    1. Definitely see your point. There are a lot of businesses out there that simply can’t afford the offerings of a (small or large) full service research company. I do hope however, that businesses that can’t afford them do as much as is possible to ensure their research is as high quality as possible. I bet there are some great researchers out there who would volunteer a bit of their time to help build good questions and give advice on methodology.

      Interestingly, this month’s issue of Vue magazine (Canada’s market research magazine) is all about non-profit/charitible organizations and their use of market research.

  3. My one experience with a DIY tool was a nightmare. I analyzed a dataset downloaded from XXXX. I’m not really sure why the data came formatted as it was or who in their right mind would design a dataset to look as it did. I think that the intention at XXXX is for researchers to use their tool to conduct their analyses.

    I can see how a DIY tool would be helpful at the front end, in order to develop a professional and visually consistent survey, but I would agree that unless you have some idea of the implications of your methodology and layout, what kind of data will come back to you, what kind of analyses might be possible, and how those analyses could be done using the tool you plan to use, you’re risking a big waste of time for very little or no analytic gain.

    The DIY tools are controversial in the research community because they lead untrained researchers to believe that they can truly DIY. They suggest that research is- instead of a nuanced professional skill- something that anybody could do, given the tools.

    People learn quickly that surveys conducted by those who aren’t knowledgeable about surveys are generally a waste of time, yielding results that are inadequate and difficult (or impossible) to work with. But in the meantime, they have wasted respondents’ time and frustrated some people who may, in the future, because of their negative experience, decide not to participate in surveys that have been painstakingly developed by professional researchers.

    It’s an important conversation to have, but also a difficult one, because it often makes professional researchers feel offended or defensive from the get-go. And many of us are often asked to recommend DIY tools to inexperienced researchers.

    1. The commenter named a specific DIY tool but I have X’d it out. The entire comment remains valid without naming names.

  4. I agree with the notion that every research methodology has pros and cons. I am not sure I agree with the last statement that only plumbers plumb, if it is easy enough I plumb too like you indicated in your DOs (…the survey is very simple….). I also want to introduce a new angle on DIY research: the web listening DIY which does not involve questionnaire design or sampling, just picking a few terms and a few sources. Another example of DIY research that can work is online user experience testing.

    1. Yup, EVERY research method is prone to poor results when conducted by inexperienced or unskilled people, whether surveys or listening or communities.

%d bloggers like this: