It’s true that for the most part, leading questions are the sign of a poorly skilled, inexperienced survey writer. When it’s pointed out, most of us can see that these are terrible questions.
- Do you agree that sick babies deserve free healthcare?
- Should poorly constructed laws be struck down?
- Is it important to fund new products that improve the lives of people?
- Should products that cause rashes be pulled from stores?
- Should stores always have enough cashiers so that no one has to wait in a long line?
But are leading questions always bad? I think not. However, these are situations that only experienced researchers should attempt. Leading questions may be appropriate when you are trying to measure socially undesirable, embarrassing, unethical, inappropriate, or illegal activities. Consider these examples.
Would you say yes to this:
- Have you driven drunk in the past three months?
What about to this?
- Many people realize that they have driven after having too much to drink. Is this something you have done in the last three months?
Would you say no this?
- Have you donated to charity in the past three months?
What about to this?
- Sometimes it’s hard to donate to charity even when you really want to. Have you donated to charity in the past three months?
In both cases, it is possible that the first question will cause people to give a more socially appropriate answer, but not necessarily the valid answer. In both cases, the second question might create a mindset where the responder feels better about sharing a socially undesirable answer.
The next time you need to write a survey, consider whether you need to write a leading question. Consider your wording carefully!
Demand that your conferences be Diversity Approved! (Tweet this post!)
When Canada’s new Prime Minister, Justin Trudeau, was asked why his cabinet was 50% male and 50% female, his answer was simple. Because it’s 2015. Such a simple answer to a long standing problem.
As I look back over 2015, I see that “because it’s 2015” didn’t apply to every market research conference. Some conferences had speaker lists that were 70% male. Some conferences had speaker panels that were 100% male. No conferences had attendee lists nor industry lists that were 100% male let alone 70% male.
There are many reasons that men might be over-represented as speakers, but few that are acceptable.
- Random chance. As a lover of statistics, I accept that random chance will create some all male panels. But since I’ve never seen an all female panel, random chance is not what’s at play here. If you’d rather see the math, Greg Martin calculated the chance of having all male speakers here. It’s not good.
- 70% or more of submissions were from men. That also is an acceptable reason. If women aren’t submitting, then they can’t be selected. So on that note, it’s up to you ladies to make sure you submit at every chance you get. And don’t tell me you’re not good enough to speak. I ranted on that excuse already.
- You haven’t heard of any women working in this area. This excuse is unacceptable. You can’t look for speakers only inside your own comfortable friend list. Get out of your box. Get online. There are tons of women talking about every conceivable industry issue. Find one woman and ask her for recommendations. You can start here: Data science, Marketing research, Statistics, Tech.
- The best proposals happen to be from men. This excuse is also unacceptable. It demonstrates that you believe men are better than women. You need to broaden your perception of what ‘better’ means. Men and women speak in different ways so you need to listen in different ways. It’s good for you. Try it.
- Women decline when we ask them to speak. It’s a real shame particularly if women decline invitations more often than men. But any time a woman declines, ask her for a list of people she recommends. And then consider the women on that list. No women in the list? Then specifically ask her if she knows any women.
- It’s a paid talk and they only sent men. Know what? It’s okay to remind companies that their panel isn’t representative of the industry. You can suggest that they send a broader range of people.
- We didn’t realize this was a problem. Inexcusable. Diversity has been an issue for years. People have been pointing this out to market research conferences for years. The right time to fix things is always now.
When was the last time you prepared a sampling matrix balanced on age, gender, and ethnicity and then were pleased when it was 70% female, 70% age 50+, and 90% white? Never, that’s when. You stayed in field and implemented appropriate sampling techniques until your demographics were representative. This is absolutely no different.
So, to every conference organizer out there, ESOMAR, CASRO, MRA, MRIA, ARF, MRS, AMSRS, ESRA, AAPOR, I challenge you to review and correct your speaker list before announcing it.
- What percentage of submissions are from men versus women? Only when submissions are far from balanced is it acceptable for the acceptance list to be unbalanced.
- Are there any all male panels? Are there any all female panels? (By the way, all female panels talking about female issues do NOT count.)
- Are more than 55% of speakers male? Are more than 55% of speakers female?
- Is the invited speaker list well balanced? There is zero reason for invited speakers to NOT be representative.
- Did you actively ask companies to assist with ensuring that speakers were diverse?
If you can give appropriate answer to those questions, I invite you to publicly advertise your conference as Diversity Approved.
Will you accept this challenge for every conference you run in 2016? Will you:
- Post the gender ratio of submissions
- Post the gender ratio of acceptances
- Proudly advertise that your conference is “Diversity Approved”
Demand that your conferences be Diversity Approved! (Tweet this demand!)
I recently debated big data with a worthy opponent in Marc Alley at the Corporate Research Conference. He stood firm in his belief that big data is the best type of data whereas I stood firm in my position that traditional research is the only way to go. You can read a summary of the debate written by Jeffrey Henning here.
The interesting thing is that, outside of the debate, Marc and I seemed to agree on most points. Neither of us think that big data is the be all and end all. Neither of us think that market research answers every problem. But both of us were determined to present our side as if it was the only side.
In reality, the best type of data is ALL data. If you can access survey data and big data, you will be better off and have an improved understanding of thoughts, opinions, emotions, attitudes AND validated actions. If you can also access eye tracking data or focus group data or behavioural data, you will be far better off and have data that can speak to reliability or validity. Each data type will present you with a different view and a different perspective on reality. You might even see what looks like completely different results.
Different is not wrong. It’s not misleading. It’s not frustrating. Different results are enlightening, and they are indeed valid. Why do people do different than what they say? Why do people present contradictory data? That’s what is so fascinating about people. There is no one reality. People are complex and have many contradictory motivations. No single dataset can describe the reality of people.
There is no debate about whether big data has anything to offer. Though Marc and I did our best to bring you to our dark side, we must remember that every dataset, regardless of the source, has fascinating insights ready for you to discover. Grab as much data as you can.
I’ve got a bad habit. Whenever I visit a new city or country, I like to overdose on one particular item. Many times it’s chocolate bars, sometimes it’s tea biscuits (thank you Dublin!), sometimes it’s macaroons (thank you Nice!), and sometimes it’s donuts.
I visited the Donut Vault the last time I was in Chicago so I searched for donut shops again. I got about 300 Dunkin Donuts as well as a bunch of speciality donut shops. And how nice to see that four of those shops were pretty much on my doorstep. In other words, behold a completely biased view of four shops.
Stan’s: Stan’s is an unassuming regular shop in a regular office building and I didn’t have to wait in line. Some of the donuts seemed quite large and kids will be sure to like the fun colours. I got a blueberry fritter and a lemon pistachio. The fritter was giant, dripping in icing, and quite possibly weighed a pound. It did have a nice blueberry flavour but I can’t imagine how anyone can finish such a huge donut. The lemon donut was normal sized and the kiddy coloured icing tasted very sugary. Stan’s works as a specialty a donut shop but with all the other choices, it’s not my first choice. I’ll just give it a C – average for a specialty donut shop.
Glazed and Infused: Turns out that this shop has a ginormous eating area and is closer to a restaurant than a donut shop. As for character, despite its size it has lots. I waited about ten minutes to place my order. You can buy a $20 cake sized donut, the world’s hottest donut with a jalapeno on it, or you can buy an apple cider donut and a pumpkin spice donut like I did. The apple cider donut was a yeast donut, really sweet and covered in sugar. It tasted more like apple than apple cider although I don’t know how you’d make it taste like cider. The pumpkin donut was enormous and covered with icing and pumpkin seeds. This donut tasted more like pumpkin than pumpkin spice – a bit of cinnamon and nutmeg would have helped. Glazed gets a B from me.
Firecakes: This shop was teeny tiny and thus had a bit of character. I had to wait about five minutes to order. I got an old fashioned donut and a pineapple bacon donut. The old fashioned was big and heavy, just how I like them. Yum. The pineapple bacon was a big disappoinment because I dropped the pineapple on the ground. WHY MEEEEEE! Anyways, both of these donuts were big wins. I also got a teeny tiny boston cream donut which was fairly ordinary but very cute. This shop gets an A from me although they need to find a way to glue the pineapple on more tightly!
Donut Vault: Lastly, this shop is indeed a bank vault and so fits the character requirment perfectly. I had to wait 30 minutes to place my order here and when I left, the line was even longer. I got a pumpkin cream cheese donut and a gingerbread stack. The pumpkin donut was delish, lightly sweet, crispy outside. The gingerbread stack seemed to be the runt of the stacks as they were quite thin. But, they were tastey and crispy. I’ll give the vault a B+.
By now you’ve heard about the three Vs of big data. Whether your concern is millions of research panel records, billions of transactional records, or trillions of web tracking records, we all have the same problem. The volume of data increases exponentially, the variety of data keeps increasing, and the speed, well, let’s think lightspeed. These issues alone make big data a worthy opponent.
Big data is also rife with missing data. It’s incomplete data, and it’s complicated data. It needs specialized analytical tools and specialized analysts. But those problems are also not the reason we’re failing.
Why are we failing at big data? Well, let’s take a step back to the survey and focus group world that market researchers love so much. When I think back to the last survey I wrote, it too was quite the beast. For just twelve minutes of respondent time, I spent many hours dreaming of, writing, tweaking, rewriting, and retweaking every single question and answer. I pondered every the, or, if, they, you, and why. I argued with myself about the possible ramifications that every single word might have on my results. In every case, I settled on the best solution, not the right solution. In the end, I had a survey that would carefully address every single hypothesis and research objective on my list. This survey was a beauty and the analysis was quick and easy.
Let’s move forward to our big data project. You know, the one where someone dumped a giant SQL database with thousands of variables and billions of records on your plate and said, “Make our program better.” You weren’t really sure what the program was, you didn’t know what was currently good or bad about it, and none of the database variables matched up with any project plans or research objectives. Actually, there were no research objectives. Except for “make better.” I can assure that is NOT a solid research objective.
Imagine if someone collected together a hundred surveys from a hundred projects and told you to “make better.” I can guarantee you would fail at that survey analysis regardless of how many years of survey analysis you had behind you.
The simple reason we continue to fail at big data is that we fail to create concrete and specific research plans and objectives as we do for every other research project. We know very well that a survey project will fail without carefully operationalized objectives but when we work with big data, we ignore this essential step. We don’t plan ahead with specific variables, we don’t list out potential hypotheses, we don’t have a game plan. “Find something cool” isn’t a game plan. Nor is “how can we improve?” Big data needs big brains to plan and organize and be specific.
Do you want to succeed at big data? Then stop treating it like a magical panacea and do the work. Do the hard work.
Live blogged at #CRC2015 in St. Louis. Any errors or bad jokes in the notes are my own.
Leveraging methodologies and optimizing your product: How NRG developed a connected home solution
- help consumers better understand how to use and control energy when we don’t really think about it unless it doesn’t work
- connect/smart homes can help you control anything from an app on your smart phone, lights on and off, check on kids, see what time people come home
- people will always want a $10 000 ferrari. stop asking about that. no one wants a $200 000 kia either. stop asking that.
- what is the most efficient product based on costs and preferences
- Learn about the efficient fontier – optimizing preferences and costs
- what features do people really like and which are undervalued
- must narrow down the features using max diff first
- people really wanted to be able to confirm that they had closed and locked the door
- price obviously had to be included
- there is no right answer but you get data to make decisions [totally agree. statistics never give you th right answer. they give you something to ponder]
Political polling 2016: what pollsters and corporate researchers can learn from each other
- “This election finally prove that most market research is probably twaddle”
- research is used to find the idioms that people like and these words show up in speeches
- RDD has been the most prefered data collection mode, and in some cases still are
- compare a phone and online survey side by side, n=500 for both, 21 questions around 5 minutes
- housekeeping variables generally matched
- no major differences between the two groups for many variables on voting, immigration, economics
- study that appends voter data from registered voters
- questions were about favorability and support, e.g., i don’t really like them but i’m voting for them
- in single select questions, trump is favoured.
- tending data is more important than single point estimates
- brand liking is not always a good predictor of buying behaviora
- change in question wording can yield substantially different data
- online data can provide a reliable supplement, if not replacement, for phone surveys [in other words, there is NO perfect data collction method. be SMART in your data collection and interpretation]
Tiptoeing through innovation quicksand: methods to die for and methods that might kill you
- Gartner hype cycle for methods
- 100 responses so far, convenience sample – researchers, supply side, corporate; skewed quant
- asked about 34 techniques, no forced answers
- 3 people said they were fired for using a technique – virtual store research, prediction marketets, A/B testing, emotion detection [i wouldn’t want to work there anyways!]
- if you did an online survey in 1996, it might have had a big impact on your career
- people said they were rewarded, promoted, raise for using a new technique; helped companies shift directions
- only 64% had used online surveys, half had used focus groups
- never use again -mail surveys, facial recogniton
- Career builders, Career opportunities, Career investments, Career challenges
- Microsegmentation – can identify micromarketing action to take, but complex to implement
- Customer journey mapping – great holistic view of customer experience, too complex or externally focused
- Uplift modeling – trying to action individual, strong ROI,
- social media analytics – unprompted items of concern, difficult to decipher, not-reliable data
- mobile intercept surveys – on the spot real data, hard to get participation
- neuromarketing – higher price, opinions vs actions
- microsurveys under ten questions – quick,better for niche audience, low barrier of entry
- facial recognition – might feel it doesn’t give new information
How better meetings and wicked problem solving propel research-based innovation by Tom Wujec, Autodesk #CRC2015 #MRX
Live blogged at #CRC2015 in St. Louis. Any errors or bad jokes are my own.
- technology will rise at an expontential rate, industries are rising and falling chaotically, human creativity generally remains the same
- we’ve built more transitors than grown grains of rice this year [what the?]
- in technology, each step is greater than the sum of the previous steps
- London has a number of trains operating undersground for …. mail! Technology lets you look at it at the level of a bolt.
- Can measure traffic patterns by creating subway systems and highway systems using a digital model
- VUCA – volatile, uncertain, chaotic, ambiguous – this is the state of the world as described to congress
- businesses are becoming VUCA
- do you want to buy the fastest VCR? no, you want DVD and netflix and newer.
- kodak HAD more patents for digital than all their competitors but they didn’t want to disrupt their category
- uber is the current post child
- there are three kinds of basketball according to the shoes needed, shoes are digitally developed to suit these specific needs, monitors are built into the shoe and you can subscribe to that service
- two robots are assembling a bridge in amsterdam
- robots can do things people can’t, can try many variations in the thousands and evaluate all for strengh and weight, can create an algorithm that a designer can use to select the attribute they would like, and all this in an afternoon [ah yes, tales of my statistics professor who did ONE factor analysis for his differtation]
- computers create solutions that look like nature, the best solutions are often designs from nature [think helicopters and planes that we designed thinking about birds]
- imagine creating a dress for an individual via 3d printing, no waste, exact fit
- technology is being adopted massively faster as the years go by, TVs took many years, iphone took no time at all
- many sports stories in newspapers are written by computers
- Watson has better predictive ability than 12 physicians
- what happens when this technology goes ito a toy? child can ask a toy any question though some questions are answered with ‘go ask mommy’
- impossible, impractical, possible, expected, required – the phases of techology
- need to change our mindsets to work with these technologyies
- showing the technology and the research is not enough
- creativity is endless and magical, we can explore more broudly and deeply than every before
- fostering innovation involves identifying problems that matter by exploring alternative and delivering elegant solutions
- in two minutes, draw a picture of how to do something trivial, inconsequential, such as how to make darkened crispy toast, no words, for someone who has never made taste before – where do you start, where to you end, what are the salient points. in different countries you get toasters or frying pans or fireplaces
- use sticky notes on the wall, move them around, do it quickly
- technology is increasing and industry is increasing but humans havent’ had a hardware upgrade for hundreds of thousands of years so creativity remains static
- where should you put your efforts?
Live blogged from Esomar in Dublin. Any errors or bad jokes are my own.
When democracy fails to deliver by Ijaz Shafi Gilani and Jean-Marc Leger
- what explains satisfaction and dissatisfaction with democracy
- democracy is the worst form of government except for all the others – Winston Churchill
- Failed as a norm? no
- Failed in specific cases? yes
- 75% of people believe democracy is the best
- 50% believe they are ruled by the will of the people
- 35% of upper income americans believe a good way to govern is to have the army rule
- Nat rep, 52 countries, n=50 000, 10 years apart survey
- countries who’ve practiced democracy the longest are most disillusioned
- correlates of disatisfaction include:
- macroeconomic factors – ecnomy, inequality, size of country
- demographic factors – gender, age, education
- identify factor – nationalism, patriotism, attitudes towards globalization
- Identify factors seemed to be most relevant for countries practicing democracy the longest
- political rights and civil liberties have taken a back seat, now its become flight of jobs and immigration
- linked to inability of govt to copy with “encroachment of globalization”, these people are most dissatisfied
- does democacy fail to deliver in a globalized world?
- democracy might need to reinvent itself
Ireland and same sex marriage by Eric Meerkamper and Aengus Carroll
- Bill Gates says he is struck by how important measurement is to the human condition
- we have a unique skillset and tools to measure
- we have relied too heavily on the same repsondent for too long – Dan Foreman
- Random Domain InterceptTechnology, based on making errors in the browser bar
- 51 countries, 51 000 respondents
- should same sex marriage be legal
- seems like a safe question but in many parts of the world, this is a death penalty for you and even your family, people need anonymity to answer this question
- across 8 other countries with marriage quality, only about 50% of population wanted it, so it is still risky
- about three quarters of of people disagree with marriage eqality in countries where sexual orientation can be a crime [naturally, you’ll be killed if you say otherwise!]
- yes campaign: what kind of country do you want to grow up in, it’s about human rights, inclusion
- no campaign wanted a civil partnership not marriage, that kids needs a mom and a dad
- 72% of young voters wanted same sex marriage which matched the campaign they used, focus on young people
- young people brought older people to come and vote
- marriage was not the issue, the issue was discrimination and exlusion
- this method allows safe measurement
Leveraging qualitative for indiginous innovations: flavour innovations by Irene Joshy
- How do i adapt the flavours of the local palette? can i copy paste? is the flavour appealing and authentic? how do i position the brand or variant? need to deconstruct and reconstruct a product
- India has two major brands in the category – lays which is global and kurkure
- pepsi wanted a flavour map of indea, map the flavours and create flavour groups that work across india as well as strong regional flavours
- identify the semiotics, embedded and emergent codes of the flavours in the context of snacking
- wanted a shortlist to test out
- india has 32 regions, 125 dishes, 75 snacks – how do we decontruct this qualitatively
- every dish has a role – staples, accompaniement
- started by mapping flavours
- started with recipes and ingredients, created and mapped clusters – cook books and online receipts, chefs, home cooks, looked for ‘lost in time’ recipes, used snowballing to find grandmothers known in their areas as great cooks and created recipes from their cooking
- got a list of ingredients and links of strength among every ingredient, created clusters of flavours
- client didn’t know what to do with the results [seriously? you need someone to tell you? sigh]
- clusters allowed them to figure out what went with wheat or lentil or potato or rice
- they could choose a base and then the flavour cluster that worked with it and then experiment by adding something fom a different cluster
- created three test products
- look at visual , olfactory, mouth feel, throat feel, overall impression
- gave consumers metaphors to choose from because they don’t have the words needed to describe their feelings
- first prorotype – flavor and emotion, tactile and emotion, colour and emotion
- is it a type of food that it playful, sensual, rebellious, celebratry, subtle, comfort
- the study was viewed as a map for the next five years
- [very interesting talk, i’d recommend finding the paper]
- huge battle for market share in the been category [really? i’ve not seen a single root beer since i got here!]
- ireland is 4.5 million, dublin is 1.2 million, is it really four main cities or just one city
- city dwellers have more income
- city is freedom and opportunity, each city has its own nuance
- tested several different heineken brands – Tiger, Sol, Desperados, heineken
- Cork, dublin, galways, belfast were tested
- had to avoid the stereotype, had to ask about culture without talking about culture, had to let personal experiences emerge naturally, had to have practical use when the research was done
- mediography – inventory of social engagement, bricks and mortor, entertainment
- talked to trend creators – influences, experts, food, fashion, music, art, opportunities for thir party involvement
- cultural brailing – essays on throughs and feeling on culture ingredients, required to take a broad perspective not just going to get a beer
- digital ethnography – looked at people in action, in interactions, in real time through out the city
- creative consumer workshops – went through all the content they collected, and asked people create ideas for brands and events, marketing could watch this happen
- Dublin – cosmopolitcal, diversity, opportunity
- Galway – laid back, wildness, embracing
- belfast – freedom, optimism, fragility
- cork – pride, traditional, banter
- Truth 1 – dublin is humble about its place in the world, loves to see itself as connected and a contemporty of other cities of interest. led to a music plaform – brought the cities of the world to dublin. “heineken sound atlas” Brooklyn an dtokyo have been featured
- Truth 2- belfast is a freedom and where some places were once closed off, jailhouse and courthouse underground connection was of huge interest but unavailable. They created an event in this area. Drove word of mouth.
- Truth 3 – want to be familiar in dublin but also show off new discoveries. “Open your dublin” which meant to go discover your city. Dine in the dark was dinner in a crypt of a cathedral they thought they already knew.
- Truth 4 – feel dublin is creative but it needs support to really see that. you can sponsor an event as long as you respect the location. Sponsored the Tiger Fringe Festival with daring creatives.
- brands grew by 50% or more
- moved from mass marketing to localized decision making
Live blogged at Esomar in Dublin. Any errors or bad jokes are my own.
Driving success through big data segmentation
- Australia affected by the global financial crisis, growth rate almost stopped, increase in unemployment, reduced household wealth
- GE consumers were trying to pay off their debt balances, also high losses
- needed to identify the new norm, how did consumers feel after this crisis, can we build a segmentation
- success factors were driving cultural transformation, fine tune value proposition
- needed to engage senior stake holders, involve agency partners at every step
- did an audit of GE’s customer data
- two samples, 3000 people around australia, 5000 people from GE database, link the two together
- what is the new norm?
- australians felt destabalized, the signs weren’t good, felt vulnerable, pessimistic, uncertain about the future
- they wanted control over whatever they could, they could control their spending, new norm was to live within your means
- needed a unique GE segmentation that lived on the GE database
- don’t sit back once the research is underway
- used videos, handbooks, training packs, reference desk stands
- everyone knows the target segments and understands them
- they saw actual change as a result – different people could talk to each other and understand each other
- net promoter score went up 5 points [cue all the NPS isn’t enough arguments]
- saw improved sales measures, increases in sales, this led to them training the entire company on segmentation
- essential to have senior management endorsement
- essential to have committment to change, high quality research agencies, education, communication, make outcomes tangible by putting numbers on the outcomes
Reinventing convenience store food
- [huge fan of the 7/11 slurpee! woot!]
- 620 stores in australia, all franchised, 1.5 billion annually, 6 customers per second
- how to become a destination for food on the go
- it wasn’t in decline, but they did make it grow
- did a knowledge audit with many stakeholders
- did a qual phase with real people, took them on a bus to various stores
- also did a quant phase with 1000 people
- 3 key things: shoppers trust quality and freshness, customers prefer no service, customers see it as fast food
- hot food still rated high on trust but not as high as chips and candy
- but people didn’t think the store was fun or attractive, looked bland, no sense of discovery
- nailed efficiency but not the experience [yup, if I wanted to talk to restaurant staff, I’d go to hungry jack]
- people want to be left alone in the store – anonymity, no judgement, freedom, naughty fun, gives them control and flexibility with flavours and additives
- on a survey, they ranked with burger king, subway, kfc, hungry jack as a fast food store – they both offer real meals – i.e., a meal is a chocolate bar, chips and a pop [LOL yeah, i get it :) ]
- each meal has different marketing needs, needed to build craving for lesser known products
- growing in appeal and affinity in this category finally
- [This is my choice as the winner, hope you win!]
Using survey data to target customers and increase ROI through digital media
- conversion model to identify consumers willing to spend more on your brand
- survey of 10 000 travelers of 7 brands of holiday inn brands, about 2200 open to staying at holiday inn in the future
- [quite the sales pitch here. you can ALWAYS discuss a product without making it a sales pitch :( ]
- scaled 2200 travels into 15 million travels with a look alike model of internet behaviour
- four outside companies played a role in creating ads, tagging and measuring the campaign, purchasing the audience
- 500% increase in bookings
- [didn’t get to see a model :( ]