Tag Archives: panel

How the best research panel in the world accurately predicts every election result #polling #MRX 

Forget for a moment the debate about whether the MBTI is a valid and reliable personality measurement tool. (I did my Bachelors thesis on it, and I studied psychometric theory as part of my PhD in experimental psychology so I can debate forever too.) Let’s focus instead on the MBTI because tests similar to it can be answered online and you can find out your result in a few minutes. It kind of makes sense and people understand the idea of using it to understand themselves and how their reactions to our world. If you’re not so familiar with it, the MBTI divides people into groups based on four continuous personality characteristics: introversion/extroversion, sensing/intuition, thinking/feeling, judging/perception . (I’m an ISTJ for what it’s worth.)

Now, in the market and social research world, we also like to divide people into groups. We focus mainly on easy and quick to measure demographic characters like gender, age, and region though sometimes we also include household size, age of children, education, income, religion, and language. We do our best to collect samples of people who look like a census based on these demographic targets and oftentimes, our measurements are quite good.  Sometimes, we try to improve our measurements by incorporating a different set of variables like political affiliation, type of house, pets, charitable behaviours, and so forth. 

All of these variables get us closer to building samples that look like census but never all the way there. We’re so close and yet still missing the one thing that properly describes each human being. That, of course, is personality. And if you think about it, in many cases, we’re only using demographic characteristics because we don’t have personality data. It’s really hard to measure and target. We use age and gender and religion and the rest to help inform about personality characteristics. Hence why I bring up the MBTI. The perfect set of research sample targets. 

The MBTI may not be the right test, but there are many thoroughly tested and normed personality measurement scales that are easily available to registered, certified psychologists. They include tests like the 16PF, the Big 5, or the NEO, all of which measure constructs such as social desirability, authoritarianism, extraversion, reasoning, stability, dominance, or perfectionism. These tests take decades to create and are held in veritable locked boxes so as to maintain their integrity. They can take an hour or more for someone to complete and they cost a bundle to use. (Make it YOUR entire life’s work to build one test and see if you give it away for free.) Which means these tests will not and can not ever be used for the purpose I describe here. 

However, it is absolutely possible for a Psychologist or psychological researcher to build a new, proprietary personality scale which mirrors standardized tests in a shorter format, and performs the same function. The process is simple. Every person who joins the panel answers ten or twenty personality questions. When they return to answer a questionnaire, they get ten more personality questions, and so on, and so on, until every person on the panel has taken the entire test and been assigned to a personality group. We all know how profiling and reprofiling works and this is no different. And now we know which people are more or less susceptible to social desirability. And which people like authoritarianism. And which people are rule bound. Sound interesting given the US federal election? I thought so. 

So, which company does this? Which company targets people based on personality characteristics? Which company fills quotas based on personality? Actually, I don’t know. I’ve never heard of one that does. But the first panel company to successfully implement this method will be vastly ahead of every other sample provider. I could help you do it. It would be really fun. 🙂

Shopper insights for foresights #IIeX 

Live note taking at #IIeX in Atlanta. Any errors or bad jokes are my own.

I didn’t do anything wrong: The inventor’s dilemma by RIck West

  • In 1989, lots of people had nokia’s which were awesome phones at the time, bricks that never broke. In 2008, smartphones started to enter the market. Why did Nokia go from 55% share to 3%? They did nothing wrong so how did they lose?
  • We don’t want to be sitting here five years from now screaming that we’re relevant
  • But I invented this and we coined this phrase!
  • Today, no one swipes their credit card on a physical charge machine. We swipe in on a Square. No bank certified that charge marchine.  Inventors of the charge machine are now out of business.
  • Five years from now, you will not be doing the same business you’re doing now without major change

Completing the consumer journey with purchase analytics by Jared Schrieber and Bridget Gilbert

  • How do people purchase alcohol for attending an event
  • Data collected from a purchase panel, people take photos of every receipt they get from every purchase everywhere
  • Groups “The Socialite” and “The Rebel” – rebel spends 20% more
  • Trigger, ready to buy, and buy – three stages of the purchase
  • Journey for millenials is straightforward – invited to some event, think about the occasion, speak to someone, mental budget, added to list, talk to friends, check the fridge section, check for sales, compare prices, buy [darn it, I tried to avoid millenial talks!]
  • Millenials are always talking to someone at some point in the journey 
  • Key differentiator with rebels is they don’t speak to people, they have ghost influencers, more likely to say they bought someone else’s favorite type of alcohol, they are thinking about friends or family or whoever will be attending the event [or is this simply self justification of a larger purchase – “it’s not for me”]
  • Socialite – liquor store, express lane, after 5pm, shop in pairs, has a baby, lower income
  • Rebels – grocery store, stock up trip, before 5pm, shops alone, has a pet, higher income 

Brands and American mythology: Narrative identify, brand identity, and the construction of the American self by Jim White

  • We are all tellers of tales, give our lives meaning and coherence 
  • We don’t construct this identity in a vacuum, it’s within our culture, the mythology of our culture, we try to align our lives with the this we’re familiar with
  • We edit and reedit our identities
  • Brand strategists need to spend  more time listening to consumer stories
  • We rarely step back and listen to customers talk about themselves
  • Six languages of redemption – atonement, emancipation, upward mobility, recovery, enlightenment, development
  • We use brands to tell ourselves stories about who we are, to try and give ourselves some reality
  • Brands can be markers in our lives, can tap into that notion of our lives
  • Understand how personal myths draw from cultural myths
  • Ask people to tell stories about themselves not about your brand
  • Find the tensions they need to resolve, can my brand help smooth those contradictions, actualize th story they want to tell

Reimagining the traditional consumer panel by Bijal Shah

  • She’s a promotions company and they have millions of purchase records in their database, they are not a data company
  • Rely on panels but there is a sever lack of scale, not enough information about the entire population
  • We try multiple data sources but often can’t link sources
  • Partner with a DMP to make your data actionable like krux, lotame, Adobe
  • Find unique data source to enhance your data assets

When does #AAPOR infuriate me? #MRX 

Let me begin by saying I love AAPOR. I go to many conferences around the world and so can make some fair comparisons regarding the content and style of presentations.  While AAPOR presentations are not known for skill in the physical presentation, AAPOR is top notch for its focus on methods and science. There is no fluff here. Give me content over decoration any day.  I always recommend AAPOR conferences to my scientifically minded research friends.  That said…

aaporToday i heard inferences that the difference between probability panels and nonprobability panels is quality. Are you kidding me? Since when does recruitment method translate into poor quality. Different isn’t bad. It’s different. I know first hand just how much work goes into building a quality panel. It ain’t easy to find and continually interest people in your (my) boring and tedious surveys. Fit for purpose is the issue here. Don’t use a data source for point estimates when it’s not suited for point estimates.

And stop asking for response rates with nonprobability panels. High rates are not good and low rates are not bad. High response rates mean every person with a low response rate has been kicked off the panel. It does NOT mean you’re getting better representativity. Instead, ask about their data quality techniques. That’s what truly matters.

I heard today that a new paradigm is coming and AAPOR ought to lead it. Well, sadly, if AAPOR members still think response rates with panels are meaningful,  nonprobability panels are worthless, and they’re still doing email subject line tests, oh my you’re in for a treat when you discover what eye-tracking is. AAPOR leading? Not even close. You’re meandering at the very end of an intensely competitive horse race.

Dear AAPOR, please enter the 21st century. Market researchers have been doing online surveys for twenty years. We finished our online/offline parallel tests ten years ago.  We finished subject line testing ten years ago too. We’ve been doing big data for 50 years.  We’ve been using social media data for 10 years.  I could go on but there’s no need.

Where have you been all these years? Arguing that probability panels are the only valid method? That’s not good enough. Let me know when you’re open to learning from someone outside your bubble. Until then, I’ll be at the front of the horse race.

Evaluating response rates for web surveys #AAPOR #MRX 

prezzie #1: do response rates matter

  • response rates used to be an indicator of data quality, are participation rates meaningful?
  • completion rates were related to higher error [makes sense to me, making everyone answer a survy includes people who don’t want to answer the survey]
  • if you only surcey people who answer surveys, your response rates will be high
  • emerging indicator is cumulative response rate for probability panels = recruit rate * profile rate * participation rate
  • largest drop in rate is immediately after recruitment, by fifth survey the rate really slows down [this is fairly standard, by this point people who know they don’t like participating have quit]
  • by this measurement, the cumulative response rate had dropped to less than 3%, across all the groups the rate was less than 10%  [tell me again that a probability panel is representative. maybe if the units were mitochondria not humans who self determine, hello non-sampling error!]

prezzie 2: boosting response rates with follow-ups

  • 30% response rate with follow up compared to 10% with no follow up using the aapor response rate
  • follow ups helped a little with hispanic rates
  • helped a lot for cell phone only households
  • helped a lot for lowest and highest   income households, adults under 50 years old, high school only education
  • [hey presenters, slides full of text might as well be printed and handed out, and since i’m on this topic, yellow on white does not work, fonts under 25 point don’t work, equations don’t work. use your slides wisely! and please don’t read your slides 😦 ]

prezzie 3: testing email invitations in a nonprobability panel

  • using mobile optimized forms 🙂
  • used short invites sent from census bureau with census logo
  • best subject line as chosen by responders was – help us make the us census better, answer our survey
  • but real data showed this was worse than the others tested
  • best message was the message focusing on confidential, and possibly even better if you specify 10 minutes

prezzie 4: does asking for an email address predict future participation

  • response rates were 2 to 3 times higher for people who gave an email address
  • but it’s not exactly the email as an indicator, it’s people open to participating in further research
  • no effects by gender or ethnicity, graduate degree people are less likely to provide their email address

prezzie 5: predictors of completion rates

  • first selected only studies with completion rates over 60% [don’t know why you would do this, the worst surveys are important indicators]
  • completion rates are higherif you start with a simple multiple choice, lower if you start with an open end
  • introductory text and images don’t help completion rates
  • completion rates decrease as number of questions increase
  • higher completion rates if you put it all on one page insteading of going page page page page
  • a title increases the rate, and numbering the questions increases the rate for shorter surveys
  • open ends are the worst offendors for completion rates, question length and answer length is next worse [so stop making people read! you know they aren’t actually reading it anyways]
  • respondents don’t want to use their keyboards
  • avoid blocks of text

[my personal opinion… response rates of online panels have no meaning. every panel creates the response rate that is suited to their needs. and they adjust these rates depending on the amount and type of work coming in. response rates can be increased by only sending surveys to guaranteed responders or lowered by sending surveys to people who rarely respond. and by adjusting incentives and recruitment strategies you can also raise or lower the rates. instead, focus all your inquisitions on data quality. and make sure the surveys YOU launch are good quality. and i don’t mean they meet your needs. i mean good quality, easy to read, engaging surveys.]

by the way, this was a really popular session!

  

%d bloggers like this: