Tag Archives: research

How the best research panel in the world accurately predicts every election result #polling #MRX 

Forget for a moment the debate about whether the MBTI is a valid and reliable personality measurement tool. (I did my Bachelors thesis on it, and I studied psychometric theory as part of my PhD in experimental psychology so I can debate forever too.) Let’s focus instead on the MBTI because tests similar to it can be answered online and you can find out your result in a few minutes. It kind of makes sense and people understand the idea of using it to understand themselves and their reactions to our world. If you’re not so familiar with it, the MBTI divides people into groups based on four continuous personality characteristics: introversion/extroversion, sensing/intuition, thinking/feeling, judging/perception . (I’m an ISTJ for what it’s worth.)

Now, in the market and social research world, we also like to divide people into groups. We focus mainly on objective and easy to measure demographic characters like gender, age, and region though sometimes we also include household size, age of children, education, income, religion, and language. We do our best to collect samples of people who look like a census based on these demographic targets and oftentimes, our measurements are quite good.  Sometimes, we try to improve our measurements by incorporating a different set of variables like political affiliation, type of home, pets, charitable behaviours, and so forth. 

All of these variables get us closer to building samples that look like census but they never get us all the way there. We get so close and yet we are always missing the one thing that properly describes each human being. That, of course, is personality. And if you think about it, in many cases, we’re only using demographic characteristics because we don’t have personality data. Personality is really hard to measure and target. We use age and gender and religion and the rest to help inform about personality characteristics. Hence why I bring up the MBTI. The perfect set of research sample targets. 

The MBTI may not be the right test, but there are many thoroughly tested and normed personality measurement scales that are easily available to registered, certified psychologists. They include tests like the 16PF, the Big 5, or the NEO, all of which measure constructs such as social desirability, authoritarianism, extraversion, reasoning, stability, dominance, or perfectionism. These tests take decades to create and are held in veritable locked boxes so as to maintain their integrity. They can take an hour or more for someone to complete and they cost a bundle to use. (Make it YOUR entire life’s work to build one test and see if you give it away for free.) Which means these tests will not and can not ever be used for the purpose I describe here. 

However, it is absolutely possible for a Psychologist or psychological researcher to build a new, proprietary personality scale which mirrors standardized tests albeit in a shorter format, and performs the same function. The process is simple. Every person who joins a panel answers ten or twenty personality questions. When they answer a client questionnaire, they get ten more personality questions, and so on, and so on, until every person on a panel has taken the entire test and been assigned to a personality group. We all know how profiling and reprofiling works and this is no different. And now we know which people are more or less susceptible to social desirability. And which people like authoritarianism. And which people are rule bound. Sound interesting given the US federal election? I thought so. 

So, which company does this? Which company targets people based on personality characteristics? Which company fills quotas based on personality? Actually, I don’t know. I’ve never heard of one that does. But the first panel company to successfully implement this method will be vastly ahead of every other sample provider. I’d love help you do it. It would be really fun. ūüôā


The 2014 Gift Guide for Geeks, Dorks, and Research Gurus

Well, it’s that time of year again!

Regardless of which holiday you celebrate¬†and even if you celebrate the holiday of “I deserve a treat today”,¬†you’re sure to find a statistics gift for yourself or your loved ones below. Just click on the image to go to the website and order. Go! Quickly before they run out! Shirts, cups, hats, toddler toys, and more, they’re all here.

If you come across other fun statistics and research gifts, leave a link in the comments for other folks. Enjoy!
normal distribution plush
wood puzzle number
statistics blocks
statistically significant cap hat
statistics mousepad
statistically significant bag tote
science shirt
statistics decal car
normal distribution dinosaur shirt
phd bs apron
phd iphone case
statistics burp cloths
cup cozy
normal signicant cross stitch
plush pie chart
pie chart poster
pi cookie cutter
pi tie clip

WAPOR Day 1: Why are Google, Facebook, and Microsoft so far ahead of us in research? #MRX

WAPOR opened with a bang as David Fan described the statistical techniques he used to organize the accepted papers into relevant bunches. The key terms included cluster analysis and the traveling salesman approach as a number of presenters were asked to determine which of the other accepted papers were most similar to theirs. One of the methodological issues that had to be dealt with was that some presenters were forced¬†to back out at the last minute such that the carefully designed grouping didn’t end up being¬†perfect. Alas, as with every research project, errors creep in.

And in case you’re curious, no, there was no parade of WAPOR figure heads each welcoming us with a short prepared talk. There were no dance routines, fun videos, or Nice tourism representatives. Yes, a room full of data geeks got a truly geeky talk from the¬†head geek. ¬†I’m still chuckling about it. ūüôā

Rather than summarize the talks I went to, I’ll mention a few interesting tidbits and a few thoughts that came to mind for me.

  • Do you ever consider responder needs, not your own needs? When you’re designing surveys, do you ever really think about what the responder needs as part of the research process? I know you want quality data and you want to design surveys that generate quality data, but do you really think about the fact that responders may want to answer a survey on a phone because they can take it to a private room or a quiet room? ¬†Similarly, do you realize that people may not want to answer a phone survey because there are other people in the room or it’s too noisy for them? Stop fussing over whether you do or don’t want people to take a survey on their phone. Give them the tools to give you the best data they can – from a quiet room, a private room, or anywhere.
  • People don’t fan pages they don’t like. One of the speakers mentioned that people don’t fan brand pages if they aren’t truly fans of the brand. Well, that’s not completely true. Many people ‘fan’ or ‘like’ a page so that they can leave a complaint or criticism on it. Or, they want to monitor what the brand is doing to see how it compares to their loyal brands. Or, they like the page to learn about discounts and coupons that they can redeem with their own brand. Whether Facebook or Twitter, it doesn’t matter what the social network names the buttons – people will click on the button that suits their purpose.
  • Social media data has yet to be validated. Someone also mentioned that social media data is taking a while to become widely used because the data itself hasn’t been validated yet. For instance, if someone tweets that they went to McDonald’s, did they really go to McDonald’s. I found this comment kind of funny coming from someone in the survey world. Hm… if someone says on a survey that¬†they went to McDonald’s, did they really go to McDonald’s? Something to ponder!
  • Why are Google, Facebook, and Microsoft so far ahead in research? This comment came up as a tangent and was never answered by the speaker, but I’ll take it on here. Why? Because they aren’t research companies. They don’t have to fuss and fret and worry that their norms and standards will be royally screwed up. They aren’t worried about fitting 412 questions into 5 minutes of survey time. They aren’t trying to figure out how to make their product ‘fun.’ We DO have to worry about these things. Actually, I disagree that we have to worry. If we keep worrying as we have been, then Google and Facebook and Microsoft will wipe our faces with their research. If we don’t get with the times and become our own thought leaders, that’s what’s going to happen. Be aware of your norms and be cautious as you change them. Make the research experience enjoyable as it should be. It’s your business at stake. Stop talking. Start doing. (me included!)
  • Are AAPOR guidelines too American? You know, I never really thought of that before. There are a number of organizations in the research world that want to be global. Given that WAPOR is the world version of AAPOR, I must conclude that AAPOR does want to be global. Yes, as was mentioned during today’s talk, most of the AAPOR guidelines are drawn with first world, English countries in mind – everyone has a phone, everyone has a smart phone, everyone has a physical legal home. Do the AAPOR guidelines make it easy or even possible for people in other countries to conduct ‘good’ research? It’s worth a ponder.
  • Let’s stop the probability/non-probability debate. Hear hear! I don’t believe there is such thing as a probability sample in the human world (generally speaking). Yet, AAPOR continues to promote the idea. You see, even if you COULD know an entire population and select a random sample, people will still decline to participate, quit participating, answer questions incorrectly, misread questions, lie on questions, etc. The assumption is that probability samples create perfect data and this is just never the case. I would love it if we could just drop the whole probability superiority complex and get on with our work.
  • Candy is a legitimate snack. Breaktime featured a fine selection of…. candy? yes, candy. For the second time today, I was happily shocked. Someone later mentioned that fruit was also available but I don’t know what that is and I didn’t see it. So they lied.

And that, my friends, is the Day 1 wrap!

GUN control, he said GUN control! ¬†ūüôā

Holiday Shopping with All Screens 24/7 by Maria Domoslawska, and Roddy Knowles #FOCI14 #MRX

Live blogging from the #FOCI14 conference in Universal City. Any errors or bad jokes are my own.foci14

Holiday Shopping with All Screens 24/7
Maria Domoslawska and Roddy Knowles, RESEARCH NOW

  • multimode starting with online, going to metered PCs, mobile phones, and then mobile diary GPS tracking which adds more engagement and gets us closer to the consumer
  • sample sizes of thousands, conducted around Thanksgiving
  • partnered with Experian for Hitwise data, tracked the top 20 retailers
  • 3% of people used mobile for purchasing but we will see mobile did play a central role
  • saw spikes for black friday and cyber monday on PC traffic
  • even though people aren’t necessarily shopping on their phone, the phone is with them at all times, they use their phones while they shop a lot more than they used to
  • 40% of consumers say they LIKE the huge crowds in the UK, and plan to do this
  • 31% of people made only unplanned purchases, impulse
  • 26% made planned and unplanned purchases
  • purchase planning by store – more unplanned at CJP, Macy’s, Kohl’s – clothing stores
  • in the UK, they plan to buy after Christmas, they give gifts on boxing day because they know they can get a better deal
  • more planned purchases for bigger ticket items like appliances and electronics
  • flowers and small gifts were mostly unplanned
  • ask people to take photos of their receipt – would people actually do it? and are panelists buying what they said they bought – did they forget something?
  • used the mobile diary to determine if people could actually find what they were looking for
  • were able to capture basket spend via diary data and determine average expenditure by store, matched well with responder data so people were able to recall quite accurately
  • 74% say gift giving makes them feel good
  • Key takeaways:
    • multi-country tracking help find subtle changes in your market which can become a big creative idea for your next program
    • mobile is in shopper’s lives and will not go away. it is an essential device for shopping even if they aren’t actually purchasing on their device
    • getting close to the point of experience yields rich and accurate data on shopping behaviours
    • not every datapoint is going to add value but layering survey data with new variables of passive behavioural data can activate your hypothesis

Other Posts

Trends for Mobility Research by Mark Michelson #NetGain8 #MRX

Live blogging from MRIA’s #NetGain8 conference in Toronto. Any errors or stupid jokes are my own.


Best Practices and Trends for Mobility Research, Mark Michelson, Executive Director and Co-Founder, Mobile Marketing Research Association, Atlanta

  • ¬†Everyone wears many hats, including mark who also plays guitar in a band
  • mobile brings all these different lives together, converges together new companies and new opportunities, as well as people who can’t get on a plane and speak live at a conference due to being snowed in
  • smartphones are only 5 years old and they have improved speed of delivery – always on, always close, real time delivery, people participate at their convenience, passive data collection via GPS or device use, longitudinal work all around the world with instant translations, pictures of products and vehicles come instantly, talk to someone on the phone while they’re shopping, dissipated workforce, QR and scanner codes to engage
  • smartphone make participation more enjoyable, more social sharing to explore each other’s lives, self-reported ethnography, live video chats anywhere, augmented reality, gamification
  • [quite neat to have a virtual presentater about mobile methods where the researcher is virtually there]
  • quant mobile mr data – survey is Q&A, passive monitoring without interaction, field conditions like merchandising brand audits or mystery shopping, sensory such as how people perceive via galvanic skin response, temperature, etc
  • qual mobile mr data – self ethnography with video photos, what people say on bulletin boards or discussion groups, emotional feelings in diarys, projective research, video chats or audio recordings in the moment
  • challenges – designs to fit the screen, user experience and training, keeping people engaged, validity of self-report, optimization for different devices, client trust in findings, participant trust
  • ethics in mobile mr – privacy is critical for legal reasons and continued open participation
  • commitments to responders – voluntary, protect privacy
  • challenges – normative data include more positive top box scores, questionnaire length, how to calibrate tracking data, how to incorporate into existing methods
  • best practices – transparency, respect, share the intent, expectations in advance, avoid burnout, keep it short, respect time zones, be screen and device appropriate, consider right question type for device
  • best practices – micro surveys, in-context for what people are doing right now, mobile friendly tools like dials sliders, graphic interface
  • share the guidelines with your staff, clients, and participants,
  • Learn more on the association site mmra-global.org

An Open Letter to Women in Research #MRX

Dear women in research,
Your research proposal is weak. Your write-up is uninspiring. Your topic has been done before and someone else did it way better than you. You have have almost no experience in the field. You’re too shy, nervous, and boring. You’re not a good speaker. No one knows who you are. You aren’t a senior member of your company. You’re only a tiny company. No one cares who you are. These are all terrible reasons not to submit proposals to all of the market research conferences in 2014. If you want someone to encourage you, ask any of the 260 people in this group who will say SUBMIT YOUR PROPOSAL NOW.

Your participation is important to us

[This was a spur of the moment comment I made on the WIRe Facebook page. Please share it with all of your colleagues, not just women, not just people from minority groups.]

These are a few of my favourite tweets Day 2 #MRMW #MRX

We’ve arrived at the end of day 2! Though the twitterverse was active again today, it was not quite as active as yesterday. I even took down my tweeting so that I didn’t hog the #MRMW stream so much but I still ended up writing the most tweets! Have no worries folks, for the #MRMW has closed for this time and my tweets will go back to normal. Until then, enjoy these lists of top tweeters and top tweets.
#MRMW HashTracking.com Report
Top 10 by number of impressions

  1. tomhcanderson: 678,636
  2. lovestats: 560,924
  3. imrananwar: 68,894
  4. jebbing9: 67,732
  5. dougschorr: 61,162
  6. berniemalinoff: 48,188
  7. bettyadamou: 38,310
  8. revelationinc: 36,142
  9. argentbeauquest: 27,693
  10. thesurveygeek: 27,478
Top 10 by number of tweets

  1. lovestats: 92
  2. jebbing9: 41
  3. goinnovateblog: 38
  4. jasonhauer: 33
  5. bettyadamou: 30
  6. berniemalinoff: 28
  7. dianetweeting: 27
  8. dirkgently: 26
  9. kristopherhull: 23
  10. thegaragegroup: 23
Top 10 by number of followers

  1. tomhcanderson: 48,474
  2. dougschorr: 30,581
  3. argentbeauquest: 27,693
  4. incslinger: 11,689
  5. imrananwar: 9,842
  6. webby2001: 7,840
  7. virtualmr: 6,485
  8. lovestats: 6,097
  9. esomar: 5,615
  10. allanalytics: 5,072

SMART Social Media Research Bootcamp Videos #MRX

I just got back from a whirlwind tour around the US during which time I presented a number of full day Conversition workshops on social media research. We started the day with a bit of theory related to data quality, sampling, content analysis, and sentiment analysis. Then, we enjoyed a lovely lunch where I focused mainly on the olives and desserts, the essential components of any meal including breakfast. Finally, we finished the day with some hands on practice using real live data in EvoListen from brands like Netflix, Walmart, McDonalds, 5 Guys, and more. If you found yourself unable to attend, have no fear, the videos are here! Each one is just a few minutes long so you can eat them up in nice little bites. Enjoy!

  1. Pros and Cons of SMR http://youtu.be/uDkDrgugN6I
  2. Maintaining Data Quality  http://youtu.be/1QTqZAIvCkI
  3. Sampling and Weighting http://youtu.be/8d09DYpDq5Y
  4. Content Analysis http://youtu.be/jNOE89IAjs4
  5. Emoticon Test http://youtu.be/0R_BOi3ZJlo
  6. Acronym Test http://youtu.be/php9Pd3D830
  7. Sentiment Analysis http://youtu.be/v_0nzbLbPZI
  8. Validating Sentiment Analysis http://youtu.be/RERKsNLkTcM
  9. Scaling Social Media Data http://youtu.be/hw_gKdrVpOc
  10. EvoListen, Variables and Word Clouds http://youtu.be/g0oD-dKl3wU
  11. Slang Test http://youtu.be/F-CRv_S65p8
  12. S.M.A.R.T. Test http://youtu.be/8SDWGcSK17Y

Dr. Seuss Interviews the Author of “The Listen Lady” #MRX

The Cat in the Hat by Dr. Seuss, 1957 It is with ¬†great honour that I share this interview with you. Dr Seuss is one of my favorite authors and I was thrilled when he contacted me to receive an advance copy of The Listen Lady and subsequently interview me. So without further delay…


Dr. Seuss: Did you write it in a house? Did you write it with a mouse?

Dr. Pettit: Um… ok…. Those are odd questions to start with but yeah, over the last year or so I wrote it in my house, at least most of it. Some parts were written in my backyard or while I was traveling. And in terms of the mouse, I tended to use the scroll pad on my laptop instead. A scroll pad isn’t nearly as efficient or comfortable as a mouse but I made it work.

Dr. Seuss:¬†“Don’t give up! I believe in you all. ¬†A person’s a person, no matter how small!¬†And you very small persons will not have to die.¬†If you make yourselves heard! So come on, now, and TRY!”

listen lady social media research

Dr. Pettit: Your encouragement is very much appreciated but I’m really not that small. I’m five foot eleven. Regardless, I did try very hard. There were many occasions when I wasn’t sure ifI could finish writing the book but then I’d talk to some new people who were really curious about how social media research worked and they had nowhere to turn to get information. Those conversations would re-energize me and I’d be eager to get my hands on my laptop again. In no time at all, I’d be back on track.

Dr. Seuss: From there to here, from here to there, funny things are everywhere.

Dr. Pettit: You’re too kind! I didn’t intend it to be a humorous novel but I did try to put funny little stories in here and then. Actually, when of the stories are drawn from real situations. Some of my colleagues might recognize a story or two. I know your focus is always the humour but I hope you enjoyed more than just the funny bits. I hope you enjoyed how I turned a social media research textbook into an interesting story, a story that anyone, regardless of their experience with research or baking can learn from.

Dr. Seuss: And that is a story that no one can beat, When I say that I saw it on Mulberry Street.

Dr. Pettit: Ok then… That’s great… I’m not sure what bookstore is on Mulberry Street but I’ll be sure to check it out. Thanks. I think.



The Listen Lady is available on Amazon.com, and on Smashwords if you’d like an ebook version. Like TheListenLady fan page on Facebook to get the latest news on availability and stay tuned¬†for more info! (No, I was not REALLY interviewed by Dr. Seuss. Check the facts jack.)

%d bloggers like this: