Forget for a moment the debate about whether the MBTI is a valid and reliable personality measurement tool. (I did my Bachelors thesis on it, and I studied psychometric theory as part of my PhD in experimental psychology so I can debate forever too.) Let’s focus instead on the MBTI because tests similar to it can be answered online and you can find out your result in a few minutes. It kind of makes sense and people understand the idea of using it to understand themselves and their reactions to our world. If you’re not so familiar with it, the MBTI divides people into groups based on four continuous personality characteristics: introversion/extroversion, sensing/intuition, thinking/feeling, judging/perception . (I’m an ISTJ for what it’s worth.)
Now, in the market and social research world, we also like to divide people into groups. We focus mainly on objective and easy to measure demographic characters like gender, age, and region though sometimes we also include household size, age of children, education, income, religion, and language. We do our best to collect samples of people who look like a census based on these demographic targets and oftentimes, our measurements are quite good. Sometimes, we try to improve our measurements by incorporating a different set of variables like political affiliation, type of home, pets, charitable behaviours, and so forth.
All of these variables get us closer to building samples that look like census but they never get us all the way there. We get so close and yet we are always missing the one thing that properly describes each human being. That, of course, is personality. And if you think about it, in many cases, we’re only using demographic characteristics because we don’t have personality data. Personality is really hard to measure and target. We use age and gender and religion and the rest to help inform about personality characteristics. Hence why I bring up the MBTI. The perfect set of research sample targets.
The MBTI may not be the right test, but there are many thoroughly tested and normed personality measurement scales that are easily available to registered, certified psychologists. They include tests like the 16PF, the Big 5, or the NEO, all of which measure constructs such as social desirability, authoritarianism, extraversion, reasoning, stability, dominance, or perfectionism. These tests take decades to create and are held in veritable locked boxes so as to maintain their integrity. They can take an hour or more for someone to complete and they cost a bundle to use. (Make it YOUR entire life’s work to build one test and see if you give it away for free.) Which means these tests will not and can not ever be used for the purpose I describe here.
However, it is absolutely possible for a Psychologist or psychological researcher to build a new, proprietary personality scale which mirrors standardized tests albeit in a shorter format, and performs the same function. The process is simple. Every person who joins a panel answers ten or twenty personality questions. When they answer a client questionnaire, they get ten more personality questions, and so on, and so on, until every person on a panel has taken the entire test and been assigned to a personality group. We all know how profiling and reprofiling works and this is no different. And now we know which people are more or less susceptible to social desirability. And which people like authoritarianism. And which people are rule bound. Sound interesting given the US federal election? I thought so.
So, which company does this? Which company targets people based on personality characteristics? Which company fills quotas based on personality? Actually, I don’t know. I’ve never heard of one that does. But the first panel company to successfully implement this method will be vastly ahead of every other sample provider. I’d love help you do it. It would be really fun. 🙂
Live note taking at #AAPOR in Austin Texas. Any errors or bad jokes are my own.
Moderator: Lisa Drew, two.42.solutions
RAND 2016 Presidential Poll Baseline Data – PEPS; Michael S. Pollard, RAND Corporation Joshua Mendelsohn, RAND Corporation Alerk Amin, RAND Corporation
- RAND is nonprofit private company
- 3000 people followed at six points throughout the election, starting with a full baseline survey in December, before candidates really had an effect, opinions of political issues, of potential candidates, attitudes towards a range of demographic groups, political affiliation and prior voting, a short personality questionnaire
- Continuously in field at first debate
- RDD but recruited RDD and then offered laptops or Internet service if needed
- Asked people to say their chance of voting, and of voting for democrat, republican, someone else, out of 100%
- Probabilistic polling gives an idea of where people might vote
- In 2012 it was one of the most accurate popular vote systems
- Many responders a have been surveyed since 2006 providing detailed profiles and behaviors
- All RAND data is publicly available unless it’s embargoed
- Rated themselves and politicians on a liberal to conservative scale
- Perceptions of candidates have chanced, Clinton, Cruz, and average democrat more conservative now, trump more liberal now; sanders, kasich, average republican didn’t move at all
- Trump supporters more economically progressive than Cruz supporters
- Trump supporters concerned about immigrants and support tax increases for rich
- If they feel people like me don’t have a say in government, they are more likely to support trump
- Sanders now rates higher than Clinton on “cares about people like me”
- March – D was 52% and R was 40%, but we are six months aware from an election
- Today – Clinton is 46% and Trump is 35%
- Didn’t support trump in December but now do – Older employed white men born in US
- People who are less satisfied in life in 2014 more likely to support rump now
- Racial resentment, white racists predict trump support [it said white ethnocentrism but I just can’t get behind hiding racism is pretty words]
Cross-national Comparisons of Polling Accuracy; Jacob Sohlberg, University of Gothenburg Mikael Gilljam, University of Gothenburg
- Elections are really great [ made me chuckle, good introduction 🙂 ]
- Seen a string of failures in many different countries, But we forget about accurate polls, there is a lot of variability
- Are some elections easier than other? Is this just random variance? [well, since NO ONE uses probability sampling, we really don’t know what MOSE and MONSE is. ]
- Low turnout is a problem
- Strong civil society has higher trust and maybe people will be more likely to answer a poll honestly
- Electoral turnover causes trouble, when party support goes up and down constantly
- Fairness of elections, when votes are bought, when processes and systems aren’t perfect and don’t permit equal access to voting
- 2016 data
- Polls work better when turnout is high, civil society is Truong, electoral stability is high, vote buying is low [we didn’t already know this?]
- Only electoral turmoi is statistically significant in the Multivariate analysis
Rational Giving? Measuring the Effect of Public Opinion Polls on Campaign Contributions; Dan Cassino, Fairleigh Dickinson University
- Millions of people have given donations, it’s easier now than ever before with cell phone and Internet donations
- Small donors have given more than the large donors
- Why is Bernie not winning when he has consistently out raised Hillary
- What leads people to give money
- Wealthy people don’t donate at higher rates
- It’s like free to play apps – need to really push people to go beyond talking about it and then pay for it
- Loyalty base giving money to the candidate they like, might give more to her if they see her struggling
- Hesitancy based only give if they know they are giving to the right and iodate, so they wait
- Why donate when your candidate seems to be winning
- Big donors get cold called but no one gets personality phone calls if you’re poor
- Horse race coverage is rational, coverage to people doing well, don’t really know about their policies
- Lots of covereage on Fox News doesn’t mean someone is electable
- People look at cues like that differently
- In 2012 sometimes saw 5 polls every day, good for poll aggregators not good for people wanting to publicize their poll
- You want a dynamic race for model variance
- Used data from a variety of TV news shows, Fox, ABC, CBS, NBC
- Don’t HAVE to report donation under $200, many zero dollar contributions – weirdness needed to be cleaned out
- Predict contributions will increase when Romney is threatened in the polls
- Predict small contributions will increase in response to good coverage on Fox News
- Fox statements matter for small contributors, doesn’t matter which direction
- Network news doesn’t matter for small contributors
- Big donor are looking for more electable candidates so if fox hates them then we know they’re electable and they get more money
- Romney was a major outlier though, the predictions worked differently for him
live blogged at #PAPOR in San Francisco. Any errors or bad jokes are my own.
The Summer of Our Discontent, Stuart Elway, Elway Research
- regarding state of washington
- it’s generally democratic
- between elections, more people are independents and then they drift to democrat
- indepents are more social liberals
- has become more libertarian
- dont expect a rebellion to start in washington state
- [sorry, too many details for me to share well]
Californian’s Opinion of Political Outsiders, Mark Baldassare, PPIC
- california regularly elects outsiders – Reagan, Schwarzenegger
- flavour is often outsider vs insider, several outsiders have run recently
- blog post on the topic – http://ppic.org/main/blog_detail.asp?i=1922
- they favour new ideas over experience
- 3 things are important – approval ratings of elected officials, people who prefer outsiders give officials lower approval, negative attitudes of the two party system
- majority think a third party is needed – more likely to be interested in new ideas over experience
- [sorry, too many details for me to share well]
Trump’s Beguiling Ascent: What 50-State Polling Says About the Surprise GOP Frontrunner, Jon Cohen & Kevin Stay, SurveyMonkey
- 38% of people said they’d be scared if trump is the GOP nominee
- 25% would be surprised
- 24% would be hopeful
- 21% would be angry
- 14% would be excited
- List is very different as expected between democrats and republicans, but not exactly opposite
- quality polling is scale, heterogeneity , correctable self-selection bias
- most important quality for candidates is standing up for principles, strong leader, honest and trustworthy – experience is lowest on the list
- Views on Trump’s muslim statement change by the minute – at the time of this data: 48% approve, 49% disapprove, split as expected by party
- terrorism is the top issue for republicans; jobs AND terrosiam are top for independants; jobs is top for democrats
- for republicans – day before paris 9% said terrorism was top, after paris 22%
- support for Cruz is increasing
- half of trump voters are absolutely certain they will vote for trump; but only 17% of bush voters are absolutely certain
- among republicans, cruz is the second choice even among trump voters
- trump has fewer voters who go to religious services weekly, least of all candidates; carson and cruz are on the high end
- trump voters look demographically the same but carson has fewer male voters and cruz has fewer female voters
- trump voters are much less educated, rubio voters are much more educated
Panel Discussion on Political Polling & Media in Canada: “Election Polling in the West – Has it Changed The Research Industry For the Better?” #MRIA14 #MRX
Live blogging from the #MRIA national conference in Saskatoon, Saskatchewan. Any errors or bad jokes are my own.
Moderator: Steve Mossop, President, Insights West; Panelists: Éric Grenier, polling analyst and the author of ThreeHundredEight.com; Tim Olafson, Co-founder, Stone-Olafson; Scott MacKay, President – Probe Research Inc.; Lang McGilp, Senior Research Executive, Insightrix Research
- [note – there is lots of debate and differences of opinions among the speakers, i have not indicated who said what]
- many voters change their minds at the very last minute, political polling is not broken in canada
- 22% of voters changed their mind in the last few days of the election
- we always ask “how would you vote today.” We need to ask the right questions
- there is still a role for public election polling, parties have the information so the public should have it as well
- campaigns will be dominated by internal polls because they will put out poll results themselves
- no more trust between public and pollsters and we need to rebuild that trust; better polling costs money, money that we don’t have, need more cooperation between pollsters and journalists is reported the right way
- pollster in-fighting looks terrible, media sees the fights and focus on the people who got it wrong
- tired of giving away free polling to build up brand recognition
- pollsters doing a crappy job of setting the context, they focus on a time frame or election, they don’t look back at the last election at society at the bigger picture
- we need to get rid of the ban on polling publication
- industry needs to be less competitive and more open with best practices
- we have civic reasons to do the polling, good for democracy
- there are many people who want us to get polling wrong
- there are too many free polls, angus reid in the west complains the most
- some people think more pollsters is better – 280 pollsters were doing it in the US, Canada probably only needs around 12. comparatively, not as many in Canada
- do engineers or lawyers offer free engineering and free lawyering? Free undervalues our work
- some firms refuse to release any public that is not paid for
- paid for polls are more accurate because you ask more questions
- we trivialize elections with so many polls based on insufficient survey questions; will the media cover costs of a 60 question survey
- polls these days are just horse race measures
- how can we prove that polling works – we’ve called elections accurately for the last 50 years, except when it’s wrong [margin of error people]
- polling used to be much more accurate, record was unblemished. what happened? we started using online panels. some panels aren’t good for this kind of research. telephone method is not dead. it works well. panels won’t work in smaller regions. Do not write off the phone at this point.
- there are region specific panels that were built carefully, based on telephone recruit. These panels are extremely accurate.
- method doesn’t matter. society has changed. it used to be the newspaper in the evening and news on TV at night. Now news is instant all day long.
- not a lot of telephone any more in ontario but any methodology can get it right
- turnout determines accuracy of polls, it’s luck
- voter turnout is declining especially among younger people which means we will need to build likely voter models, this is new for many people
- some region have publicly available voter lists, can be purchased, can determine who has and hasn’t voted
- is it intention, past behaviour that predicts best?
- we don’t ask the right questions, need to probe the undecided better, shouldn’t focus solely on undecided voters and they could be leaning heavily into one camp
- maybe we don’t know what’s going on
- how can we do a better job of predicting elections? voter models which we really haven’t been using [seriously? you aren’t using models? i’m seriously shocked.]
- a publication ban is not a polling ban, we should keep polling until the end so we get a better sense of what’s going on
- perhaps publish your numbers as an exit poll
- people dislike polling because ‘we’re wrecking democracy,’ we’re telling people ahead of time what will happen
- need more transparency, show the numbers, show the questions, show the weighting – this helps to avoid in-fighting
- prediction markets – one happened in BC and followed the polls exactly but it was wrong at the outcome like all of the polls
Live blogging from the #MRIA national conference in Saskatoon, Saskatchewan. Any errors or bad jokes are my own.
Opening Keynote Presentation
Darrell Bricker – The Big Shift
- What region is less likely to believe it is disrespectful to put the Canadian flag on their underwear – BC
- What province is more religious – Atlantic
- What region has most fire detectors, fire protection – Atlantic
- What region is less likely to believe Canada would be better off immigrants when back to where they came from – Alberta
- What region cares most about environment – Atlantic
- What region believes Anne Murray should be put on Canadian money – Manitoba, Saskatchewan
- What region believes they belong to the country first, province second – Ontario
- What is Canada’s favourite doughnut? – Boston Cream
- You THOUGHT you knew Canada but it’s not what you thought
- We used to be English and french and very white, more rural and focused on natrual resources, driven by Montreal and Ottawa, trusting of authorities, fearful, paternalistic, conservative, judgmental, anti-american, used to be equal male female across the ages
- The age pyramid we used to know is gone. Fewer kids being born, average Canadian lives a lot longer.
- You need 2.1 kids per couple to grow but we are only producing 1.7 kids per. Japan is worse at only 1.4 per. Is this why japan is obsessed with creating robots? Germany’s population is shrinking.
- Population of prairies is actually growing more now than in 1961, BC is growing much more than it did in 1961. This changes the balance of power.
- Calgary is the fastest growing city in Canada. Montreal has the oldest population, isn’t growing, high debt.
- Why the surge? It’s immigration. Three times as many immigrants 1986 vs 2010. We accept more immigrants than any other country in the world.
- Far more immigrants coming under economic class, family class is now far behind, and refugees is even further behind. New software engineers, new pipeline workers, service jobs – these people are coming with skills.
- They are coming from Philippines, india, china, UK, US, France, Iran, UAE, Morocco, South Korea. In 1970 it was pretty much UK, US, West Indies, Italy, Portugal. Very very different. Settling around Toronto, and suburbs of western Canada. They aren’t going to Quebec or Atlantic Canada. We are becoming a pacific population and a brown population. Families are all moving west as well because that’s where the opportunities are.
- Toronto is 49% foreign born, Vancouver is 39% foreign born. Miami is 37%, LA is 35%, Montreal 20%, Halifax 5%.
Why are immigrants coming here? We need them. By 2020, we’ll be one million skilled jobs short.
- Canada is third in the percentage of people assessing the current economic situation in their country, western Europe is a disaster. Canada feels twice as good as they US 60% positive vs 30% positive.
- Immigrants feel we are polite, beautiful country, friendly, good health care, care about the environment, educated, different from Americans, well off, welcoming, tolerant of different people
- Canada third on list of “country you’d move to that is at least five hours away and you’d live there for at least two years”
- The NEW Canadian mindset
- more urban, multi-cultural, world traders, older, more female, tolerant, opinionated, demanding, more ideological divided, less engaged with traditional institutions, aggressively Canadian
- We’ve had to learn how to join a french community and an english community live together, we are doing it without guns and fighting, we don’t have massive civil disruption
- Maybe you should market to single, older, women – their housing, travel, personal needs
- respect for every institution has declined, people are demanding more choice and more voice
- new Canadian pride – less apologetic and more in your face
- the emerging Canadian political landscape – consolidating right
- old way to win elections was about Ontario and Quebec, keeping Quebec happy
- in 2011, new coalition emerged with the west, rural areas, and Toronto suburbs, Quebec population has decline and had less impact on elections; government is part of the problem now, new Canadians support the monarchy not the ‘older’ Canadians
- NDP is now about to get over 25% which means Liberals can’t win.
What follows are some of my silly musings and key take-aways of the session.
Brian Singh, ZINC Research, Insights from Nenshi Campaign
– Who is Nenshi? No awareness nor familiarity, no money for this political candidate in Calgary
– Used multiple methods, translated social media into REAL ACTION
– They needed to “seed” advocates, point people to where you want them to look, give something for people to talk about
– Are the people you are trying to speak to even on social media?
– They needed to connect to the hyper-engaged so that those people will build the information
– (They had a research strategy with their social media research!)
– Really nice perceptual map of leadership by personability, Nenshi was higher on personal, lower on leadership, very in the middle of the other political candidates
– Found the most common searches were “is he gay” and “is he muslim” – so they created videos around these questions
– Crowdsourced material is extremely important and usually the most popular
– People love the homemade political messages, barney images, handmade voting signs, graffiti
– Need to marry social media research with political polling, you must be literate in this method
#Netgain5 Keynote Roundup: Last Thoughts
Brian Levine: Neuroscience and Marketing Research
Brian Singh: Insights from the Nenshi Campaign
Monique Morden: Online Communities, MROCs
Ray Poynter – Overview of Online Research Trends
Tom Anderson: Web Analytics
Will Goodhand: Social Media Research and Digividuals
Toronto is in the midst of a heated contest with two major mayoral candidates, Rob Ford and George Smitherman. Which means we’re going to hear a lot of numbers being thrown at us. So here is a quick and easy guide to what those numbers mean.
In this case, a poll means that a bunch of people have been asked who they will vote for, perhaps hundreds or thousands of people. Maybe you find out that 30% plan to vote for one person and 35% plan to vote for the other.
When you ask only a few hundred people who they will vote for, you have a bigger chance of making a mistake than if you ask thousands. Pollers call that chance of mistake the margin of error. I call it the jiggle factor.
So, the 30% and 35% are jiggly numbers. If you only ask a few hundred people in your poll, those numbers will jiggle a lot. If you ask thousands of people, the numbers won’t jiggle a lot.
With a margin of error, or jiggle factor, of 3 points, the 30% might jiggle as low as 27% or as high as 33%. And, the 35% might jiggle as high as 38% or as low as 32%.
See how we applied the jiggle factor(margin of error) to both numbers? It means that the low number is actually somewhere between 27 and 33 and the high number is actually somewhere between 32 and 38. Notice that those two sets of numbers overlap on 32 and 33. This tells you that the 30% and the 35% are not different from each other and that our two candidates are in a dead heat.
Even though the one number is 5 points larger than the other number, they really are equal to each other. The important part is to apply the jiggle to both numbers.
In the end, the only way to know if they are different is to vote. So vote!
(It’s a lot more complicated than this, but hopefully the general idea is clear.)