Tag Archives: big data

The economics of revelation by Susan HayesCulleton #ESOMAR #MRX 

Live blogged from #ESOMAR in Dublin, Ireland. Any errors or bad jokes are my own.


  •  The human mind always makes progress but it is progress in spirals
  • does how we pay affect GDP? cash, cheque, credit. this was answered by going to market researchers and it affects capital expenditures, government policy, innovation spending, central bank decisions, taxation externalaties, trade balance, security spending, real estate requirments, bookkeeping productivity, marginal propensity to consume
  • If you want to know th road ahead, ask those coming back
  • what is thought leadership?  
  • training – what will happen in the future
  • consulting – 
  • bloggers – they have immense power, inch wide but a mile deep, people turn here for news not traditional media [oooo, could that be me?]
  • retention – becoming bigger, companies want to hold nice events but they want thought leaders in their industry to be there
  • leverage – send people something of value that doesn’t cost a thing, use anniversaires of events as reminders, don’t make it a sales pitch, just make it information  [bingo word :)]
  • marketing – companies realize they can’t be left behind in the content area, maybe don’t have the resources to do it, partner with outside companies to produce niche content
  • ecosystem – people issue more white papers as lead generation [except it’s kind of overdone now, everyone wants your email address before you can read it :/ ]
  • business models – businesses that are simply thought leadership
  • speaking – people want to hear about what’s five years from now
  • People are often scared by big data, you feel like you should use it all but why can’t you just pick up what’s relevant and synthesize that part, we’re on the cusp of big data becoming huge
  • companies are turning to market researchers for help with big data [actually, no they aren’t. they’re turning to big data companies because market researchers have their head in the sand]
  • all the young people have turned to snapchat and facebook is only used for stalking
  • big data turns into new models, new language, business insights, marketable commodity, content management, market valuations, informal mentoring, new language like impressions and click through rates
  • why is facebook valued more richly than a company like toyota or disney, we want to know what’s happening RIGHT NOW, we want to get in on the conversation, we are watching the hashtag for esomar
  • RonR – return on relatonship [i thought that meant research on research, yeah, that’s better]
  • 1 – number of impressions
  • 2 – click through rate of the company
  • 3 – conversion rate – how many email address will be left once you click through to the landing page – for 1 email address i need 1000 impressions
  • 4 – how much will this cost
  • 5 – first time purchasers are more likely to look online for researchers, count them as two
  • 6 – for investment services, people turn to internet for reassurance, fount them as more then two
  • top 3 divided by bottom 3 – created a proprietary indicator, this gives her the x-factor dealing with customers
  • work ON your business, not IN your business while at congress

Big Data and the Dawn of Algorithms in Everything by Dr. Morten Middelfart, Social Quant, Inc. #ISC2015 #MRX

MRALive blogged from the 2015 MRA Insights & Strategies Conference, June 3-5, 2015 in San Diego. Any errors or bad jokes are my own.

  • has been a professional programmer since the age of 14
  • Wrote “Calm – computer aided leadership and management”
  • Level 1 – access to information; Level 2 – speed; LEvel 3 – autonomy of removing human from equation
  • Maersk is the biggest container shipping company in the world, they could tell their client exactly where their container was to the millisecond in the 90s. Then they wanted to tell you where an item within the container is but there was too much data to go through and easily process.
  • built software that used menu driven programming in 2002
  • level 2 was speed from 2005 to 2014 – access to data is now a commodity
  • what is excellence in management and leadership and how can we apply computing to do more and better like that
  • John Boyd is the father at the Top Gun school – concept called OODA loop – loop of observation, orientation, decision, action. Found a way to make planes a lot faster.
  • what tools can we used in each of these areas, how do we communicate each of these
  • orientation is analytics, simulation, data mining
  • observation is dashboards, reporting, agents
  • action is storyboard, communication, workflow
  • decision is search for additional knowledge
  • is there data we don’t’ want to analyze? why are you collecting data if you aren’t analyzing it
  • analytics and reporting must be completely integrated
  • each team in an organization must cycle these loops
  • OODA loop includes strategic, tactical and operational levels
  • the stealth bomber can’t be run by people because it is far too fast, if the computer breaks, the plane crashes, you get manoeuvrability that is impossible with a human pilot, it is essentially invisible. Human decision making isn’t fast enough
  • if you shut down your power for two months, which organizations would survive? we can’t go back to pen and paper anymore. We are already in the seat of the stealth bomber.
  • [i always think, in ten years from now we’ll look back at our innovative technologies and think they’re hilariously old]
  • now we have a lot of external data that is free, 90% of world’s data has been created in the last 2 years. any small percentage of the data is garbage, but the amount of relevant data is outside your company. What is its availability, quality?
  • if you clean garbage data, its still garbage data.
  • Kevin Slavin – How algorithms shape our world, a TED talk – talks about locations of stock companies so that they are physically closer to the cables to save milliseconds of processing time

  • why do you need a dashboard? let the computer tell you if when something is wrong [hell yeah!]
  • if you aren’t using algorithms, assume all of your competitors are. it’s not scifi. it is mainstream media.
  • how can you use computing to accelerate ourselves, not replacing, doing it differently.
  • SocialQuant startup – don’t ask what you want, have computer tell you what you want,
  • Eli Pariser: beware online filter bubbles – TED talk, there’s a risk that whenever you ask a computer to show you something, it will always give you junk food, maybe it needs to show you things that aren’t what you want

  • do you trust technology so much you’d bet your life on it – cars, planes, trains – can you trust google to show you the world? are ten results the truth? we generally believe it
  • do we trust facebook to take what is relevant about all our friends and put it into one stream
  • only 1 in 4 people are helped by cancer drugs, only 25% get a life prolonging effect, chemo by default is not a good choice
  • when its a confined game, rules are set, computer will win every time

Scotiabank – Big Data, Small Brains: Making Effective Decisions in a World of Data Overload by Lisa Ritchie #MRIA15 #MRX

MRIA15,MRIA2015Live blogged from the 2015 MRIA National Conference in Toronto. Any errors or bad jokes are my own.

Scotiabank – Big Data, Small Brains: Making Effective Decisions in a World of Data Overload
Lisa Ritchie, Scotiabank

  • Started department with 2 people many years ago and now there are about 180 people – data warehouse, data support, campaign execution, analytics and insights, brand and communication research
  • Major changes in big data – market research was the core, the catalyst. Companies started getting repositories, more information became relevant. It morphed and became bigger. Reliance on information became bigger.
  • Quant is rational, concise, part of the why; Qual is exploratory of the why, more emotive
  • Big data – is there such a thing as little data, little information? Vendors made this thing sound so much bigger. Technology gives us information at a faster pace.  The notion of big data came from technology vendors.
  • The information was always there, we’re just getting it faster. It is blinding. The big brain comes into play here.
  • Big data is now at your fingertips. You know how many times people have walked into your branch or made a savings deposit or what channel they use to deposit.
  • Were able to understand that their bank wasn’t doing well with young customers.  Had to figure that out – how to attract young customers to a bank, any bank. This information at their fingertips helped. Needed to see that people open an account because their parents opened it, and then they go off to college. Now, it is the best at attracting young people.
  • Canadians are the highest users of loyalty programs.
  • Biggest key to success is going back to basics and creating a structure. Data can give you any answer so research needs discipline.
  • Can you tell a powerful story with big data? It’s different for structured and unstructured data. Piecemeal doesn’t work. Some analytics people are brilliant at the data but can’t tell a story. And vice versa.
  • Secret to integration – need to hypothesis, synthesize, know what you’re looking for. It’s not about looking for one thing
  • Most data has 8 to 10 years of history. You can go back.
  • Don’t lose sight that a project from ten years ago might reveal new insights.
  • The problem is knowing what to do with the big data. Need to learn how to ask questions to use that big data. “Yeah, yeah, yeah, but what I meant was…”
  • Researchers need to be integrators, be proactive. Need to be real consultants. This is where we’re losing business – to people who ARE consultants.
  • Journey will be fast and furious. The googles of the world are using data, and we need to be intuitive as well.
  • She just got an email “Welcome to Metcan” but she’s been a customer for 10 years. They just sent a generic email even though their big data could tell them very different.
  • Learn the skill of synthesizing. Think forward.
  • Suppliers need to work with clients in a tighter relationship. Bring learning from elsewhere. This is what consultants do.
  • It’s not just presentations. It’s communicating and interpreting and suppliers can help do this.
  • She says her success has been luck [You make your own luck!]
  • 330 terabytes of data [wish my laptop had that!]
  • There is a push for everyone to have access to data. Need to make sure data is anonymous. [READ ONLY! READ ONLY!]
  • When everyone has access to data, interpretation becomes really key.
  • Don’t archive things so fast – what’s old is new again. Don’t underestimate the usefulness of old research.
  • Does technology mean faster and cheaper – no. It’s a myth. Asking questions takes time.

Brand Building in a Digital, Social and Mobile Age Joel Rubinson, Rubinson Partners Inc. #NetGain2015 #MRX

Netgain 2015Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.

Brand Building in a Digital, Social and Mobile Age

Joel Rubinson, President and Founder of Rubinson Partners Inc.

  • Picture of brand success has to change
  • We are no longer in a push word, consumers pull information at their leisure
  • We engage in shopping behaviours even when we aren’t really shopping, we are always IN the path to purchase
  • Brands must become media
  • Starbucks is the best example of a marketer that gets it. 40 million fans on facebook. millions of website visits. millions have downloaded their app. Every interaction generates data they can use, can be used for personalization, to amplify brand communications. They no longer have to pay for every message.
  • The rise of math experts in advertising  – lift from using math to place advertising is a repeatable success
  • Programmatic messaging is key. Think about impressions that are served up one user at a time. marketers goal is serve the most relevant ad at the right price. And this needs to scale.
  • Research is missing in action when it comes to math – we lack digital metrics, still rely on survey based tracking, we have a post-mortem mind set, we are failing to change how marketing works
  • We must get serious about integrating digital – why isn’t this happening, why are we locked in a survey world
  • Our comfort zone is surveys. We know how to construct 20 minute surveys. Our learning zone is the mobile area where we unpack our surveys into smaller pieces.
  • The panic zone is digital, we don’t understand it. We must move digital into the comfort zone.
  • lets start by just looking at the data, look at page views, look at themes in social media, how big is your brand audience, how many likes on facebook, how many twitter followers, how many newsletter signups. These are unambiguous measures. Look at clicking and sharing and conversions.
  • Stop treating social media as a hobby, not specialty projects, not ancillary thing to look at. You must find ways to increase positive word of mouth.
  • Do we really need feedback from consumers every single day on attributes they never consider? Can’t social media which is much more organic do this?
  • Bring in data that you can’t get from a survey that has action value. Some online panel companies already use a social login called OAuth.  Append all the Facebook data to your survey and use it for targeting.
  • Data aggregators have lots of profiling information for targeting ads throughout the web which means different people get different ads based on cookies from their browser
  • You can also link in frequent shopper data to your survey data.
  • You don’t have to guess whether an ad is working. You can run an experiment and serve the ad to one group of people and see the change in group behaviour.
  • MR needs to know that brand meaning is done completely different now. People seek out knowledge, digital delivers information in real time. But marketing research hasn’t changed.
  • Think digital and do something big. Shift some money into datascience or integration. Conduct in the moment research with smartphones.

Emerging Technologies – Are They Still Emerging? Lenny Murphy, GreenBook Blog and GRIT Report #NetGain2015 #MRX

Netgain 2015Live blogging from the Net Gain 2015 conference in Toronto, Canada. Any errors or bad jokes are my own.

Emerging Technologies – Are They Still Emerging?

Lenny Murphy, Editor-in-Chief of GreenBook Blog and GRIT Report

  • Attitudinal, behavioural, and intrinsic data
  • Foundational research is no longer taking months but hours
  • Moving from questioning to discussing, from asking to observing, from data to insight, from understanding to predicting, from the big survey to data streams, from rational to behavioural, from quarterly to real time, from siloed to converged
  • the traditional survey as the primary driver of information will decline
  • Data science is not a hoity toity term for a statistician. It’s information technology and algorithms and languages and hadoop and R. It’s statistics on steroids.
  • The future looks very different.
  • Over the next five years, we are in the realm of DIY, non-conscious measurement is emerging such as facial scanning and automated emotion measuring, automation and AI in terms of very very smart devices, internet of things where all of your things will collect and share data from your shoes to your car, virtual and augmented reality will change our media habits
  • DIY – there are many free DIY tools
  • The ‘make it’ revolution – consumers can ‘print’ their own things, print some shoes, do an ideation session using a printer. cost of these devices can be as low as $100.
  • Emotional measurement – facial scanning, shopping behaviour videos, eye tracking
  • AI – tons of money going here, google has spent millions on quantum computers, these will just be part of everything we do
  • Internet of Things – Internet as we know it might disappear. Daily lives are just always all connected. e.g., Microsoft’s hololense.
  • Do a virtual shopping experience without a computer. But you still feel like you are in the store.
  • Imagine a connect fridge [will it shop for me once it notices I’m out of BREAD AND MILK!!]
  • Google Glass succeeded in every aspect they hoped. The real product will come out in the next couple of years.
  • Gamification has never taken hold but many companies are working in this area. Game to map out neurons.
  • Which companies will be our competitors for clients and budget?  Google, IBM, Apple, facebook, AOL, Verizon, Comcast, Disney, at&t, GE, groupm, WPP, amazon

The Internet of Annie #MRX #IOT

I did it. Yes. I broke down and spent my Christmas money. Let’s put aside the fact that I still get Christmas money from the moms and move on to what I spent it on.

In just six to eight weeks, this pretty little plum coloured Fitbit will arrive at my door. (The “make it pink so girls will buy it” marketing scheme works on me but plum is just as good.)

2015/01/img_0065.pngSupposedly, it will monitor my heart rate all the time including when I am awake and asleep. It would have been cool to have it a few weeks ago when my four wisdom teeth were ripped out of my face but I’m sure some other quite unpleasant event will greet me soon enough.

I’m quite looking forward to learning:
– how consistent my sleep is, and how many times I wake up at night
– whether my heart rate speeds up or slows down when I get ready for work or leave work, or when I go toy awesomely fun ukulele class
– how incredibly nuts my heart rate is when I speak at conferences, show up at cocktail hour, plow through a crowded exhibit hall. Though I may seem calm and relaxed, it really takes a ton of mind games to turn quiet me into loud me.

And at the same time, I’ll be wondering… If someone gets their hands on my data, what will they do with it? What products will they develop as they learn about me? What heart rate medications will they need to sell to me? What fitness products will they need to sell to me? Will I need to buy the shirt version to measure electrical outputs? The sock version to measure sweat outputs? The earbud version to measure brainwaves? What will marketers and brand managers learn about me and my lifestyle?

Now that I think about it, this is MY form of gamification. I can’t wait to see charts, watch trends, and compare Norms. And now that I’m learning Python and rstats, I would love to get my hands on the dataset of millions of people and millions more records. With permission of course.

Big Data and Privacy: The Legal Landscape Affecting Corporate Research by Shannon Harmon, JHC #CRC2014 #MRX

CRC_brochure2013Live blogging from the Corporate Researchers Conference in Chicago. Any errors or bad jokes are my own.

Big Data & Privacy: The Legal Landscape Affecting Corporate Research by Shannon Harmon, JHC

  • our lives are a series of data points
  • more opportunity vulnerability and the potential for greater abuse
  • smaller entity might purchase data from 3rd party
  • who owns the data, who has the right to access the data, what steps are taken to keep it secure
  • goal of any regulation is to protect personally identifiable information form breach and misuse
  • you can identify people with very little information so keep in mind a lot of information is PII
  • Notice and consent: need to provide notice of how the data will be used, and then obtain consent – this is the core of the law related to privacy, you need to make sure the right practices were followed to do this
  • Where do we look for oversight? Right now, state attorney general, FTC, FCC, FDA
  • Fair information practice principle – only collect what you need to collect and only retain it for as long as is necessary to fulfill the specified purpose
  • FIPP – data quality and integrity – organizations should ensure that the PII is accurate, relevant, timely and complete and this is difficult if you’ve purchased the data, supplier should have a structure in place to ensure this
  • Consumer privacy protection bill of rights – google search this – things corporations should do to protect privacy, this area will become increasingly more regulated so think ahead
  • Fair Credit Reporting Act – example of what big data protection framework should look like, right to review your credit report and make sure it’s accurate and get it fixed if it’s not correct, this is where we’re headed, your digital dossier is being collected and you don’t know how decisions about you are being made, you can’t contest your big data points… right now
  • special considerations for health data – apple has stated that any app developers cannot use any of the health data for advertising, or data-mining except to help an individual manage their health or for medical research. but is apple responsible for developer compliance? what if a data broker got the data from someone who wasn’t supposed to have it in the first place?
  • considerations for researchers
    • where is the data being obtained, what are the sources
    • what practices are being used to obtain it and what is your confidence in your aggregator
    • how is the data being trained to arrive at conclusions? what algorithms? what human manipulation?
    • think about the vendor/subcontractor relationship, is the contractor independent? a substandard contractor impacts you
  • we need
    • use restrictions – can’t use big data to discriminate on age, race, etc
    • oversight – protect against unregulated digital dossiers
  • be intimately knowledgeable about your company’s data gathering practices – informed consent, opt-out, internal user access controls
  • be ready to evolve as the law is only beginning to be developed in this area

Neuro to Big Data to Segmentation: Multi-mode wins #ESOMAR #MRX

esomarLive blogging from #ESOMAR Congress 2014 in Nice, France. Any errors or bad jokes are my own.

Car Clinics 3.0: Designing better cars by peering into the consumers brains by Fatima El-khatib, haystack International, Belgium, Ronny Pauwels, Toyota Motor Europe, Belgium, Wim Hamaekers, haystack International, Belgium

  • Something didn’t feel right about a car they were test driving, but they didn’t quite know what
  • How do you measure the unconscious? Combine qual quant and something new, neuromarketing
  • Customers don’t say what they do and they don’t do what they say. So why ask them everything.
  • Protypes were highly confidential so couldn’t use them. Had to use older available materials.
  • EEG captures long term engagement and relevance, based on avoidance and approach theory
  • Lab test showed computer generated images, 5 views of the exterior, 8 views of the interior, film was about 3 minutes
  • People liked the wheels of one vehicle but not much else. For the other vehicle, everything was fine and average.
  • Because of biometric results, focused on the specific positive and negative features
  • Verbal results shows little differences between the vehicles but EEG showed one vehicle had much more positive feelings. Could see the specific details that people were not able to express verbally.
  • What about asking people about the fabric and dashboard ornaments.
  • VW Polo and Hyundai ornaments performed well but Citroen and Peugeot 208 performed awful based on Galvanic Skin Response.
  • Consumers have difficulty expressing everything verbally. Overall engagement doesn’t matter, it’s all the individual elements that matter. Even the tiniest details of a car can have a huge effect.
  • Neuro is now an official tool for Toyota. They look at the same business questions from different angles. It helps to optimize the car development process.
  • Neuro is not the holy grail – multi-mode is the holy grail. You still need experimental research designs.
  • Be brave, be daring, use the new techniques and see if/which ones add value.

Communication Analytics: Effectiveness Research for Conversion Based Campaign Planning:  How to measure effects of (offline) campaigns on web visits sales and conversion by Erik Prins, Validators, Netherlands
Iris van Dam, Validators, Netherlands, Martin Leeflang, Validators, Netherlands, Sander Pot, Ticketveiling, Netherlands

  • “Moneyball” with Brad Pitt is all about big data. Baseball is all statistics. Used all the statistics to put together an unlikely team that came second place in the end. Cost per player was $250 000 when other teams paid 2.5 million per player.
  • Can we do money ball in a media campaign.
  • Can you correlate campaigns and web visitors, sales, and conversion. Of course. Can calculate cost per anything – media, shopper, clicks.
  • Know the media schedules by the minute, TV, radio, everything. Know all sales, new and old customers.
  • Import all this data into one platform. Calculate cost per mille – how much to reach one thousand people. Cost per sales, cost per shopper, cost per click.
  • Calculate how many people visited website after commercials over an entire year – It cost 0.25E to get someone to their website for one specific channel. Another channel ended up at 12E per customer.  The time of day matters, midday was so much cheaper.
  • Online is winning in Netherlands because they can measure views and clicks.
  • Outdoor advertising is activating existing customers. For new clients, you need TV and radio. Online media is more expensive
  • For Ticketveiling, the win it midday programming. Outdoor format was highway signs. Radio target was a few very specific channels.  Don’t burst all your funds at once, drip your funds is much cheaper.
  • There is less need for traditional research now, need to shift into research consulting, and clients understand this more.

New Perspectives: How a segmentation provided new ways of looking at consumers thereby unlocking sales potential by Alastair Liptrot, Simplot, Australia, Neale Cotton, The Lab Strategy & Planning, Australia, Paul Labagnara, The Lab Strategy & Planning, Australia, Peter Stuchberry, Nature Research, Australia

  • Start somewhere different if you want to end up somewhere different. Try starting at the end. How will you apply your research in the end?
  • Invest your money in a safe bank or lose it all at a casino. Or invest it in a segmentation [I much prefer the segmentation option 🙂 ]
  • Simplot is products in Australia in the freezer, to chiller, to house
  • Normally small packs, large packs, kiddie packs. Need to look beyond demographics
  • Most don’t have longevity or are only demo based, and may not complement existing tools or data.
  • Had to work with current categories and brands, as well as future brands.
  • developed four pillars – involvement – how much you love cooking 2) health 3) convenience 4) value
  • Decided on 8 segment model.
  • used Nielsen homescan – people who scan all their supermarket purchases. tagged everyone with a segment, used personality, demographics
  • Had to inspire the team to embrace the segmentation. Need to make the people feel a part of it, encourage acceptance and engagement. Had them engage from the very beginning. Include them in naming the segments so they truly understand what the segments are.  Created a game show for the marketing team to better understand the segments and how to use them.
  • Delivered 30 million in revenue for a $250000 investment
  • The project would have gone on the shelf if they hadn’t though about how they would use it in the end

Global Trends in Marketing Research with Murphy, Chadwick, and Poynter #MRIA14 #MRX

Live blogging from the #MRIA national conference in Saskatoon, Saskatchewan. Any errors or bad jokes are my own.saskatoon

Panel Discussion on Global Trends in Marketing Research
Moderator: Greg Rogers, Global Director of Consumer Market Knowledge, Proctor and Gamble


Panelists (via Web Conferencing):
Leonard Murphy, Chief Editor & Principal Consultant at GreenBook, Simon Chadwick, Managing Partner of Cambiar, Ray Poynter, Director of Vision Critical’s University

  • Lenny:  Change is indeed happening has evidenced by trend data; seeing decline in revenues from the largest companies which means those funds are going to other places likely non-traditional places.
  • Simon: The change is technologically enabled, fewer resources but budgets have remained the same, more social media, more predictive analytics, more synthesizing happening. Trackers are being transferred to fully automated systems.
  • [kind of hard to hear, sorry for missing portions]
  • Some tools get a lot of hype and some are gaining traction
  • Three chunks related to big data – been there, done that; what’s upcoming; what the hell is big data
  • Mobile is the future, two types – forced to rethink our traditional techniques because of the device being used to access surveys, responders made this decision not the researchers; type 2 is all the great things we can do with mobile like geo and in-the-moment research
  • Measuring emotion is becoming important, CMOs are really interested in big data but correlations of big data aren’t everything, customers need emotion and empathy and we need to measure this
  • Big data is good around the edges, the margins
  • There is so much data with so much value and we probably won’t be able to solve this for another five years
  • Privacy is becoming more of an issue particularly in Europe and Canada
  • A phone can scan a face, there is an app to scan the facial expression, BeyondVerbal seems to do this
  • Big data will soon be emotion data
  • Technology is precipitating change in the industry
  • Winners will be the big agencies that buy the innovative companies
  • Ray suggests that marketing and market research will merge and we will need to figure this out
  • Two types of innovative companies – challenging, disrupting companies and then peripheral companies that are new companies from the technology side
  • we’re good at analysis but it is different than synthesis, we need to synthesize the stories [oh my goodness, premonition for my pres later today!]
  • Find something to be really good at whether it’s ethnography or something else
  • Lenny: Are you a marketing researcher or are you in the business of helping people?  Use your curiosity to fix things and answer questions. We come across as number crunching accountants.
  • Lenny recommends exporting more poutine from Canada  🙂
  • 17 million people added to middle class every year, like adding France every year
  • Canadian researchers are up there with the best in the world, and large international presence
  • Vision Critical, Hotspex, RIWI innovative companies coming out of Canada
  • Canada is more influential globally than australia
  • Read more of Ray’s thoughts here: http://newmr.org/dialling-in-to-the-mria-conference-the-shape-of-things-to-come/

Other Posts

Enhanced by Zemanta

The Power Of Big Data: How We Predicted the World’s Largest Music Poll Using Social Media by Nick Drewe #CASRO #MRX

Live blogging from the #CASRO tech conference in Chicago. Any errors or bad jokes are my own.

The Power Of Big Data: How We Predicted the World’s Largest Music Poll Using Social Media by Nick Drewe, Creative Technologist, The Data Pack

  • it used to be taboo to say your real name online
  • Nick Drewethere is no line between what is signal and what is noise. what is gold to me is trash to you
  • when you know that hundreds and thousands of people have posted noise about a brand, it’s no longer noise
  • Radio station called TripleJ, a national funded station, like NPR or BBC and it’s aimed for a younger audience. They post the hottest 100 songs as voted by listeners through the year. it’s a national institution. 1.4 million votes cast last year.
  • results used to be closely guarded up until number 1 song
  • station had people share what the voted for in hopes of getting more people to vote. every page was hosted on a unique URL which suggests every vote has a page. other little bits of code with info were on the page too. if they could find and collect enough of these pages, they might be able to predict.
  • used twitter api and found 40 000 votes in a few minutes, a sample size or 3 or 4%
  • created a list that seemed realistic but didn’t know what to do with it yet
  • set up a website where people could see their predictions and play the songs
  • turned the website into a disclaimer, people had to scroll way way way down to get to the number one song
  • got a ton of traffic, more people saw it than people who voted
  • made the front page of one of the biggest newspapers
  • not yet sure how accurate they were yet
  • colleague ran a bootstrap of 3.5% sample and concluded they’d get 90 songs 100% accurate or 95 songs at 90% accurate, and #1 song with 83% accuracy
  • the next year, the station closed all the social sharing features
  • found 400 votes that were posted as screencaps to twitter, their confirmation emails
  • but photos are also posted on instagram, found 20 000 votes there after searching for them
  • even if you really really really don’t want people to share something online, they will do it anyways
  • predicted 82 out of 100 songs in the second year with half the amount of data
  • it was an experiment in social data
  • most networks have free APIs to share and use data, most networks don’t really know what to do with the data
  • posts don’t have to sit in isolation online, we can turn these into insights
  • people don’t post the same things in social media that they post on surveys
  • 60 million posts on instagram every day, rich with metadata, a photo contains geolocation, 20 million photos a day have a location [i always turn off my geolocation, decline, decline, decline]
  • can search on username, hashtag, and location – but it must be part of a hashtag not a description
  • youtube is still the largest music sharing site
  • can use youtube, twitter, facebook data to predict music you will like [try me – rankin family, leahy, michelle branch, vanessa carlton]
  • a single message is rarely valuable but a group of messages is, particularly with all the metadata
  • every link tells you something about the person who shared it – what they like, don’t like, know and don’t know, cat gifs too
  • google’s page rank looks at links to your website, more websites gets you a higher page rank, and greater likelihood to appear in a search result – this is a social graph and can be done on a personal level, not just what they’ve shared about a specific topic but everything else they’re doing
  • [Nick is wearing the same shirt today that is shown in his bio. LOVE that as I find matching people to photos very difficult]
  • everyone should try a social api, it’s not a difficult to use as you think it is, point isn’t to start writing code but to start thinking about big data and social data in a different way


Other Posts

Enhanced by Zemanta
%d bloggers like this: