Features

Are digital metrics bullshit?: Mumbrella360 video

In this light-hearted lunchtime debate from the Mumbrella360 conference, Mark Ritson and Louise Barrett pit themselves against Ashley Ringrose and Dan Monheit to debate the legitimacy of digital metrics. Tune in to find out who was crowned the ultimate winner.

The debate about digital metrics at Mumbrella360 was so full, people had to be turned away at the door. It opened with Ritson declaring “I measured my penis this morning” and only got better from there. 

Ritson centres his argument around an unlikely appendage

In the video below, which documents the whole debate, Ritson makes an entertaining case for the motion that digital metrics are over-complex, unreliable and just plain bullshit.

Joining Ritson in the case against digital metrics is Louise Barrett, GM of network sales at News Corp, who argues that digital is a space without true independent measurement, and therefore isn’t to be trusted.

Arguing against the motion that digital metrics are bullshit is Soap Creative co-founder Ashley Ringrose and Hardhat strategy director Dan Monheit.

Monheit argues that traditional media’s metrics are full of far more BS than digital, and Ringrose makes the point that instead of playing dumb to the world of digital metrics, we instead need to arm ourselves with knowledge.

The debate is moderated by Nikki Clarkson, head of marketing and communication at Southern Cross Austereo.

Speakers’ transcript

Affirmative: marketing professor Mark Ritson

I measured my penis this morning. I do that a lot. I’m staying at the Sheraton on the Park over there. I measure my penis and I have a nice little leather tape measure I carry around with me. I got the tape measure out – it’s a good tape measure and it gives me a good consistent metric. I take two measures. I measure the length from the tip to the base, and I take the girth as well, because we all know that’s important too. So I take those two measures and I put them in a little notepad that I take around with me. And I don’t measure it every day, but I measure it most days. And I sat there and I looked at the measures and I though “Hmm”.

So I rang my mate Charlie Murdoch. Charlie’s also a cock measurer too. I rang Charlie and I said “Hey mate, you been measuring your cock this morning?” and he said: “You read my mind mate.” I said “well where are you?” he said “I’m just coming into work” I said: “Well swing by the Sheraton because I want you to check this for me.” So he came around, we had a cappuccino, got the tape measures out, and just double checked each other’s figures.

I think in that little story you learn a lot about me, about Charlie, about our cocks, but also, about what good metrics should do. Good metrics should be reliable, they should give you the same level. If I start getting a different length for my penis because the metric is wrong, it’s going to send me all kinds of the wrong kind of signals, right?

I think you don’t need too many measures. I think I could measure a thousand other things related to my genitalia, skin circumference and all these other things, temperature, and I could measure it all day long, I could have a gauge all day long there in my pants giving me constant readouts on my iPhone, but would it really help me measure it any better than I’m already doing with my tape measure in the morning?

Finally, it has to be comparable. I have to be able to ring up Charlie, we have to compare dick sizes, and then we have to measure each other’s dicks. It has to be something where we’re open to measuring and checking. I think that’s a good metric for metrics if you will.

If you take those three criteria and you look at digital metrics, I think you can see where currently they are filled with bullshit. They’re not internally consistent, and we should pick on Facebook and Google here specifically, partly because they’re the only two companies that matter. They’re doing two thirds of all the media spend and 100% of the growth. In Facebook’s case in the last 12 months, let me very quickly give you a list of all of their errors just so we’re clear what we’re talking about:

In September, it was announced that the average time spent watching video had been overstated between 60 – 80%. In early November, organic reach for posts was overstated 30 – 50%. Page posts organic reach appearing on screen was oversold by 20%.

In late November, overstated time spent reading Instant Articles by 8-9%. In December, [Facebook] overstated its referrals metric by more than 30%. December overstated mobile graph API numbers – whatever they are – by up to 71%.

Reaction data incorrectly measured 25% overstated. Didn’t measure iPhone traffic for Instant Articles correctly. Mischarged advertisers this month for clicks on video carousel ads. Understated videos watched at 100% length, the figure was actually 30% more than Facebook was saying.

Now it’s interesting there, there’s ten errors, of which seven or eight are overstated and one understates, which is an interesting balance there to start with. Why are there more overstatements than understatements, when it should just be purely random? I think there’s too many errors, I don’t think it’s consistent.

If you don’t believe me, we can ask the World Federation of Advertisers. That’s the big boys and girls club of the biggest advertisers in the world, the 50 CMOs with the most media spend in the world, combined they spend more than AUD $100 billion per year on advertising. That’s five times more than we spend in the whole of this country. They are the 50 biggest CMOs of the 50 biggest companies.

Of that group, 62% last month agreed that digital media metrics were not satisfactory. Now, that’s not saying it’s “bullshit”, but it’s as close as Americans get to saying bullshit in public. Almost two thirds of them are not happy, even though they’re spending billions of dollars, with the way digital is currently being measured. So if you don’t trust me, trust them.

I think the next explanation of why it’s full of bullshit explains a lot of these mistakes. Carolyn Everson is the very impressive vice president of marketing, when she uncovered one of these mistakes she said “look, no-one likes mistakes, we’ll now go back and we’ll look at all 220 of the metrics we use to measure Facebook advertising to see if there are any more errors, which there were. 220 is too many, and I think the digital industry – and you see examples of it at Mumbrella – mistake measuring lots with measuring well.

I’ve been lucky enough to work with some very famous, very bright economists in America, one step away from a Nobel Prize, and many of them use the same word when we talk about very advanced quantitative research. They talk about elegant data. “It’s a very elegant piece of data”. Elegant and simple. Less is more when it comes to measurement. 220 measurements in these screens and dashboards and overuse of data don’t help us understand, they actually stop us from understanding, as anyone who works in programmatic will tell you.

So – too many metrics, and yet too many metrics that we cannot actually compare. Where’s the chart where we see how YouTube is performing in simple metrics vs Facebook video? I still look for these and I miss them, right? We’re still not able to do simple, straightforward comparisons.

Where’s the ability to compare how a 30 second TV spot on Channel 7 compares with the digital video on Channel 7? These digital views don’t make any sense to me. I have a PHD in marketing and it doesn’t make any sense to me or to any other clients, and it’s okay to admit it. It’s not you that’s stupid, these digital metrics are bullshit.

And again if you don’t trust me, trust Mark Pritchard, the chief marketing officer of Proctor & Gamble. He said last January: “We spend enormous amounts of time trying to understand, analyse and explain the differences between Facebook, Instagram, Twitter, Snapchat, Pinterest, Pandora, YouTube and the dozens of different viewability standards claimed to be right metric for each platform.”

The analogy he uses is an American football game, where every different team in the NFL has different rules, has different measurements, has different pitch arrangements, and then trying to get a game to work. It’s deliberately obfuscating so you can’t actually work out what the hell is going on, which is bullshit.

And finally we have the issue of walled gardens. We have the issue of why it’s not possible for someone else to independently measure this stuff. It’s one thing to have third party verification. It’s quite another to have someone else doing the measuring, someone else doing the homework. We shouldn’t be relying on these two giant companies because we “trust them”, particularly when they keep making mistakes. Neilson isn’t perfect, OzTAM isn’t ideal, but it’s good enough and it’s fit for purpose, and there’s no incentive for Nielsen to overstate Seven vs Nine.

There is enormous incentive for Facebook and Google to change, alter, obfuscate their data for their own interests. If I was running those companies I would do the same thing. It’s bullshit, and I find it deeply troubling and deeply fucking ironic that Google, a company whose mission broadly stated is to share information around the world with everyone, won’t fucking let anyone get in there and properly measure their data. Strange.

So, what can we conclude from my session? We conclude I think that digital metrics are bullshit. They have too many errors. They have too many measures which confuse and make everything difficult. They have too little comparative metrics that enable us as marketers to make better choices. And they are too bereft of external verifications. And last and most importantly: I’ve got a twelve inch penis.

Against: Dan Monheit, director of strategy and owner at Hardhat Digital

PHD – pretty huge dick right? In this room full of overachievers, because why else would you be at Mumbrella360, you guys all know that making decisions without data, or worse still, based on data that is full of bullshit, is short changing ourselves, the brands we represent, and the companies that pay us to make great decisions on their behalf. Which is why today’s debate is so important. Because if as an industry we are interested in improving our performance based on metrics, and if those metrics are full of bullshit, we might as well pack this whole thing up now and go home.

So come with me as we investigate, interrogate and explore, and as we do what we do in the digital world: we get to the facts and we see what’s there.

Now I should say from the outset – we know there will be issues. How could there not be issues pulling together, analysing insights from trillions of data points collected every second of every second of every day? I mean just last week our own beloved outdoor media association pulling together data from just 31 sources managed to overstate monthly growth by 80%. So despite all the technology and the apps and the bots, this is clearly still an industry filled with humans. And where there a humans, there will be a degree of human error. Unlike the humans working in TV, radio, print, and outdoor, the humans working in digital have had all about a decade to get our house in order. Which is about as long as it’s taken for digital to move from obscure side act to the only thing people without grey pubes actually care about. Less than a decade.

You guys know when the first TV ad ran, right? It was July 1 1941. I’m sure you remember it Mark, it was for Bulova Watches, it ran before a Dodgers Philly baseball game, it cost $9 to run the ad. So you guys know what that means right? It means we’re just a couple of weeks away from celebrating the 76th anniversary of running ads on TV without a fucking clue if anybody’s seen them or not. But I digress.

Today we’re here to prove to you that despite what Jerry Harvey and the captain of the Titanic here have to say, digital metrics are not full of bullshit and they are in fact the greatest gift that marketers of our generation could have asked for. First up, I’ll be looking at the alternatives. We cannot evaluate the quality of the integrity of digital metrics without looking at the alternative metrics we get from TV, radio, print and outdoor. It’s going to be fun and upsetting for a lot of you. Once we’ve shattered some dreams there, my co-pilot Ash Ringrose is going to be rounding out the definition of digital metrics. See, Ash is a nice guy and he just wants to make sure that when crazy old man Ritson goes bonkers calling things full of bullshit he actually understands what things he is referring to.

So let’s begin, and where better to begin that with TV, right? Because it feels to me like understanding what drives the metrics, the ratings, and ultimately the price behind AUD $3 billion worth of TV advertising here in this country would be a pretty important thing for us all to understand. Personally I was interested to see where 76 years of progress has landed us in terms of TV metrics reporting. I was not disappointed. TV media is metrics are measured and managed by the completely independent organisation OzTAM. OzTAM will tell you on their own website they are so independent that they are wholly owned by Australia’s three commercial networks, network seven, network nine, and network ten. No vested interest in improving metrics at all. But it gets better.

So, where do they actually get the information of what is being watched in nine million Australian households? Well, I’m glad you asked. You see, OzTAM employ a merry band of telemarketers who interrupt mealtimes across this country looking for about 5,000 truly representative households who are not on the do not call registry, home and available to take a call, happy enough or lonely enough to actually talk to a telemarketer, and are willing to have a random box placed in their living room which records everything they watch. And there’s more.

The way these boxes actually know who’s watching TV is by the resonance and guests within that house manually telling OzTAM every time they leave the room using some sort of magical OzTAM remote control. That’s right guys, the data driving $3 billion of advertising spend are metrics from an audience that is not randomly sampled and self reporting. This is from people who will talk to telemarketers, let strangers into their house, and have no incentive one way or another monetary or otherwise to provide accurate information. In fact, they have an incentive to provide inaccurate information because it is way easier. Who can be bothered telling the magical OzTAM remote every time they get up to take a piss?

For some context, there was a wonderful study published in 2013 in the International Journal of Infectious Diseases, and this looked at the behaviour of almost 500 people who has suffered respiratory infections and their behaviours around taking their antibiotics. It found, shockingly, that 70% of these guys missed some of their tablets, they forgot to take some of their tablets, leaving an adherence rate of just 30% for something as life critical as taking antibiotics after a serious respiratory infection. So honestly, what adherence rate can we expect from our telemarketing-friendly families, when it comes to telling the OzTAM remote that they’re off to make a cuppa? This is total fucking bullshit.

In case you were wondering, this exact same combination of representative sampling and self reporting failed to predict both Trump and Brexit within just the last 12 months. Turns out when you ask people things, sometimes they forget and sometimes they just fucking lie. And as embarrassing and archaic as what I’ve just described is, our TV rating system is not only the self-proclaimed gold standard, but it’s actually the darling of Australia’s TV advertising world. At least the OzTAM remotes are electronic.

Let’s spare a thought for the almost 12,00 participants out there completing radio surveys in 2017. These guys take with nothing more than good intentions a paper diary, ballpoint pens and fucking stickers, these poor guys and gals are required to manually fill out this paper diary every 15 minutes for an entire week. That’s over 400 manual entries per participant. And what is even crazier than taking out your workbook in rush hour traffic with no incentive is that the industry openly acknowledges that this is a memory test, people are asked what they remember listening to. I don’t even remember my kid’s birthdays! These metrics are driving $1 billion of advertising spending in this country, what the fuck is going on here? Total fucking bullshit.

Outdoor media is calculated using yet another world class methodology, we really are the lucky country. It’s a methodology called move and this revolutionary system is determining the rates behind $700 million spend in outdoor, built entirely around a concept called “likely to see.” Nothing says rock solid fucking metric like “likely to see”. In the list of factors that determine likeliness to see include something like proximity of a billboard to the kerb, which side of the road it’s on, and all sorts of other wonderful things. But how do we even know if the people we’ve paid for who are likely to see our billboard ever even saw it? We don’t, because their are no metrics. So if there are no metrics and we can’t prove things one way or the other, then it’s wonderful because we don’t have to worry about such annoying things such as rebates or recourse for all of those people we paid for who were “likely to see” but never actually saw it.

To round it up, I would love to talk about how readership and circulation were calculated in print, but since major publishers Bauer Media, Pacific magazines and News Corp all threw their toys out of the pram last year and withdrew from the audited media association of Australia’s Independent reports, it is really anybody’s guess. Why, you might ask? They will tell you they were not happy with the metrics being reported in the independent audit. Total fucking bullshit.

Ladies and gentlemen, clearly digital metrics are not without issues, but at least let’s be sensible and mature enough to realise that the issues are to do with our own shortcomings, interpreting the oceans of data that is now raining down upon us. This is light years away from the bullshit cocktail of apathy, arrogance, and deceit that sits at the core of metrics in every traditional channel and has done so for decades. To wrap up, it is insane to put digital metrics on trial when by comparison, traditional metrics don’t even exist. When it comes to traditional metrics, we’re all victims of habituation. We’ve been sniffing the bush around it for so long, we don’t even notice the smell any more.

Affirmative: Louise Barrett, executive general manager, Newscorp national sales

I didn’t realise we were here to debate all the other metrics? I know they’re not all equal. I think the most important fact that I’m going to make – and no, I’m not going to be funny, I’m not going to be hilarious – but I’m going to send a really important message.

We know that all metrics aren’t equal and I think the key that I want to make here is that third party, independent measurement is key. So let’s just quickly diverge, because all other media owners disagree with their vision systems at times, we know that, there were plenty of times at Channel 10 when we thought the ratings were better than they actually reported, let me tell you.

But as a media owner, we knew the value of an independent industry measure. We might have disagreed with it, but we supported it. The problem we had in digital was that media owners recognised the need for independent measures, but tech companies like Google and Facebook pick up the ball and go home when they don’t like it.

Digital is the new kid on the block, it has the biggest house, so people question and ask: “How the hell did that happen?” Actually, it’s not just a house, it’s an entire block of all different sizes and dimensions, and that then makes it hard to measure. The industry is incredibly complicated, and the complexity of the digital value chain is not one size fits all. This means independent measurement is critical.

The Interactive Advertising Bureau (IAB) actually delivers that independent measurement system. It’s independent, and it’s transparent. Industry data is inclusive and reports on all properties in an equal manner, regardless of their involvement with the IAB. That’s to provide the industry with comparable data on all players.

It’s a transparent system that tracks over 6,500 media organisations, brands and channels. It’s reviewed by the IAB’s measurement council, which has representatives in 17 media organisations and the Media Federation of Australia (MFA). The IAB requires Nielsen to undergo annual independent audits as well. But as with all industries, you’ve got companies that are willing to invest, companies like News Corp, companies like Fairfax, like Yahoo! 7 – premium content producers who are committed and happy to have independent metrics. Companies that are comfortable standing by their numbers. So let’s take News Corp. We believe measurement is important, and we don’t mark our own homework.

Others, however – back to Google and Facebook – are cowboys. The problem with our industry is you’ve got all sorts of players, and we all end up getting thrown into the same bucket. Then you have a case where all digital ends up hurting, because people are scared. It’s a bit unfair actually, because there’s some of us in the industry who deliver time and time again, and more importantly, deliver for Australians by providing professionally generated content. The biggest issue we face as an industry is trust. The fact is, that Facebook and Google are the ones that people don’t trust, because they measure their own homework, and they measure it badly.

People have talked about Facebook before, and their issues of reporting, they’ve had no less than four statements of ill reporting, and they continue to avoid being transparent by limiting third party measurements of campaigns run by their platform. Their metrics are bullshit. They’re not playing by the rules, why don’t they stand up and start using industry standard data?

Now, let’s talk about Google. Do you know where your ads are running? Next to content that someone else created. Content that can’t deliver you with brand safety. It’s like all care and no responsibility, we’ll take your money, but who knows where your ads are going to go? Google don’t see themselves as a media company, they see themselves as a tech company, so they don’t care about the quality of the content, just the quantity.

Advertisers need to find the right partners, they need to ask the right questions, they need to talk to the companies that are playing by the rules, the companies that care about content and how it’s measured. The right partners will deliver for them, because they deliver audiences for themselves every day through professionally generated content, so you know where your ad’s appearing, and it’s appearing in a brand safe environment.

News Corp, as you would know, is a member of the IAB, as are many companies, and we’re committed to the digital industry, we believe in what we do, and we can back up what we do with independent data. So my message is stay close to your agencies, and stay close to the metrics. We’re working hard with the IAB to improve our measurement and the launch of daily digital content rating shortly will provide more detail granular, comparable data across digital publishers. Agencies should be working hard to be as transparent as possible to their clients in terms of what and whom they’re delivering.

My final point is that all digital measurements should focus on helping marketers achieve effective, and efficient outcomes. Make sure you know what digital metrics are bullshit. The IAB metrics aren’t. So if you care about independent, digital metrics, then as an advertiser, or people who are booking ads for agencies, you need to support those of us who do play by the IAB rules, and who do support the industry standard. Currently, your investment decisions say you don’t care about independent measurement, because you’re investing in tech companies that don’t provide it. These are the guys with the bullshit metrics.

Against: Ashleigh Ringrose, Soap Creative co-founder 

I can see why Mark and colleagues and you guys think digital is bullshit, I really do. It’s because it’s complicated. There’s a lot of numbers going around. I’ve been doing this since 1999, and every day I’m still learning. It’s evolving, it’s expanding, it’s changing daily. Then you’ve got people like Mark, who are like the climate change deniers of the marketing industry.

We’ve all seen his type: they’re loud, they’re well-spoken, they’re opinionated, and they’re smart, but they’re also ignorant to any views, and information that doesn’t confirm their world view. They revel in controversy, they love the sound of their own voices. Why? Well, because if it bleeds, it leads. Having a moderate discussion doesn’t get anyone excited. But I’m not going to deny the kinds of points that Mark has brought up over the last couple of years – and try and guess which one I made up, because they’re all full of hyperbole. There are some major points that we need to shed some light on.

So where to start? I’ll start with non-human traffic, or, attack of the bots. It’s a scary thing. Mark loves sharing this graph, which shows that from the world federation of 50 old CMO guys, 88 – 98% of clicks on these channels were bots, that’s outrageous. However, I actually read the report, all seven pages of it. I don’t think Mark read the report, because he gave credit to the wrong people for the report. It wasn’t the World Federation of Advertisers who suggested that, it was just in their compendium.

How much do reckon they spent writing this report? £100. Ahh. Would you write up a $2-300 billion dollar company because someone spent less that what you’d spent on a round of drinks? I call bullshit so much that I actually paid for them to run the research again with my own money. I used Facebook because it was the most familiar to me, and I spent $1,200 of my own money writing the study. The result? 24%. That’s not great, but it’s not 94%. I asked him: is this usual? And he said, yeah, that’s about usual. There’s hundreds of pieces of data out there saying anywhere between two to 98% adbot traffic. So which one is it? It’s a bit of a riddle. I don’t know.

The thing is, ad fraud is real, and I’m not denying that. Ad fraud is in every single medium, the thing is, with this, you can detect it. If they can detect the bots, you can too. That’s the power of digital. There are companies that can help you detect it. Some of them specialise in it, some of them are actually here. You guys, the marketers, the ones spending all the money, you guys need to start asking harder questions, you need to start actually being smarter about how you spend your money, because with digital you can.

There are answers to these issues. And if a publisher doesn’t want to run with these guys and won’t authenticate the spend, don’t work with them. These guys will help you. Bots could be 2%, they could be 98%, it seems no-one knows, but you have the power to learn for yourself because you can detect it. It doesn’t seem like bullshit to me it seems you just have to work a lot smarter, you can’t just spend your money blindly and hope for the best. No, you have to go a bit harder.

Facebook used to count a video view as three seconds. Now it’s ten. Facebook use five seconds for their true videos, after five seconds you get charged. Five seconds? That’s not enough time. I love this research from an eyetracking study, almost half – 40% – of all press ads are viewed for one second. What’s going on? That’s bullshit. 60% of all press that you’re buying aren’t even viewed for one second! That’s bullshit. That’s because people’s attention spans are getting shorter. We’re so busy arguing over three seconds, one second, ten seconds, we’ve forgotten about what we actually want to do which is to sell to people. We have to adapt.

Everyone’s saying how Facebook do it without sound: “Sound! Everyone needs sound.” But actually, the world is changing, and it’s dominated by mobile phones, and these platforms are adapting the usage to people’s lives, which are dominated by mobile phones. So many have adapted their content to their channels, and so should we, because attention spans are getting shorter.

The world is consuming a lot of video content in silence. They’re viewing it in portrait. If you went to a cinematographer and said “we want to shoot this thing in portrait”, they’d kill you. No-one wants to shoot in portrait, but that’s how people are consuming it. It’s not bullshit, it’s human behaviour. Don’t fight it, adapt.

But what if five or ten seconds don’t help you? I don’t get about ten seconds, I want to know how many people watched till the end of my ad – you can! You can drill down and you actually have the real data there. With all other media, it’s guesswork. How many people watch it 75% of the way through on your TV? I don’t know. I guess, 22%?

Let’s talk about digital more than just social media. And I know – it’s not marketing, it’s not digital, all we’re talking about is social – it’s the tip. Just talking about a few errors on Facebook of Google? Digital is more than just marketing. If you’re using digital channels as just a marketing tool, then you’re not using it to its full potential.

What is the most unsexy digital channel you can think of? Email. The death of email. Poor email, it’s been dead for ten years according to this. It’s not just unsexy, but now we’re going to say email’s full of bullshit? Don’t tell that to Red Planet, a spin off from Quantus, who are using 27 years of frequent flier data to make loads of really strong campaigns that get results. Tell them it’s all bullshit. Tell that to Woolies, tell that to Coles, who are spending millions investing in data and metrics to make their campaigns better.

We spend $23 billion on online retail alone. Online retail has grown really quickly. How? By using behavioural data. Not paper diaries, not sticker sheets, but real behavioural data. Take Showpo, she’s grown her company into a $25 million yearly run rate, and there’s no rules, she probably didn’t even have a business plan. She probably doesn’t even have an MBA.

So I’m going to put out a proverbial olive branch to my fellow debaters and say, you know, take off the bullshit-coloured glasses. Just take them off. Don’t be scared, and have a think about what we all need to do, we all need to make a decision. It’s time for everyone in this room to make a choice. We can either throw out the digital metrics, and stick with what we’ve got, or we can embrace digital, and challenge ourselves to arm ourselves with knowledge and not defend ourselves with ignorance.

ADVERTISEMENT

SUBSCRIBE

Sign up to our free daily update to get the latest in media and marketing