The signal and the noise in advertising
The advertising and media industry needs to focus on better modelling if it’s to stand a chance of accurately predicting campaign outcomes, argues Simon Lawson.
If your experience is anything like mine, then your newsfeeds have been overflowing in recent days with advertising’s gurus making their predictions for the year ahead. I’ve noticed them more this year because they’re being made at the same time I’ve been reading Nate Silver’s book – The Signal and the Noise.
For those unfamiliar with Nate Silver, he’s a statistician who came to prominence after correctly predicting the winner in 49 out of 50 states in the 2008 US election and gained further stature after picking all 50 states in the 2012 US election. Prior to his involvement in political forecasting, he used his statistical ability to forecast the performance and changing value of major league baseball players. Think Moneyball.
In his book, the noise is the increasingly overwhelming volume of information in today’s era of ‘Big Data’, while the signal is the meaningful relationships within the data that can best help to make an accurate forecast.
There’s a lot of noise in advertising.
Consider some of the metrics we’re asked to navigate on a regular basis; sales, trial intention, market share, ad recognition, message take-out, brand linkage, net promoter, tarps, impressions, clicks, average weekly reach, average frequency, cookie windows, engagement, participation, views, likes, shares, click through rates, search volumes, affiliate table rankings, share of voice and cost per acquisition, to name a few…
If that’s the noise, what’s the signal?
Back up for a second, if the signal is the data relationships that help to make a forecast accurate; what is the subject of the forecast for which we most need the signal? In my opinion: What is the likely outcome of a campaign? Is it likely to achieve the objectives as briefed? Is it going to work?
We know this … right?
Yes and no. Advertising’s signal is a work in progress, but I believe it lays in the current efforts to improve the effectiveness of econometric and cross-channel attribution modelling. What is the demonstrable path to acquisition using your historical data sets?
Which of the touch points along the path have proven to be the most influential? What effect has your product’s position in market had on past results? What is the value of superior creative?
Finding the signal amongst the noise has to be one of the most critical, if not the most critical issue facing marketers and agencies today. Advertising options have exploded in number over the last 5 years and our ability to optimise them to achieve our objectives can appear to be struggling to keep up with an increasingly complex system.
Nate Silver writing about the period following the spread of the printing press through Europe in the 15th century:
“Meanwhile, exposure to so many ideas was producing mass confusion. The amount of information was increasing much more rapidly than our understanding of what to do with it, or our ability to differentiate the useful information from the mistruths.
“Paradoxically, the result of having so much more shared knowledge was increasing isolation along national and religious lines. The instinctual shortcut that we take when we have ‘too much information’ is to engage with it selectively, picking out the parts we like and ignoring the remainder, making allies with those who have made the same choices and enemies of the rest.”
Ok, so maybe it’s not as severe as that, but the parallels with the advertising world today appear fairly obvious. I’m reminded of the battles that regularly take place on Mumbrella between the social media disciples and the less ardent believers. Some divisions also still remain between online and offline, along with media and creative. It’s making it harder; increasing the noise.
Accurate and actionable modelling may be our best chance to find the missing signal.
It’s a way for us to rise above the noise and get closer to an objective analysis of the communications tactics that are best contributing to our objectives, to recommend an optimum weighting between them, and to forecast advertising outcomes.
It should be said that Nate does point out examples of where modelling has failed rather spectacularly: The 28% default of AAA-rated Collaterised Debt Obligations (CDOs) during the global financial crisis when only 0.12% were forecast to default. But he also points to the successes: Meteorologists have significantly improved our ability to forecast the weather over the last 30 years or so.
The book cautions us to the dangers of over-confidence and asks us to avoid falling into the trap of making definitive predictions: “You are likely to sell 1,345 vehicles this month”. Instead, a probablistic approach, to better reflect the uncertainty inherent in forecasting, is suggested: “With this plan, you have a 45% chance of selling 1,200-1,300 vehicles this month, a 30% chance of 1,300-1,450 and a 15% chance of selling <1,200.”
There are clear barriers to a world where advertising outcomes can be better forecast.
Today’s models are far from perfect, and the need to improve and update the assumptions they’re based on is constant. With an ever changing communications landscape, our models need to be built in a way that leaves room to test innovation and the new tactics that come with it. The cost of modelling can be high, and the talent mix in agencies needs to continue changing to better reflect the growing importance of advanced statistical skills. It won’t be easy, but it will be worth it.
Back to those predictions for the year ahead: I did find one that I like. Writing in Advertising Age, one commentator predicted that 2013 will be the year that agencies marry the creatives and the quants. Let’s hope he’s right.
- Simon Lawson is a business director and communications strategist at PHD Melbourne. Follow him on Twitter: @simonislawson
Just to endorse Simon’s thoughts – I also spent Christmas reading The Signal and the Noise.
I really recommend it for anyone working in the industry.
One thing I learned is to be careful about making predictions. But I will predict that in the same way if you were working in the industry a decade ago you’d have been embarrassed to admit not having read the Tipping Point , The Signal and the Noise is going to quickly take a similar place in agency conversations.
Cheers,
Tim – Mumbrella
Perhaps there aren’t any people within agencies who can create this model and forecast…
User ID not verified.
A marriage between creatives and quants….won’t somebody think of the children!
User ID not verified.
Nice post. It’s a few years old now but ‘The Black Swan’ by Nassim Nicholas Taleb is another great read on probability, the dangers of overestimating our predictive ability and how/why we fool ourselves into creating ‘illusory correlations’ and fall into statistical bias (which is particularly relevant to the marketing industry!)
User ID not verified.
… and never confuse correlation with causality.
User ID not verified.
Agree with Daniel’s comment. Doing this properly requires a far greater/different skillsets and resources than exist in traditional agencies. The link also needs to be broken between those that do the modelling and subsequent planning recommendations and the rebate gravy train that underwrites the existence of most media agencies.
…and yes John, never confuse correlation with causality.
User ID not verified.
JG as usual you’ve hit the nail on the head here. I totally agree we must be diligent when we look at any models so as not to boldly proclaim that cigarette lighters cause lung cancer.
We need to do a better job looking beyond individual campaign data and using our increasingly available ‘big data’ for working out which inputs are the real drivers of campaign performance and which inputs are merely passengers along for the ride. The field of econometric modelling is getting a lot better at doing that but many of the media models I have seen to date are very raw with little predictive ability. Until our media models start incorporating more granular inputs (e.g. reliable measures of creative strength plus reach and frequency measures for all forms of media as a minimum) then their value will be remain somewhat limited.
User ID not verified.
Does anyone at PHD have a PHD?
User ID not verified.
Quick disclaimer I haven’t read Nate Silver’s book and I am not a statistician, however I have a few that work for me. We undertake media modelling. The usual suspects
– Advertising effectiveness
– Future level of demand
– Importance of price
The first two are focused on prediction the last on causality. There are big issues around undertaking all of them. As already outlined the quality of the inputs are as crucial if not more so than the quality of the model itself.
With this kind of (media) modelling we also have to deal with the huge amounts of inter correlations between factors – lots happening at the same time. There is also the effect of a variable lag on both the explanators and predicators.
There also seems to be an element of art as well as science (often hidden in a black box) in the subjectivity around weighting the importance of certain factors.
To John’s point and although it may seem obvious getting someone other than the statistician to ’eye’ or ‘sense’ check the model, ideally with some understanding of what is being modelled, is invaluable.
Although I am biased this is best left to the professionals – and yes that isn’t advertising or media agencies. This isn’t a sales pitch I’m anonymous, more of a warning, I have seen outputs posing as econometric modelling from agencies and some of it (not all) was frightening.
User ID not verified.
Me again. Apologies, I also I haven’t read Silver’s book, but I have kept a close eye on his work and progress.
I also note with interest that the bedrock of his work and successful political predictions is re-analysis of what is primarily ‘traditional’ research polling. That is, the underlying research ‘data’ is not fundamentally different, but his re-analysis technique takes into account the underlying (deliberate?) biases of the US political polling to ‘recalibrate’ that data to reflect what occurs at the four-yearly polls.
As an example of what this means using local data you will notice that the political polls vary here in Australia (sometimes wildly) between the major pollsters (Morgan, Newspoll, Nielsen and Galaxy). They all use different methods, questionnaires, modes, timing and weighting to ostensibly arrive at the ‘same’ result. But even within a polling organisation there are/can be underlying differences. For example, if you look at Morgan’s polls over the past year and a half (around 60 of them), ‘mode’ appears to be a significant variable. On average, their phone polls produce an LNP result almost 3 points higher than their face-to-face polls – same questions, different answer. However, with this knowledge you can take into account questionnaire mode and produce a much more reliable prediction (as well as learning which mode returns results closer to the actual election results and reading too much into trend without understanding the mode). They key is that while both sets of data are subtly different they are BOTH usable. When you extend this across all the major pollsters you clesarly get a better picture.
Researcher echoes the point of inter-correlations in marketing analyses. The better models are generally parsimonious – that is they tend to ‘sacrifice’ slightly higher ‘fit’ to keep only the truly significant variables by discarding the auto-correlated variables. My rule of thumb is if you have more than six explanatory variables have another look at your model. Consider using a single ‘relationship variable’ rather than multiple variables. For example, you may be able to use your brand’s price relative to the market average or relative to your key competitor, rather than loading in all the prices.
Also, look for non-linearity. Very few relationships in real markets are linear (e.g. price elasticity) so if your model has lots of linear relationships and straight-line graphs in it, then that should be a warning sign.
Also, be wary of models that claim to predict “96% of all sales movements” and if they do, look for a variable called ‘unexplained’. If your model can explain sales movements 70+% of the time you’re doing well. Don’t forget that we’re dealing with real people and real consumers out there and they change their minds and buying habits all the time.
Penultimately, test and re-test your models. And when you’ve ‘got it right’ throw some more data at it and test it again and again. Withholding data from the model or doing ‘split’ models is one way of checking predictabailty.
Finally, then ignore all the stats and make sure it passes the “does this make sense” test. If it does, then show it to the client.
User ID not verified.
uggg pls refrain from using “causality”.
correlation =\= causation is the preferred expression!
User ID not verified.
Great review, and I am glad to know other people in the Advertising business are paying attention to Silver’s book — and to the topic in general. At our agency, which historically has been mostly about mass advertising, we now have a 40-person department working on all things statistical. Over the last several years they have made some impressive contributions to the work. The creative work, even.
Here is my review on the book, also an Ad Biz POV: http://admajoremblog.blogspot......noise.html
Thanks!
User ID not verified.
+1 to Mike’s comment from 9 Jan.
And Taleb’s new book highlights the folly of almost all prediction.
Whoever cracks applying Bayesian statistics to advertising outcomes will be a multi-millionaire.
Personally, I don’t think it can be reliably or robustly done.
But open to being proven wrong.
User ID not verified.
Nate Silver shows that prediction is very difficult. Many predictions are totally unreliable. Professional forecasters know their limits.
Yet brand equity firms tell us they can predict a brand’s future from just a few simple survey questions. How gullible do they think we are?
And marketing mix modellers tell us they can predict a brand’s future from just a few (now historic) correlations. How gullible do they think we are?
User ID not verified.
There’s a cynical old definition of an economist as “someone who can predict the past with 50% accuracy’. Seems about right …
User ID not verified.
Byron, while I respect your cynicism I have seen numerous econometric models which have surprising success doing just that. Any marketing mix modeller who says that they can predict EVERY brand in EVERY scenario with absolute surety is clearly a snake-oil salesman. But to dismiss every model for the sake of the few (or even the many) is equally disingenous.
I suspect a lot hinges on the definition of a brand’s “future”. My rough rule of thumb is “six for one”. If you want six months forecast I will want three years of (at least weekly) data. And in my opinion there are few brands whose historical data older than three years is still relevant.
There are various techniques which can test the predictive capabality of the model. Let’s say you have three years (156 weeks) of time-series data. Build the ‘complete’ model on that data set. Then remove the 26 most recent weeks and build the model on the remaining 130 weeks of data. First, the two models should look similar – similar brand drivers at similar levels. If not, you simply have noise and tell the client that. Second, you check the predictions from the n=130 against the actuals from the n=156. Again, if there is not an acceptable fit then tell the client that – at least they now know that their brand is either unprecitable, its category is unpredictable or they don’t have the data that underlies the key brand drivers (if indeed they do exist).
As your own book How Brands Grow shows, brands (in the main) are built on long term marketing efforts, which is consistent with the underlying premise of such modelling and data sets.
Cheers.
User ID not verified.