‘A lot of the measurements we were using weren’t giving us the results we wanted’: Doing Data the Netflix Way

The journey to data nirvana involves a challenge between man and machine, says Netflix’s Satya Kunapuli.

Kunapuli, was speaking at the Doing Data the Netflix Way event in Sydney where he pointed out that, despite all the advancements in adtech and martech, marketers are still largely using old-fashioned ways of measuring advertising effectiveness.

Netflix’s House of Cards

The clash between Netflix’s data scientists and the company’s marketers was one of the reasons for the company embarking on its journey, Kunapuli told the audience at the Interactive Minds event.

“A lot of the measurements and techniques we were using weren’t giving us the results we wanted and fundamentally we knew these weren’t the right measurements as we’re a data-driven company.

“We had the benefit of having some very strong points of view from our product folks. The constant challenge from their perspective was that marketing was all fluff and there were was no concrete metric they have like they had from their data.

“It lead us to rethink how we should approach measurement and what techniques and measures we should use,” Kunapul, who leads Netflix’s paid media marketing analytics team, explained.

For Netflix, managing the scale of a business that extends to 90 countries, offering 45 billion viewable hours per year to 100m members worldwide means the company is always looking at how to automate its operations.

A problem for Netflix’s management is the sheer number of campaigns is difficult to manage manually and making decisions at the speed and scale of the business is almost impossible.

“A lot of the Netflix marketing was ‘CPM based’, with today’s diversity and scale, CPM doesn’t work any more,” Kunapul said.

Kunapul breaks his data journey into four steps ranging from traditional marketing at the most basic end through analytics, predictive models, A/B testing and applying incremental-based algorithms.

Traditional marketing is pure correlation in Kunapul’s view and in many cases there’s latent demand which isn’t affected by marketing.

“There may be an 8% click through rate, but 5% was going to happen anyway.”

Kunapul’s point is there are many variables that might affect a data point that have little, if anything, to do with marketing or advertising and all too often campaigns get too much credit.

Analytics is the next stage of the journey with Kunapul emphasising the word ‘thoughtful’ as the company correlates masses of data that allows allows predictive models to be built.

Those model allow analysts to strip out incremental factors such as seasons and other campaigns, and cites an example doing a deeper dive on a campaign found video was 10 times better than rich media rather than the 3.5 factor originally claimed.

A/B testing remains a powerful tool for Netflix with Kunapul illustrating an example which found advertising on weather sites in New York didn’t change subscriptions regardless of how bad the conditions were, despite the prevailing wisdom and superficial data indicating it did.

While A/B testing is effective, it’s time intensive and impossible to carry out at the volumes Netfllix demands leading to the company to increasingly use incremental algorithmic methods to test advertising.

“This whole thing has to happen at ridiculously high scale”, says Kunapul. This leads to the philosophy of ‘explore and exploit’ where at any given time you’re always testing but something is successful, you take advantage of it quickly.

Ultimately Kunapul sees sharing knowledge and working together as necessary for marketers in a world awash with data: “This kind of information sharing is going to float all boats equally, at the end of the day as marketers, as agencies, as platforms, the more we all collaborate, the better our return.”


Get the latest media and marketing industry news (and views) direct to your inbox.

Sign up to the free Mumbrella newsletter now.



Sign up to our free daily update to get the latest in media and marketing.