Stop using vanity metrics to measure behaviour change

In this guest post, Hugh Stephens argues why campaigns like Dumb Ways To Die should avoid using ‘vanity metrics’ to measure behaviour change.

Dear marketers of Australia,

A polite request: please stop launching behaviour change campaigns. You don’t know much, if anything, about measuring the nature and quality of behaviour change coming out of your campaigns.

Did you know there is extensive research about how public awareness campaigns change behaviour? This work is not undertaken by marketers or publicists. It is undertaken by academics, and focused on complex behaviours such as the prescribing habits of doctors, drug adherence in HIV patients, or smoking cessation.

They don’t spend their time evaluating ‘reach’, ‘views’, ‘impressions’, and their studies usually extend several years to accurately determine the impact of any kind of  intervention.

So when you produce your next cross-integrated-multi-platform viral video campaign instructing the public to change behaviour, please remember that ROI is not measured in reach, impressions, click-throughs, eCommerce purchases or Klout. It is measured in actual behaviour change.

Let’s look at the recently launched Metro Trains’ ‘Dumb Ways to Die’ campaign which is getting considerable buzz. Yay, exciting ‘viral video’. Yay, tumblr. Yay, microsite. And yay, 11.9, views (at time of writing). I’ve seen a lot of people share it, discuss how catchy the music is. It looks great, I’m sure probably win some awards. Props to McCann and the team.

The campaign aims to curb preventable train-related deaths by creating ”a lasting understanding that you shouldn’t take risks around trains, that the prospect of death or serious injury is ever-present.” But will it change the way people act around trains? I doubt it. In that moment when a consumer is bopping along to their iPhone, not realizing that they are standing too close to the edge of the platform, are they suddenly going to think about this ad?

I welcome McCann to respond to this question: have they undertaken a decent- sample-sized pre and post surveying of an appropriate population to determine to statistical significance if the campaign has changed attitudes toward the issues they’re targeting? Did they use a third-party research organization to ensure transparency? What methods did they use?

Let’s have a look at the issue you’re trying to reduce here: rail deaths. According to the TrackSafe Foundation, 186 people died on the railways nationally in 2011. Of those, it is estimated 150 took their own lives. The remaining 36 died when trespassing or at a level crossing.

Let’s talk ROI. Let’s talk cost-benefit. Are we going to see a reduction in the 36 people who die nationally as a result of trespassing or at a level crossing? I may have to eat my words, but I doubt it – regardless of this ad. If the objective of the campaign is to reduce rail deaths, it would seem far more appropriate to prevent rail suicide.

Why can’t we put money into the issue that seems to cause far more deaths: suicide? It’s not sexy, it is less targeted at ‘the masses’, and the current campaign could arguably only trivialize the idea of taking your own life rather than direct people to appropriate services, but what’s wrong with targeting this population?

The most successful behavior change campaigns are highly targeted at the population at most risk of the issue. ‘Dumb Ways To Die’ does not. Not at all. Neither is its impact going to be measured beyond vanity metrics like “virality”.

Monash University’s extensive research report highlights the factors relating to successful behavior change in mass media campaigns in road safety. The findings reveal important learnings for public awareness campaigns, including:

  • Campaigns with a persuasive orientation and those that use emotional rather than rational appeals tend to have a greater effect on the relevant measure of effect. In contrast, information-based and educative campaigns have been associated with less effective campaigns.
  • The use of explicit theoretical models and prior qualitative or quantitative research to inform the development of road safety mass media campaign messages and execution has been found to increase the effectiveness of campaigns.
  • The use of public relations and associated publicity appears to be more important to the outcome of the campaign than the use of enforcement. However, the combination of public relations and enforcement as supporting activities shows particularly large effects.

Monash University also recommends that future research examine the longer-term effects of mass media campaigns, rather than the standard analysis undertaken during or immediately following the completion of the campaign.

Remember this, ad world. There is strong evidence showing how you create a change in behavior. And accurately and meaningfully measuring the impact of a campaign is something that involves time, money and expertise, not just your Facebook Insights and a pretty infograph pasted into a report three weeks after the campaign concludes.

Hugh Stephens is the director of Dialogue Consulting, a specialist social media


Get the latest media and marketing industry news (and views) direct to your inbox.

Sign up to the free Mumbrella newsletter now.



Sign up to our free daily update to get the latest in media and marketing.