Stop using vanity metrics to measure behaviour change

In this guest post, Hugh Stephens argues why campaigns like Dumb Ways To Die should avoid using ‘vanity metrics’ to measure behaviour change.

Dear marketers of Australia,

A polite request: please stop launching behaviour change campaigns. You don’t know much, if anything, about measuring the nature and quality of behaviour change coming out of your campaigns.

Did you know there is extensive research about how public awareness campaigns change behaviour? This work is not undertaken by marketers or publicists. It is undertaken by academics, and focused on complex behaviours such as the prescribing habits of doctors, drug adherence in HIV patients, or smoking cessation.

They don’t spend their time evaluating ‘reach’, ‘views’, ‘impressions’, and their studies usually extend several years to accurately determine the impact of any kind of  intervention.

So when you produce your next cross-integrated-multi-platform viral video campaign instructing the public to change behaviour, please remember that ROI is not measured in reach, impressions, click-throughs, eCommerce purchases or Klout. It is measured in actual behaviour change.

Let’s look at the recently launched Metro Trains’ ‘Dumb Ways to Die’ campaign which is getting considerable buzz. Yay, exciting ‘viral video’. Yay, tumblr. Yay, microsite. And yay, 11.9, views (at time of writing). I’ve seen a lot of people share it, discuss how catchy the music is. It looks great, I’m sure probably win some awards. Props to McCann and the team.

The campaign aims to curb preventable train-related deaths by creating ”a lasting understanding that you shouldn’t take risks around trains, that the prospect of death or serious injury is ever-present.” But will it change the way people act around trains? I doubt it. In that moment when a consumer is bopping along to their iPhone, not realizing that they are standing too close to the edge of the platform, are they suddenly going to think about this ad?

I welcome McCann to respond to this question: have they undertaken a decent- sample-sized pre and post surveying of an appropriate population to determine to statistical significance if the campaign has changed attitudes toward the issues they’re targeting? Did they use a third-party research organization to ensure transparency? What methods did they use?

Let’s have a look at the issue you’re trying to reduce here: rail deaths. According to the TrackSafe Foundation, 186 people died on the railways nationally in 2011. Of those, it is estimated 150 took their own lives. The remaining 36 died when trespassing or at a level crossing.

Let’s talk ROI. Let’s talk cost-benefit. Are we going to see a reduction in the 36 people who die nationally as a result of trespassing or at a level crossing? I may have to eat my words, but I doubt it – regardless of this ad. If the objective of the campaign is to reduce rail deaths, it would seem far more appropriate to prevent rail suicide.

Why can’t we put money into the issue that seems to cause far more deaths: suicide? It’s not sexy, it is less targeted at ‘the masses’, and the current campaign could arguably only trivialize the idea of taking your own life rather than direct people to appropriate services, but what’s wrong with targeting this population?

The most successful behavior change campaigns are highly targeted at the population at most risk of the issue. ‘Dumb Ways To Die’ does not. Not at all. Neither is its impact going to be measured beyond vanity metrics like “virality”.

Monash University’s extensive research report highlights the factors relating to successful behavior change in mass media campaigns in road safety. The findings reveal important learnings for public awareness campaigns, including:

  • Campaigns with a persuasive orientation and those that use emotional rather than rational appeals tend to have a greater effect on the relevant measure of effect. In contrast, information-based and educative campaigns have been associated with less effective campaigns.
  • The use of explicit theoretical models and prior qualitative or quantitative research to inform the development of road safety mass media campaign messages and execution has been found to increase the effectiveness of campaigns.
  • The use of public relations and associated publicity appears to be more important to the outcome of the campaign than the use of enforcement. However, the combination of public relations and enforcement as supporting activities shows particularly large effects.

Monash University also recommends that future research examine the longer-term effects of mass media campaigns, rather than the standard analysis undertaken during or immediately following the completion of the campaign.

Remember this, ad world. There is strong evidence showing how you create a change in behavior. And accurately and meaningfully measuring the impact of a campaign is something that involves time, money and expertise, not just your Facebook Insights and a pretty infograph pasted into a report three weeks after the campaign concludes.

Hugh Stephens is the director of Dialogue Consulting, a specialist social media
consultancy

Comments


  1. Shamma
    21 Nov 12
    4:43 pm

  2. This is just crazy. How does the author know what Mccann have and haven’t done? Or Metro? Where is it stated anywhere that they are solely using ‘vanity metrics’. Your smug opening makes you look conceited.

    BTW – many people in the behavioural change industry are aware of how behaviour change is measured in an empirical sense. This may be a shock to you.

    The cynic in me says this is nothing more than a bit of business development from someone with something to sell, using the topicality of Dumb Ways to Die to get some attention.

  3. Hugh Stephens
    21 Nov 12
    4:45 pm

  4. Just including links to the missing references for those who are interested:

    – TrackSafe article: http://www.theage.com.au/victo.....28luk.html
    – Monash University study: http://www.monash.edu.au/miri/.....arc220.pdf

  5. Andrew
    21 Nov 12
    4:59 pm

  6. Agree with Shamma I think this is a crazy bit of writing, how can he sit in judgement about what Mcann have or havent done when he has not been involved in the process at all.

    Way over thought in the way he is trying to understand why even a campaign like this would be developed.

    FYI – pretty sure its PUBLIC awareness not PUBIC awareness campaign that has been developed.

  7. Alex
    21 Nov 12
    5:02 pm

  8. I wonder if before Hugh wrote this piece if you surveyed a significant sample of marketers to empirically show that marketers only use vanity metrics to measure behaviour change? I’m sure if he did he would understand that reach and impression are not crucial metrics for behaviour change campaigns and that academic literature often feeds into the whole process

  9. Essie
    21 Nov 12
    5:12 pm

  10. Call me cynical but I’d say Metro’s tactic is more about branding and creating buzz as opposed to behaviour change.

    And if generating buzz is what they are after, this article is a perfect example of it working…

  11. Meg
    21 Nov 12
    5:20 pm

  12. @Shamma, totally agree, just trying to get a plug by insulting the profession of marketing. Kind of surprised that Mumbrella even published it, let alone tweeted it! Yawn!

  13. Nathan
    21 Nov 12
    5:36 pm

  14. Who is this guy?

    Yeah mate, an agency as big as McCann, a client as cautious Metro (with good reason)… have they done pre-testing?.. what do you think buddy?

    Seriously.

    Sure, ‘only’ 36 people have died in rail accidents.. but what about the many that haven’t died? What about the trauma a train driver goes through when he/she hits someone?

    If we listened to this dude, nothing great would ever get made.

  15. derrick
    21 Nov 12
    5:37 pm

  16. hmmm, i would that the objective of the campaign was to reduce rail deaths THROUGH raising awareness of the dangers of railways and trains

    some valid points in this though

    let the debate commence!

  17. Felix
    21 Nov 12
    5:40 pm

  18. It’s cute that you think this is really just about changing behaviour.

    This ad is doing its real jobs by improving brand perceptions of Metro and giving McCann something to throw at awards shows.

    I agree that the ‘dumb ways to die’ work isn’t likely to cause a great deal of behaviour change. Yet it will be entered in every awards show under the sun as a the second coming of Train Safety Jesus. However, that’s how the industry works – they don’t give awards for boring but effectively targeted ads which had an impact that can only be reliably measured several years later.

    It’s also worth noting that all the long term outcomes stuff is studied – that how you get the numbers you mentioned in your links, but funnily enough, you’re not going to hear about the ongoing shift in behaviour which may be partially correlated with a campaign that ran a couple of years ago. There’s too many other factors to make it economical to study detail, besides which, as you mentioned, it’s not such a big issue anyhow.

  19. David C.
    21 Nov 12
    5:43 pm

  20. How to take a fun little viral clip, over analyze it and suck all of the fun out of it!

    I appreciate the concern bit feel something like this should be taken at face value for what it is.

  21. Hugh Stephens
    21 Nov 12
    5:45 pm

  22. @essie – I agree that one of the purposes (perhaps even the primary?) was likely to be changing the sentiment of the discussions around the metro brand – a simple Twitter search from prior to the campaign shows a lot of negativity about trains running late etc, while now it is about the campaign – so it has certainly been very successful there!

    How long said change in sentiment lasts after the campaign is a different matter though, but will be interesting to watch I think to see the longer-term impact on sentiment.

  23. Nat
    21 Nov 12
    6:13 pm

  24. Sorry Hugh, but while I absolutely welcome a discussion around effective measurement – especially conversations around impact, not just engagement – I’m not sure going after a reputable agency like McCann is the best way to start the discussion, particularly as you seem to be making some pretty wild assumptions around the lack of research, planning and insights they (and their client!) have put into the campaign?

    As far as I’m aware, no one is claiming that the add has yet changed behaviour – just that it’s proving engaging? What’s wrong with that? You’ve got to get people to pay attention and participate before you can get them to do anything else – maybe it’s the first part of a long-term multi-channel campaign? Either way, it seems to be a bit cart-before-horse for you to slam them for shoddy processes and false claims?

    Yes, academics have a great deal to add to debates around effective measurement; yes there’s a big body of evidence around public health campaigns and behaviour change, but to suggest that marketers are incapable of executing meaningful campaigns that change behaviour (which it sounds rather like you’re doing) is plain wrong. Most marketers worth their salt know that no matter whether you’re trying to get someone to buy a chocolate bar, make someone pay attention at level crossings, or even ask for help before taking their own lives, you’re going after a change in behaviour – and there is certainly more than one way to effectively do this, including ways we haven’t even thought of yet!

    There’s some outstanding work being done by agencies both here and overseas, as well as some great new thinking coming from behavior design experts like BJ Fogg, who himself claims that the reason he splits his work 50/50 between industry and academia is so that he can be better at both!

    So by all means, let’s tackle the issue of measurement – it’s a biggie! But perhaps there are better ways to get started.

  25. Caitlin
    21 Nov 12
    6:57 pm

  26. There’s a difference between changing awareness and changing behaviour. They’re sometimes confused, but sometimes the former IS the objective, and hence the vanity metrics and impressions matter.

    It’s all too common for public health campaigns such as this to begin with good intentions, an alarming social statistic, a desired behaviour, an identifiable solution, and a million and one ways it could be achieved.

    But then reality sets in and the influence of steering groups, funding criteria, (un)accessible resources and ‘Oh hey, the Internet’s free’ means interventions can often be boiled down to an attention-grabbing awareness campaign.

    As with any social change, there’s only so much that one can do, as it’s really up to the parts to all do their bit to form the greater and impactful ‘sum’.

    Oh, and thanks for the suicide statistic. The ad, the jingle, the intimation, the satirical treatment of really dumb sh*t are very well executed. But I I feel for the thousands of grieving families and friends out there who are hurting over this video, which insults their loved one’s tragic, and entirely not dumb, death. But that’s a comment for another day.

  27. Pot, kettle, flack
    21 Nov 12
    7:45 pm

  28. Umm aren’t all awareness campaigns aiming for behavior change by reaching the largest possible audience? One feels if his minor consultancy had 12 million views he’d have an entirely different view. This article exists entirely to inflame and self-promote.

  29. Danny McTroll
    21 Nov 12
    9:28 pm

  30. What use is a PhD in social research or a related field and a bucket of peer-reviewed publications when all you actually need is Bachelor of Commerce. Same shit right?

    The butthurt is strong in this thread.

  31. Ben Harris-Roxas
    21 Nov 12
    10:52 pm

  32. I don’t work in advertising or marketing so I don’t pretend to have any expertise in appraising campaign effectiveness. I’m struck that so many of the comments: emphasise the agency’s reputation rather than the nature of the work; and don’t/can’t identify what the campaign’s goals are, which shouldn’t bee too hidden should it?

    If this is regarded as a public health intervention, and I think it should be, isn’t asking what behaviour change it actually brings about reasonable? Particularly given any expenditure needs to be weighed up against alternate interventions, such as making physical design changes to rail accident hotspots, etc. It’s a catchy ad but it’s also an intervention and comparative effectiveness is an issue worth discussing.

  33. Mike
    21 Nov 12
    11:37 pm

  34. Agree with those who have pointed out that the author has no idea of the brief, analysis or where this campaign goes form here.

    Imagine some of the ‘Dumb ways to Die characters, very visible at level crossings and on platforms. This would provide a huge incentive to change behavior in-situ, by either irresponsible behavior or deliberate means. No one wants to be associated with making “dumb” decisions so the message has some power in that regard .

  35. OP
    22 Nov 12
    3:00 am

  36. ‘Dear marketers of Australia, A polite request: please stop launching behaviour change campaigns.’

    Incredible. What an opener. Conceited. Arrogant. Smug. Pathetic.

    Looking at his business bio, it opens with ‘Hugh is a social media expert.’ Incredible. Who calls themselves a ‘social media expert’ without a sense of irony.

    I hate to be negative, and will stand back from ugly nouns so this comment isn’t blocked but wow, Hugh. You need to get over yourself; grow up a little.

  37. Circling sharks
    22 Nov 12
    6:46 am

  38. I’ll put it as simply as I can (in my view) – Dumb Ways to Die is an award winner, not a life saver.

    End of story.

  39. Cognitively DIssonant
    22 Nov 12
    10:57 am

  40. Oops. So many angry ad people, so little bandwidth.

    You poked advertising with a fact stick, Hugh. Dumb way to die #6 from memory.

  41. John Ham
    22 Nov 12
    1:02 pm

  42. There are many examples of advertising changing social behaviour. The TAC campaign, when it first ran, had a huge impact, lowering deaths by something like 30-40%.

  43. Oli
    22 Nov 12
    1:17 pm

  44. This article is great. Well said.

    All these comments continue to depict exactly what is wrong with ad-land at the moment.

  45. Dave
    22 Nov 12
    2:00 pm

  46. Let me guess Hugh. You pitched for this work and didn’t get it.

    How conceited to assume that just because something has become popular and viral that there is no strategy or research behind it.

    If you know anything about behaviour change, you would know that the first step is in raising awareness of a particular issue, which this campaign has truly done so far. You do not know what research has gone into it, and I seriously doubt that money would have been placed in a strong campaign such as this one without being backed up by research.

    You also do not know what the entire campaign strategy is. What is going on around the bells and whistles that you have seen in the viral element.

    Before coming out and attacking a campaign, maybe do some research first, speak to some people and ask some questions from the people you are attacking.

    Attacking an organisation and a company in a public forum, without facts to back you up and then asking them to prove you wrong is idiotic and shows your lack of understanding of the social media space.

  47. Caitlin
    22 Nov 12
    2:22 pm

  48. @John Ham Done well, of course advertising can have an impact. Done well.

    But to compare Dumb Ways to Die with TAC campaigns is to compare apples and oranges. TAC ads are short, sharp and shocking, and they depict as much reality as possible. The people getting hit by cars and the people doing the hitting are relatebale because they are you and I.

    TAC ads are not infectiously cute, 3 minute long, entertaining videos that feature jelly beans and take 2 minutes 20 to get to the message. I don’t know anyone in ad land (and I’ve watched Gruen, like, heaps) that would call taking-2-minutes-and-20-seconds-to-get-to-the-message truly good advertising?

  49. Matt
    22 Nov 12
    2:23 pm

  50. Would I be right in saying the purpose of your story is to ‘change behaviour’ of marketers? To encourage them to base campaigns on available research and measure ‘real’ campaign effectiveness? They’re good ideas for an article and certainly worth debating.

    It’s just a pity that your potential to have people listen to your view has been reduced because of your seemed lack of understanding of the topic in question, changing behaviour. Evidence… nobody listens to someone they have just been insulted by.

  51. CS
    22 Nov 12
    3:07 pm

  52. What an absolutely brilliant and refreshing observation and really well said. Hugh – you have hit on a sore spot because I suspect there are many, many people out in ad land that sold their soul long ago and know what they are doing is absolute fluff.

  53. GC
    22 Nov 12
    3:15 pm

  54. Wasn’t this all about getting Metro Trains out of the @metrotrainspr parody gutter?

    By that yard stick I think they did pretty good.

  55. Rob
    22 Nov 12
    3:24 pm

  56. Did I just read that TAC ads reflect reality? Seriously?

    Let’s call a spade a spade, they do not reflect reality in all cases. For example, the early wipe of 5km/h ad – the reality of the impact wasn’t enough “impact” – so they OK’d manufacturing some additional “impact” by chaining a vehicle down. Reality wasn’t real enough!

    Let’s not forget the debacles related to their motorcycle PSA’s – there was a substantial lack of reality in that lot and pandered to common misconceptions that vilified the riding community. The “the ride” ad had quite a few clangers in it, culminating in a completely implausible fall and crash into an oncoming 4WD. It had riders throwing rotten tomatoes at their own TV screens!

    And the more recent motorcycle reconstruction ad? …filled with flaws for the sake of delivering a saccharin, unrealistic, “on message” message, not in keeping with physics at all, or motorcycling reality, or reality in general given that the driver was at fault but yet the rider was blamed. Riders turned off in droves. “…nobody listens to someone they have just been insulted by.” Very true in this case!

    @John Ham, which TAC ad lead to a 40% decrease in fatality stats? That wouldn’t be 0.05 related ads would it?? Wasn’t this the first time ever that the public were given a proper education into the reality of drinking and driving? It caused some self reflection and realisation that alcohol impairment is a real accident causal factor. In conjunction with some pretty stiff penalties, the status quo was doomed. A drover’s dog could have scripted an ad on this topic that would have had a similar effect.

    TAC have done some good with their behaviour changing ads, but as they’ve strayed from the low hanging fruit like BAC, their metrics for success have become dominated by market metrics like recall, rather than aknowledged message delivery and behaviour change. This article could have been written about the TAC.

  57. payedharddonegood
    22 Nov 12
    4:07 pm

  58. I played this to my 10 year old son and asked him what he thought it was about. He replied “Being safe around trains Dad” delivered with a fair note of condescension at the fact that I even asked. Seems pretty effective to me.

  59. Peter Mountford
    22 Nov 12
    4:40 pm

  60. Why is everyone so concerned with the effectiveness of this ad? At least it is good. There are so many terrible and ineffective campaigns being churned out every day that get no reaction at all.

  61. Mark Davie
    22 Nov 12
    6:05 pm

  62. One thing that does annoy me, is people that count views or Facebook likes instead of sales. Some brands don’t even ask you to buy their product anymore, instead like them on Facebook.

    However, the fact that we’re talking about this makes it undoubtedly effective… and it has all happened off zero media budget. I believe you’re right, this might not have a huge effect on our local, “small number” of accidental deaths. But maybe they’re going to run the last 15sec on TV locally. And they still have 100% of their media budget (whatever it is) to do that. At the rate this is going viral it might actually reduce the number of train related deaths world wide. Not only that, Metro have a re-branded and likeable image.

    You started out saying that there is extensive research on the topic and that it takes several years to accurately determine the impact of any kind of intervention, however you’ve attacked them 1 week after their campaign started.

  63. I didn't enjoy reading this
    22 Nov 12
    6:15 pm

  64. At this point in time, how else can they measure it Hugh? How many lives do you think one ad can save in five days?

    And let’s not forget Metro’s job is to run trains, not public safety campaigns. I’m pretty sure they didn’t have to do this.

    If it only saves one life, then it was worth doing.

  65. oh dear
    22 Nov 12
    8:22 pm

  66. a pretty transparent and sad grab for attention, this.

    you’ve slammed the existence of the campaign based entirely on a series of assumptions that you’ve pulled out of thin air.

    you don’t know how the campaign was developed.
    you don’t know how it is actually going to be rolled out longer term.
    you don’t know what other elements will make it up.
    you don’t know exactly what metro hopes to achieve with it.
    you don’t really know much about it at all.

    in short, you are making the ridiculously spurious assumption that they just chucked a video out onto the interwebs and figured that’d change the world. I suspect that’s not quite how it’s going down.

    btw, I seriously doubt the agency would respond to your piece, given both its enormous flaws and the lack of respect you’ve afforded both them and their client. Maybe build some kind of reputation before demanding others meet your lofty standards, yes?

  67. Um
    22 Nov 12
    9:44 pm

  68. If they’re standing too close to the edge of the platform, bopping along to their iphone, listening to this ad, that they bought from itunes, then yes, they will think about this ad.

  69. Andrew Smith
    23 Nov 12
    3:28 am

  70. I agree to an extent, evaluation based upon too much indirect causal linking while avoiding direct feedback and analysis from those who know, i.e. target audience.

    Monash University is a good example, like other universities which spend $millions annually onshore/offshore on marketing and promotional campaigns to attract candidates while analysis is limited to (often) unrelated variables, short term variables.

    However, the only valid and reliable analysis can be conducted through asking candidates who actually enrol, and at least internationally everyone knows which channel most candidates emerge through , “word of mouth”, but that result is not very popular for professional marketing and communications personnel…..

  71. Liam
    23 Nov 12
    6:01 am

  72. Having worked on a safety campaign before, I can give you exactly how these campaigns are analysed:

    1) Number of people in related injuries before campaign
    2) Number of people in related injured after campaign

    If 2<1, then we did well :)

  73. Adam Ferrier
    23 Nov 12
    8:37 am

  74. The hubris and nievity of the commentator aside there is a reasonable (and obvious) point being made. That is, There is a massive gap between how marketers prove cause and effect compared to science / academia.

    That said I think both disciplines could learn from the other.

  75. Nigel Smith
    23 Nov 12
    10:03 am

  76. From what I have read, this viral video is the first step in a CAMPAIGN, which will include other non Social Media aspects. Does everyone forget that bit of info as they launch into lauding or lambasting this video?

    As a piece of viral work its amazingly successful. As a behavioral change campaign, it’s the first step.

    So many comments and articles seem to forget that marketing communications campaigns are made up of many parts. It’s great to see something generating so much interest and discussion, but why not suspend judgement until we actually see the campaign in it’s entirety, and some results?

  77. Jo
    23 Nov 12
    10:33 am

  78. Hmm, I find this is a little condescending and confusing. Behavioral change is a very big undertaking and so as marketeers we need to create incremental measurements with which to identify uptake.

    If the campaign managers are smart they will continue to check in on their greater mission on a regular basis and track progress against research and what the boffins are measuring too. If you ‘reach’ lots of people you have more change of changing behavior then not. If you ‘reach’ people and ‘engage’ them even more so.

    Where did you read that the campaign team were not going to conduct research into the long term effects of this campaign? I am sure this is part of a long term strategy which I would suspect you haven’t been involved in……..

  79. Caitlin
    23 Nov 12
    12:28 pm

  80. For those questioning whether it’s even possible to measure behaviour change – Yes we can!

    As the suicide data shows, unless the campaign targets the causes of the accidental fatalities (that is the 20% that weren’t suicide attempts) such as depression prevention and treatment interventions (heck, what about fences to completely prevent people reaching the tracks?), this campaign may not greatly reduce fatalities.

    However if they’re aiming to measure a reduction in ‘unsafe behaviours’, such as the # of people standing on the edge of the platform, or the # of people crossing the tracks with an iPod, evaluation can be done!

    But the data isn’t collected for you. You can’t log in to an analytics website and have it handed to you in a nice table or timeline. You have to go find it, count it, and analyse it. It’s called research.

    To see a kick-ass evidence-based social marketing campaigns in action, look no further than Cancer Council Victoria’s Centre for Behavioural Research in Cancer (CBRC). The CBRC conducts research into community attitudes and behaviours relating to preventable cancers and their risk factors (ie, tobacco, obesity, skin cancer, as well as cancer screening and support services). Check it out at http://www.cancervic.org.au/ab…..rch_cancer

    The information gained from these studies is then used to identify critical public health priorities for future awareness campaigns and health promotion activities. The research is also often replicated to evaluate the effectiveness of the awareness and behavioural change objectives of the campaigns.

    For example, (and I’m paraphrasing greatly here from a presentation given at a 2010 conference).
    Prior to one summer, the CBRC conducted research into the ‘sun protection behaviours’ exhibited by the general public. Field researchers were dotted around public areas such as beaches, swimming pools, park and playgrounds, and they counted how many people are wearing sunglasses, how many people are wearing hats, are the hats peaked caps, full brim, wide brim, how many are wearing long sleeve tops, 3/4 sleeve, short sleeve, capped sleeve or no sleeves, how many people are in the shade, how many are not, how many people are overtly sunbaking?

    Now let’s say the research showed that females aged 12-30 were the group least likely to exhibit sun protecting behaviours – not wearing long sleeve tops and overtly sunbaking. A social marketing campaign is born, going through all the usual creative channels, focus testing, etc to address this behaviour.

    The campaign runs on our TV screens all summer. We see posters at the train station and behind toilet doors in shopping malls. Post-campaign, the research is conducted yet again, counting all the same behaviours as before. Behavioural change has been achieved if changes in behaviour are observed.

    Behaviours can and are often evaluated, but I’d guess often more for the sake of informing policy, funding and services, rather than general community awareness.

    But when the two work together it’s pure magic.

  81. Shamma
    23 Nov 12
    1:02 pm

  82. Adam – good point … but marketing will always err towards less rigourous methods to measure real behaviour change as they are motivated by financial gain to ‘sell the work’. Generally those commissioning the work are also motivated to sell the dream/manage up to push their own career agendas too.

    This also happens in science (when a researcher perhaps gets too close to their own hypothesis) but there are methods in place generally to separate subjectivities from actual causes.

  83. Craig
    23 Nov 12
    5:56 pm

  84. Dear Hugh Stephens,

    It appears that you have little understanding of how government agencies function, or their focus on social marketing and behavioural change.

    Government agencies that wish to influence behaviour – from smoking, dangerous drinking to train safety – conduct their campaigns over 3-8 years with an emphasis on behavioural change, not on marketing metrics.

    They already know what you pointed out – that most marketers focus on meaningless ‘vanity’ metrics – and have to spend a lot of their time disabusing advertising agencies of the notion that getting lots of views is more valuable than saving lives.

    They measure their campaign by behavioural change over time, tracking awareness, intention and action. The ‘vanity’ metrics are used only to help amplify reach, which can support the actual campaign intent by building awareness of the campaign at a much lower cost and saving the public money.

    I suggest you educate yourself as to the goals and processes used in government, you’ll find they know what they are doing most of the time.

  85. Hugh
    23 Nov 12
    6:25 pm

  86. Dear Craig,

    I’m well aware of how the government evaluates public health campaigns. I’ve worked on some and know many of the researchers who work in the space.

    But the melbourne train network has been privatized since 1999.

    m.theage.com.au/victoria/taken-for-a-ride-how-metro-inherited-a-dog-20110121-1a02t.html

  87. Bob Loblaw
    26 Nov 12
    11:20 am

  88. hahahaha Craig. nice one mate

  89. Dumb
    26 Nov 12
    4:08 pm

  90. A lot of how well this will do in terms of behavioral change will depend on which behaviors you choose to assess.

    The biggest problem is that those people that stand too close to the edge, or cross the tracks in between trains, etc are (as evidenced from the song title)… dumb. Because of that stupidity, it will be even more difficult to enact change. Further, I think too many people will listen to this and think ‘I’m not dumb, I’m perfectly safe standing where I choose to stand’ (until its too late). The song therefore parodies others, rather than being recieved as a recommendation to consider changing one’s behavior.

    Let’s just hope at the very least, that on a minor scale of behavioral change, a smart person on the platform at least gives the lunatic standing too close a ribbing for being a bloody idiot.

    And let’s not also discount measurements of a reduction in hair fires, grizzly bear abuse, piranha attacks, etc.

  91. Kristian
    27 Nov 12
    12:16 pm

  92. It is all quite simple.

    The effectiveness of the campaign depends on how you define what the goal of the clients campaign is.

    If the goal is to have a video that has a lot of views, then all you need to do is count views. Of course, there is no need for the video to be related to anything – you may as well post a hilarious keyboard cat video. “Look at all those likes – campaign successful!”.

    If the goal is to have a video that changes behaviour and reduces deaths, then all you need to do is do a longitudinal study, that measures changes in behaviour AND seeks to test whether the changes are casually linked to the viewing of the video (either directly or indirectly). You’d want to do this over no less than 5 years, to measure if there is any return to previous behaviour. You’d use surveys, interviews and perhaps mapping foot traffic using CCTV (both before and after the video is released). You’d need a large enough sample to have a reasonable margin of error, and it’d have to be across multiple demographies. To be more thorough you might want to then confirm the circumstances of any remaining deaths, to rule in or out suicide, murder and any other cause of death unrelated to the video. Then you’d want to submit the paper for peer-review. Then you might be able to say there was reason to believe the video was effective.

    But counting Facebook likes is much easier, completely meaningless, but much easier.

  93. Kristian
    27 Nov 12
    12:17 pm

  94. *Causally, not casually linked. LOL.

  95. Offal Spokesperson
    27 Nov 12
    2:50 pm

  96. I havent read all of the comments as they seem very much to focus on how important and good it is that the Ad industry makes these things.

    Well, i think the author has some strong points.

    Will this bring about any more care being taken at train tracks? – i doubt it
    Will it reduce the trespassers and their accidents? – i doubt it.

    The 90% of deaths that are from suicide are preventable, fuck funding flowery shitty tunes and adverts..spend the cash helping those poor people who feel they have no option other than to commit this horrific act.

  97. @ Offal Spokesperson
    29 Nov 12
    3:21 pm

  98. While I’m guessing here, suicide prevention as a subject would seem well outside the remit of a train network. It’s such a delicate subject a dedicated suicide prevention agency (whatever that may be) would be far better equipped to deal with it.

    That problem isn’t suicide on trains, but suicide or self harm at all. A far bigger job.

  99. @offal
    29 Nov 12
    4:58 pm

  100. so you’re saying that metro trains should fund or establisgh suicide prevention programs, instead of trying to reduce accidental injuries and deaths at train stations and crossings?

    that would certainly be a novel approach I guess.

  101. Grant
    30 Nov 12
    4:03 pm

  102. The author and other posters lost me when they said marketers needed to have an effective ROI measure of their campaigns success rather than what they deem a vanity measure, and then proposed between 3 – 5 years worth of research costs would be needed to determine that poperly.

    Its a campaign that cost a few thousand dollars to intiate. Having 3-5 years worth of research to determine the campaigns effect would be like spending $5K on an annual service for a 120Y each year.

    I agree people need to understand if a campaign is having an effect, I just don’t think it’s the best ROI of the clients money to spend 10 times cost of a campaign on research to figure that out.

  103. Andrew Smith
    30 Nov 12
    7:23 pm

  104. The sceptical view would be that many organisations, especially in the public sector, have to be seen to be doing something, and using their budget (because if not they receive less next year). Effectiveness is not objective, but the activity based round an event, campaign etc. for perceptions of results in short term to please “commissioners”.

    We still have universities and TAFEs, via conventional “outbound” promotions, endeavouring to communicate directly with prospective candidates offshore at expensive one off events, distributing “marketing materials” then measuring how many brochures given out, how many enquiries, and how many “applications” (meaningless, just EOI to maybe start process). However, what they will not do is assess from actual starters on campus how they learnt about institution and made application, why? Because 99%+ via digital/internet, word of mouth and offshore agents all year round…. (it’s called job preservation).

    Another one in the news is Tourism Australia whereby many hark back to the success of their Paul Hogan centred campaign in US late 80s, but while no one is privy to metrics, much evidence points to not just “Crocodile Dundee” (+ Oz film/music renaissance, i.e. content) effect raising Oz profile, but the all year round running of Paul Hogan comedies on free to air TV. Fast forward to “Where the bloody hell are you?” campaign which was deemed by many as unsuccessful, even the spoof was much better, with original going viral as an example of how not to do it, so what happens? The commissioners, i.e. the Minister (via spin meisters) then moves the goalposts by claiming the intention was for the campaign to go viral……

    At (start) and end of the day, what is being measured? If rubbery enough any campaign can be hailed as a success.

  105. Paul McNamara
    2 Dec 12
    7:01 am

  106. IProfessor Zimmet, director emeritus at Melbourne’s Baker IDI Heart and Diabetes Institute, would almost certainly agree with you Hugh. His critique of the $49 million spent on healthy lifestyle advertising campaigns is an echo of some of the concerns you have raised here.

    http://www.theage.com.au/natio.....z2DpeQcauE

  107. @Paul McNamara
    2 Dec 12
    1:42 pm

  108. ”To spend more than $40 million on social marketing campaigns without having an integrated strategy is a waste of money.”

    Read more: http://www.theage.com.au/natio.....z2DrCZUkXm

    Indeed. But Hugh’s missive (which seems to have raised his own profile – and I notice he’s dining out on this on crikey as well… perhaps he’s not immune to the odd spot of vanity metrics himself) was based on one enormous assumption:

    That Metro was doing NOTHING other than a single youtube video to address the problem. He had/has no knowledge of whether or not they have, as Professor Zimmet puts it, an “integrated strategy”.

    Hugh made several assertions, none of which were based on factual evidence, about Metro’s intentions or strategy. Very poor form.

    And the one suggestion he does make, that being Metro should spend their money trying to prevent suicides

    “Why can’t we put money into the issue that seems to cause far more deaths: suicide?”

    is just plainly ridiculous within the context of this safety campaign. Is he really suggesting that Melbourne’s train operator is meant to tackle the complex and enormous issue of suicide in Australia? Really?

    And who is this “we” anyway? I assume he means all of us, the government or whoever. Well, Metro is a private company, so they can do as they please.

    Summation of Hugh’s article:

    Marketers shouldn’t run behavioural-change campaigns because they just don’t work. (you could argue this either way, and Hugh isn’t the only social media consultant with an opinion on this matter)

    Metro does not have an integrated strategy for addressing rail safety (a fact he pulled straight out of thin air with no evidence to support it)

    Metro is solely aiming to reduce deaths, and not injuries as well (an assumption jumped to because of the title of the video I imagine, but again he’s stating personal opinion as fact. very unscientific of him)

    I’d say the main reason he wrote this piece was to promote himself and his consultancy. Whether or not it works, I guess Hugh will be able to tell us in 3 years after extensive analysis and research. Vanity metrics, though, would tell us it did the job.

  109. Miranda Likeman
    7 Dec 12
    10:37 am

  110. I actually know a person who was killed by a train accidentally. He put in his headphones and walked along the track to the next station with his back to an oncoming train. This was undoubtedly a massive mistake, but I wonder if he had seen this video before he might have thought twice about doing either of those things. And at the end of the day, you can’t measure how many people you DID help with a cute ditty like this one, because they are still alive. Hugh may come across as smug, but he makes some worthy points – I wait with baited breath for their ‘If you are going to kill yourself, don’t do it here cos it messes up the trains’ video.