Mumbrella to launch major research project examining advertising agencies performance
Mumbrella is set to undertake one of the most ambitious studies of advertising agency capabilities ever undertaken in the Australian market.
The major project is being led by Mumbrella deputy editor Robin Hicks with the results revealed at this year’s Mumbrella360 conference.
Criteria to be examined include creativity, effectiveness, integration, planning, talent, commercial success, account management, impact on the industry, momentum and client stability.
In its first year, the study will examine Australia’s top 25 advertising agencies.
Mumbrella’s insights will come from three major areas:
The first involves the creation of an expert panel of industry watchers whose jobs expose them to all of Australia’s major agencies. They include pitch consultants, clients and former clients, headhunters and a number of recruiters.
This panel will provide in-depth assessments of each agency’s abilities. Their anonymised comments will be compiled by Mumbrella as part of the report. As well as the panel’s qualitative input, they will also provide scores on each of the criteria for each of the agencies. The panellists will be announced when all participants have been recruited.
A second part of the assessment will see Mumbrella call on its email database of more than 23,000 people working in the media and marketing industry – probably the biggest database of its type in Australia. Mumbrella readers will be asked to complete a survey which will provide data on industry perceptions of each agency.
A third – and smaller – element will assess performance in industry awards. To avoid a multiplier effect of a single campaign winning at various events, the awards included in the scoring will be limited to performance at a small number of key events. Initially this will be The Cannes Lions and The Effies awards which recognise evidence of advertising effectiveness.
Each agency will also be offered the opportunity to contribute its own assessment on its performance for publication as part of the process.
The results of the project will be revealed for the first time at Mumbrella360, the major media and marketing conference being organised by Mumbrella in Sydney on June 7 and 8. The full findings will be available in print afterwards.
After the initial results of the study are published, Mumbrella’s expert panel will update its scores once a year. Mumbrella’s readership will be surveyed once a year – and at a different time to the expert panel. Scores for agency awards will also be updated, with old scores discarded as new events take place.
As a result, the score for each agency will change regularly across the year. Mumbrella’s full report – which will also be available in print format – will be revealed once a year as part of Mumbrella360.
Other elements of Mumbrella360 already announced include a session on ethics in marketing, a discussion on whether advertising has become too boring, and parallel streams focusing on marketing to Gen X, Gen Y, Gen Z and baby boomers. Meanwhile, senior media agency bosses who will lead a conversation on the move to automated trading, while another session will work to create a manifesto for the future of the media industry. There will also be a live marketing experiment curated by Naked Communications.
Discounted early bird tickets for Mumbrella360 are available until next Friday. For further details click here.
Tim Burrowes founded Mumbrella in 2008. He worked on Mumbrella until 2021. At various points he was editor, content director and editor-at-large.
Linkedin
Hope “Talent” covers HR issues such as turnover, retention, training, performance reviews and remuneration.
User ID not verified.
I cant believe the 3rd element is focussed on awards. I thought the industry had grown up and gotten over the so called importance of awards?
User ID not verified.
It would be neat to take ‘how much a client spent with an agency’ and compare that to ‘increase in client sales’.
User ID not verified.
Hope it’s not toooooooo long this time!!
User ID not verified.
Hi Dragon,
I agree with you that there is an over emphasis on awards – and particularly on rankings of awards where a single good piece of work can get hugely over-recognised in league tables if it wins in more than one place.
But equally, creativity does matter. And when the world’s best creatives sit down and arbitrate against a world standard – as they do at Cannes – their views mean something. So we’d be foolish not to include it in the mix at all.
Similarly, when agencies go to a great deal of trouble to demonstrate effectiveness – again, an essential – it adds an additional usueful signal.
But to emphasise, awards will be one small part fo the mix because it seems foolish to ignore them altogether – but they are not the main game.
We’re lookign for a far broader story than that.
Cheers,
Tim – Mumbrella
Quality of work life / opinions of employees would be a good measurement to add in the mixer
User ID not verified.
brlliant idea. clients will love it for the quality of info. major agencies may sulk if they dont score.
User ID not verified.
Tim, what is the source of the Top 25 agencies?
User ID not verified.
Tim,
The moment you included awards, you lost me. Creativity comes in many forms. Awards only concern themselves with one school of creativity. Under your criteria an agency that wins a Cannes Bronze Lion for a one-off piece of print work is considered more creative than an agency that creates something like the Bunnings campaign, or JB Hi-Fi’s work. These campaigns exhibit great creativity because their creators understand their categories, their target audiences and what motivates them perfectly. They mightn’t be fashionable, but sometimes the most creative thing you can do is NOT be anything like a Cannes winner.
User ID not verified.
Anonymous contribution of pitch consultants, clients, former clients and – headhunters? How many dinners at Bilsons do I need to shout to fix these results?
On a serious note, I love the fact Tim is taking the initiative to find a more holistic approach to ranking agencies rather than who’s scratching who’s back. If I were to add a few pointers:
1. Client wins 1 point, client losses -1 point unless it was a result of a global realignment. This is a good measure of relationships with clients.
2. Effectiveness / ROI – +1 point for every 10% measured, -1 point for every 10% lost. This should come from the client, not agency case studies.
3. Staff retention – indicator of how happy the cows the agencies milk are. +2 for every employee that’s stayed another year, +1 for every employee hired, -1 for every employee that’s left within three years, -2 for every employee that’s left under 3 years.
4. Innovation / bravery – +1 point for every campaign that is shifting advertising forward, no matter if you ‘like’ the creative or not, -1 point for every ad that’s been done before, 0 points for maintaining the status quo. This is important for the industries health as a whole, and could also encompass media platforms.
5. Awards – +1 point for any campaign that has won in the major local or international shows, this doesn’t include a single campaign that’s won at every show, e.g: Best job in the world = 1 point, not 20.
6. Relationships – +1 point for every positive relationship from a client AND supplier (including directors, photographers and headhunters), -1 point for every bad relationship.
7. Averages – overall points X number of clients ÷ number of staff should average things out a little better.
8. Billings divided by staff numbers to scale the agencies health.
It would be great to see the scores broken down as well, an agency may be great at making a profit and treating their clients well, but treat their staff and supplies poorly and produce bad work. This would help a marketer in making an overall decision.
Great initiative.
User ID not verified.
All sounding very much like the TV Week Logie Awards for Agencies!
Further, how does this panel inject objectivity into evaluation? Example: a pitch consultant (!) rejected by agency x, decides to mark down agency x.
A fatuous PR stunt.
User ID not verified.
Hi Sam,
That’s a good question – and one we’ll loo to our panel on. Having done a calculation on the back of an envelope, my sense is that the top 20 or so will be resonably obvious. Where it may be slightly more controversial will be in the marginal calls. I suspect there’ll be another 10 or so vying for places.
Cheers,
Tim – Mumbrella
Tim,
Great initiative. Just promise us all that you’ll come up with some sort of neat data visualisation that displays how all the agencies rank against each other. That way the ad industry can finally be rid of Mr Lynch’s (edited by Mumbrella) Hot or Not Chart. It would be so refreshing to have something with some sort of robustness behind it.
User ID not verified.
Hi Tim,
You almost lost me as well when you said you’ll also include performance in industry awards.
I have never chosen an agency based upon their success with awards and none of the companies I’ve worked for have either. It has simply never been a criterion for assessing the capabilities of an agency.
I’m also not sure how truly valuable this will be without the input of the agencies own clients. From my point of view there is really only one organisation that can genuinely assess the ‘success’ and ‘performance’ of an agency and that is the client.
I’ve always been involved in measuring the performance of an agency through processes like ‘Project Level Management’ which has the marketers and agencies rate each other’s performance on a particular campaign or program immediately after it has completed. These scores are than aggregated for a year round score and issues surfacing through this process are actually addressed going into the next year.
To be fair to agencies, I’ve seen as many bad briefs coming from marketers as I do bad response to briefs and I’ve always said performance is a two way street between the client and agency.
The Project Level Management process would evaluate both the agency and client’s performance on a scale of 1 to 10. I’ve put up a screen grab of just one version of this type of performance evaluation spreadsheet, this one for agencies – http://twitpic.com/4hjeae There are others out there and Forrester has a pretty good evaluation spreadsheet as well but you’ll also find no mention of awards.
I applaud the initiative but without significant performance evaluation from each agencies clients, I can’t see how it can be considered useful, accurate and truly representative of each agencies current capabilities?
User ID not verified.
Like it! Time to arrest the traditional approaches and create more wholesome benchmarks. One thing I’m a little confused about though is that this process will examine ‘creativity, effectiveness, integration, planning, talent, commercial success, account management, impact on the industry, momentum and client stability’
Seems a bit of an overkill, given that a firms creativity should really only be evaluated on how it CREATES client value, which from my experience comes from effectiveness, integration, planning, talent, commercial success, account management, impact on the industry, momentum and client stability…
I suggest to keep it simple and just judge creativity – in its true form, not in terms of being flamboyant.
And maybe include all firms that offer this value proposition to those charged with marketing. Or does 360 stand for ‘thousand’ of minimum budget?
User ID not verified.
A long overdue initiative. But the argument over methodology and criteria will rage for longer than the newspaper readership debate I’m sure. Those who won’t do well will already be lining up to shoot it down. Strap on your bullet proof vest Tim and go for it. Research should only ever be an aid to judgement and agency evaluation sure needs it.
User ID not verified.
How about losing 1000 points for wrongly entering an award category?
User ID not verified.
I agree it’s a fantastic initiative. The ‘talent’ section must be a real focus of this report. Staff turnover is a huge black mark on this industry and needs to be addressed. Agencies have to be more accountable for staff development and retention. I’d love to see an update on % of staff leaving their agency role in the first 18 months of employment.
User ID not verified.