How reliable are radio ratings?
In this guest posting, Jason ‘Jabba’ Davis wonders how accurate radio ratings can be, since the data is collated from handwritten diaries.
So, the radio ratings season gets underway tomorrow. After a well-earned break, Australia’s commercial radio stations will renew their obsession with figures to see how many of us are listening. Are they winning or losing the ratings war?
The much feared radio survey is the only way to measure the success or failure of a station’s playlist, talent, promotions or even good old Black Thunder crosses. With six-figure salaries riding on the make-or-break nature of ratings, just how accurate are Australia’s radio survey results?
Sorry Jabba, I disagree. Of course Nielsen are going to tell you their system is perfect, that’s how they make money.
We’ve all heard the stories about the radio station who found out where the diaries were and got hold of one. The fact that you can skew a whole survey with one diary ( and I’ve seen it happen) proves that the system is flawed. You only have to graph the survey results to see the swings across all stations from demo to demo that occur purely because of a small sample size.
If this article wasn’t a paid ad for Nielsen, I don’t understand the point.
Jabba, I agree with Pinocchio, to be a piece worth the prominent placement on Mumbrella, you would need to balance up what you’e being told on a phone call with facts and opinions, and then make your conclusion. This might need some time on research and speaking to people on all sides of the industry. Good to see you’re writing.
It may not be the best system but it’s the only one we have. Can you come up with better ?
First son, I agree that there is no perfect system. There never has been and never will be.
But you say that swings between surveys are ‘PURELY because of a small sample size’.
And yes we have all heard the stories about the radio station who found out where the diaries were and got hold of one. We’ve seen bugger all proof of it though.
Do the maths. One household breaks their confidentiality agreement and fills in 24/7 listening to a single station. For starters, a diary like this gets discarded as being ‘rorted’. But in a sample of 2,400 (Sydney) that one home is 0.04% of respondents, but your trained eye can detect a skew for the WHOLE survey with that ONE diary as you have seen it happen.
OK son, I’ll call you out on this one. Name the survey and the market. Here. On Mumbrella.
As I started with, yes the diary system has flaws and can be improved upon, and no system is perfect, but I think you are drawing too long a bow.
I guess I used too much balsa when I attached the head.
The point of my piece is to highlight that the current diary system is far from perfect, as it utilises an archaic method to collect data. Until a more suitable alternative comes along, then hundreds of people working in commercial radio are subject to scrutiny and the ramifications of that scrutiny, be it positive or negative, at the whim of a bunch of random people ticking or not ticking the radio stations they listened to in little more than an exercise book. The hour long phone call involved a blow by blow account of exactly how the system has been tweaked over the years to avoid the rare occasions where the research was polluted by meddling radio people and my conclusion at the end of the call was that Nielsen have done what they can to provide the best possible data to ACRA. If ACRA were not satisfied that the results are worth the cost then I expect they would take their business elsewhere. Not sure where I can post the audio of the phone call with Nielsen which would reveal my extreme scepticism at Nielsen’s approach, (which was met with an extremely informed and honest response by their MD). He agrees it’s not perfect, but it’s the system we have. I would gladly have written many more words on the subject, but Encore’s budget allows only for 600 words. Perhaps if you were to renew your subscriptions Pinocchios 1 & 2 then you could enjoy more of my writing. Appreciate the comments. Cheers. Jabba.
This is written as if paper diaries are the only methodology for measuring radio audiences. This is patently not the case. There are a number of technologically advanced data collection systems allowing for passive measurement. It would have taken about two minutes on Google to find this out.
This is awful research related journalism. The author should have done some research. I am sure Nielsen are doing the best they can tweaking an antiquated system to remove bias and dodgy overclaim from the reported sample. It is still claimed behaviour.
Personal peoplemeters have been around in various forms for a long period of time. These devices capture the sound (audio) an individual is exposed to passively. The respondent carries a device (pager, phone or watch) which captures the audio signal. That audio is then sent to a central referencing centre to identify the radio station listened too. Think Shazam as a similar technique. There were issues with this form of measurement in the early days but now it produces high quality results. It is not perfect as it is hard to recruit a representative sample of people to accept placement of the devices and take them with them at all times, but at least it does not rely on claimed behaviour.
The main problem here is it is an expensive solution and probably more importantly it produces different figures to diary measurement. Intuitively most believe they are more accurate. The passive measurement favours smaller stations with less brand equity. The diary favours smaller number of the bigger stations you listen to, rather than the stations you actually listen to. Particularly in the car people radio surf to find songs or avoid adverts, not always aware of the station on which they spend time or even listened.
The upshot is diaries have much lower numbers of stations listened to than passive electronic measurement delivers. This does not have a massive impact on the reach of big radio stations for either form of measurement, however electronic measurement sees a marked increase in the reach of smaller stations. What does tend to happen is the time spent with stations measured (passively) decreases for the large stations. This may have more to do with why this form of measurement isn’t adopted, it will likely reduce revenue for the bigger players. The diary method favours and prolongs the commercial status quo.
For me what demonstrates most clearly the problem with a recall diary is the average number of stations each respondent recalls listening to. The figure is always frighteningly low, much lower than you would expect. Granted some people have the radio on one station permanently and regardless, but the figure always seems low. Maybe Peter from Nielsen can share that figure with us.
Apologies for the length of the post, I am sure I lost most of you at peoplemeter.
Hi Researcher,
I should point out that Jabba’s piece refers to the current methodology which is entirely diary based, not the digital alternatives that I’m sure will be trialled here fairly soon. I should also point out that the article first appeared in the print edition of Encore, where it was written to a set (and relatively short) word count, so there was not space to discuss the people meters you mention above.
But I’ve got a funny feeling that this is just the first time of many this year that we’ll be talking about people meters in Australia. Over to you, CRA…
Cheers,
Tim – Mumbrella
What sort of incentive is there to fill out the book? I did it once as a teenager. I remember filling in my favourite shows regardless of whether or not I listened to them that day. I imagine a lot of busy people wouldn’t bother doing it.
Geppetto.
Perhaps leave the trolling to the Billy Goats Gruff.
So Pinocchio, I take that as a ‘no’ that you won’t (or can’t) name a whole survey that has been skewed by one diary.
I agree with Jabba – the system is not perfect but it is the one we have.
Researcher – 10 out of 10.
We all know that for years TV, radio and print have been padding the crap out of their figures.
TV keeps inventing new stats to chuck in. “Let’s add the people who borrowed a recorded copy from their neighbour’s mum!” Radio has always been more art(ificial) than science and print…
Check out the skips behind any given university newsagency on any given day – you’ll find half the so-called circulation of our major masterheads rotting in the sun.
I’ve written about the flawed survey many times, both in print and on my blog. As I wrote recently (http://debritz.net/node/2666), the radio industry will be forced into the 21st Century simply because other media (especially online media, but television too) can provide better audience data.
As times get tougher, and more advertising channels present themselves, dvertisers are not going to continue to base big spending decisions on a system that relies on people a) remembering what station they actually were listening to and b) writing it down at the time.
If the system worked properly, radio stations wouldn’t be spending the bulk of their marketing dollars during the survey period — not just to get people to listen, but the imprint their brand on the minds of those crucial few thousand with a survey diary.
People still listen to the radio?
Hey Az. So I take it you think that if someone recorded the Golden Globes because they weren’t home last night, and watches them tonight that theyshouldn’t count in any way. Any recorded programme NOT viewed within 7 days is ignored. Any fair-minded person would say … yeah … seems OK to me.
And as for those copies in the skips rotting in the sunat the back of university newsagents. First, the copy must have been under a purchase arrangement with an individual studen – this sets the maxiumum possible copies. Second, returned copies must be accounted for. Third,educational institution copies are itemised separately and they come nowhere near your estimate of ‘half the so-called circulation of your major mastheads’.
Sometimes I wonder if people bother to actually look at the data, or prefer to spout off ill-informed shibboleths.
With regards to the measurement system for radio audiences, whilst flawed it is still the best we have at this stage. Why? Think about how you consume TV, usually it’s at home the majority of the time therefore a set top box works well in measuring data. Moving onto online and mobile advertising, the benefits are that IP addresses get measured and most people are creatures of habit with their browsing experience across their various devices, laptops, tablets and mobiles.
With regards to radio think about the way you listen to radio, at home, driving to / from work, at work, online. There are a number of different devices of which the majority are analogue which means unlike TV / online it is hard to measure electronically. Until a new solution becomes available then this is the best we have.
Hi John
Can you fill in the blank for me please –
Television, Radio and Print advertising reach has —– dramatically over the last decade.
(The incorrect answer is: Grown.)
Thanks!
Az. Please define what you mean by ‘advertising reach’. There are numerous answers to this question – depending on your definition.
Hi Az,
I think the answer is…grown. It’s fragmented more so it requires more $$ but the reach would have grown across TV and radio – print would be down but only marginally
Where do I start. First John, when answering the inane idiocy that is a post from AZ, you need to be a little clearer on measuring playback.
‘Any recorded programme NOT viewed within 7 days is ignored.’
Whilst this is true (at least for now) you should also add that the recorded programme has to have been broadcast within the last 7 days prior to playback. This will have to change with the advent of smart TVs, non linear references will need to be added as will a longer window, potentially 30 days.
Paul, I suggest you read my post earlier. There is an alternative to the diary. It is a meter but not a fixed one as is used in TAM (TV Audience Measurment) it is a personal meter that the respondent takes with them, this device captures the audio thay are exposed to. Arbitron in the US use this method to measure Radio audiences.
Lastly Tim Mumbrella. Please read the article again, the headline is clear. My comment deals with the issue the article should have covered. It reads like a puff piece for Nielsen. It could have and should have been much more even in 600 words.
My final point would be that even though radio diaries are out of date and seem anachronistic. I am sure they are a far closer reflection of reality than any online audience measurment data that is produced anywhere. Now that truly is unreliable.
It seems either anachronistic or dishonest to continue to use such a manual and human error/fraud prone system to measure audiences for an industry worth hundreds of millions.
The biases are only magnified by relying on a system which is based on finding people willing to complete complex logbooks over long periods of time. They have already self-selected.
With tools like Shazam it is a trivial matter to identify which stations a person is listening to using a remote wireless device carried by an individual or placed alongside or within radios. A GPS would verify that the device was In the correct household.
Alternatively any radio should be able to be simply modified to track the frequency being listened to and report back by itself. This would not tell you who was listening (though this could be supplemented by manual means like a simplified logbook or periodic survey),but would provide precise data on the stations listened to.
If I can come up with two more accurate alternatives in five minutes I do not see why the radio industry cannot.
Unless they do not want to…
Agree Researche. By saying that the recorded programme had to be viewed within a 7 day window, I was implying that it must have also been broadcast within that same 7 day window (or at least I thought I was).
The issue of non-linear is not as big as the issue of ‘non-referenced’ content. For example, if someone watches ABC content on their TV using iView, then the current system recognises that content and duly credits it to the ratings (as time-shifted viewing) as long as it is still in the 7 day window. Further, this system is also capable of being ‘scaled’ so that the viewing of (say) iView content on a computer rather than a TV can also be included. This however raises the issue of viewing of ‘non-referenced’ content – for example the viewing of new series from the US over the internet. It also poses the question of whether such content should be included in “the Australian TV ratings”, which are designed to measure the bulk of TV viewing but not all of it – for example public-place viewing is not measured. There are ways of measuring public-place viewing, but they are expensive and the incremental audience is probably not worth the investment apart from a few stand-out events like the Melbourne Cup, Grand Finals and World Cups. It is a commercial decision.
Similarly, the issue of ‘extending’ the 7 day window is also a commercial decision. In essence this would mean that the ‘final ratings’ would not be available for 30 days – meaning that client post-analyses would be more than a month after the event. It also raises the issue of commercial content being seen within the longer window, however the product or or ‘half-price sale’ may no longer be on offer, so what commercial use is that to the advertiser. What we may be gaining on the audience swing we may be losing on the commercial utility roundabout. The current 7-day window seems a pretty good middle position to me.
I’m amazed that I’ve ignited such passionate debate about a topic that to me was bordering on mundane. I wanted to blow off some steam about how Survey Diaries wield so much power in the commercial radio industry. I have been on the receiving end of both ends of the stick, and was looking to discredit the existing system as I had no real grasp of what was involved. Following my chat with Nielsen I had a much greater understanding of their particular methodology and with further research whilst writing my piece could not see any superior method of collecting data. Personal people meters…? Please. An online survey? Please. I would dearly love to hear from anyone else who has actually taken the time to fill out a diary in 15 minute increments. I personally cannot conceive that a vast majority are doing this, but maybe I have a tiny mind. As one post above mentions it is more likely participants fill it in when it is due and tick the stations and shows that are top of mind. As for being a puff piece for Nielsen, I can find no evidence to discredit their sampling system, other than to mention as I did in the original piece that it seems old fashioned. In the time they have been providing the service they have sought to improve the quality of the data, and surely as a data collection business it’s in their interest to do so? So until the gripers come up with a viable alternative, it’s what the industry uses to award bonuses, set rates and hire and fire and as I have pointed out, we are stuck with it. Since the story was first posted on Mumbrella, another story about trials for new methods has gone up and makes for a good read. https://mumbrella.com.au/radio-industry-to-test-digital-measurement-system-71090 It also covers some good points I was unable to fit into my piece due to size constraints. JD.
Hi Jabba. I tend to agree with your comment “I could not see any superior method of collecting data.”
The tough thing about radio is that it is actually three different audiences we’re trying to measure – in-home radio, in-car radio and at-work radio. TV is only now starting to face issues such as “TV anywhere, anytime”.
The fact is that there is no ONE superior way of collecting radio data.
To my way of thinking, the days of having a single data capture methodology are numbered – and hopefully sooner rather than later. For some demographics (such as my in-laws) a paper-diary would be the only method they would consider and they would be extremely diligent at it. For others (hello nieces and nephews) either a portable meter or a smartphone app would do PART of the trick – despite what everyone says, they do NOT have them on and carry them 24/7. Meanwhile for others (hello office workers) a website version of the paper diary could very well be optimal. In fact, I think we need a way to use all of the above (without blowing the budget) and as long as we can be sure that the ‘ratings’ collected via the different modes are directly comparable (or have known empirical differences that can be accounted for), then I think we would be making real progress.
So Jabba …thanks for raising the topic.
Yeah – bad system – I filled in one of those diaries last year – and as a result my listening habits changed for that week “channel surfed” to check the other stations out.
John Grono is on the money with the breakdown of demographics of who would use different methods to record radio listening habits. I say if Nielsen can’t find a way to move beyond the much maligned paper diaries, then someone else will. Craig, as for coming up with “two more accurate alternatives in five minutes”, just writing down an idea is not the same as developing a feasible system. Shazam works because there is a finite catalogue of songs it can choose from. Given that radio (apart from songs & ads) transmits fresh material every second (insert own topical joke here about shows that rehash audio), then unless the radio station were to broadcast a specific tone to be captured by a device, then, I can’t conceive of technology that could identify what station is being listened to. Secondly, who is going to pay for existing radios to be modified to “to track the frequency being listened to and report back by itself.” Not likely. I’m sure if someone can develop an application that uses technology in an affordable fashion to accurately report listening habits then it would be warmly welcomed. Just maybe not by Nielsen. JD.
hey jabba great post man 🙂
I agree here with what Jabba is saying. He is “Keeping it Real” which is more than we can say for some! So how do you measure your ratings? Most people cant be bothered to complete any surveys unless there is something in it for them! Also apart from the North Shore most other areas are full of Migrants who cant read or write .
The way I see it how come “Kyle and Jackie O” are always number 1? I really find that hard to believe as if you look at the popoulation we have school kids who listen to 96.1FM and then the 40 plus who listen to 101.7Fm and over 50’s who listen to 2UE AM radio so that leaves the 25 – 35 group. Are you telling us that we have more of that group overtaking the others. I dont think so.
Anyway I really think they pick and choose which households they will get to do surveys . I have never in 20 years heard of anyone that I would know complete a survey!
I always get calls to change my home phone plan or some Indian call centre telling me Ive won something !
Keep it real people only Jabba does !
You know Jabba, you look like a very good looking version of Eric Cantona.
Lets be honest, ALL forms of audience/reader measurement are flawed. Give away newspapers, online adblockers and dodgy diaries are just the tip.
The fact that advertisers and agencies put so much faith in them still amazes me.
I’ll take any survey. Regional radio hasn’t surveyed for many, many years.
I used to work in regional radio, and all you had was the personality of the station.
“We’re a rock station that targets males 35 +”
And guess what, it did! And the advertisers were successful in selling to that market.
You don’t even need ratings. A good salesman can sell a station that has crap ratings. The key is if there’s high enough frequency, it’ll work. talk to 300000 people 5 times a week, and you’ll see only ok results. Talk to 20,000 people, 30 times a week and BOOM! It’ll work.
I totally agree that people filling in these diaries are filling in what’s top of mind, last minute. But, isn’t that just as powerful? A radio’s programming, talent, music will suit these peoples taste and that’s what they listen to. And what ever suits them the most is the most tp of mind, they’ll fill that in.
I’ve just come up with a system!
Have even more people surveyed, and have them agree that they will be contacted at random times throughout a month.
Like on their phone, they’ll get a text that says “what station are you listening to?” and they reply. It could happen while in the car, in the office, or while in bed shagging the next door neighbour. That saves people having to fill in and hand in anything. And puts them on the spot. If they don’t reply in 5 minutes, then it’s not counted.
BRILLIANT!
I want royalties!!!!!!!!!
I just like it when anyone in the media takes leave, it’s described as “well earned”. It’s just so comforting to hear decade after decade. Like pulling on worn out saggy underwear.
I will be interested to hear from Nielsen what are the inclusion criteria?
How do they select the people for this? What sort of bias they use?
Do they randomly select the people to whom they sent the diaries or do they target the specific age, gender, population, geographic distribution, radio reception etc?