Luckily, media buying is not brain surgery, but it costs more money, so it should be subject to the same, if not better processes.

cui bono?

First, the TV panels are actually not based on people – they are based on households (currently only in-home TV viewing is measured). This makes sense as TVs are (generally) shared devices. So, all up there are 7+m homes in Australia and the panel is just over 5,000 homes – just under 0.1% of the households.

The sample is random. There are no ‘qualifications’ (apart from having a TV in the home). There are no minimum or maximum viewing requirements for example.

The process undertaken is to augment the ABS Census (and inter-censal) data with an ‘Establishment Survey’ (ES). The ES is conducted as a continuous ‘wave’ by telephone (using random-digit dialling – i.e. we make the ‘phone number up so it is genuinely random) and conducted on 10 times the panel size – amounting to a survey of some 50,000 homes a year (it may be lower as the ES ratio in the regional areas may be lower – I can’t recall) into things that the ABS doesn’t collect that affect TV viewing. This includes things like number of working TVs, digital vs analogue, DVRs, household composition etc.

This then allows the statisticans to ‘segment’ Australi geographically, and within each geographic segment design the “target sample”. That is, how would you allocate the households if you had a blank sheet of paper. This results in a rather large matrix of “targets”. For example, in Sydney you would end up with targets by region (e.g.the northern region etc.), targets for no. of TVs (1 TV, 2 TV and 3+ TV), targets for presence of Subscription TV, targets for PVR homes, targets by houshold composition (e.g. single-person households vs large family households) and so on.

The panel is monitored daily and kept in balance against these targets. For example, a home may drop out of the panel and the statistician would look at the next ‘ideal’ home to balance the panel. This may result in a recruitment target of a 3-person home, with 2 TVs, digital, with a PVR, GB Under 55 that lives in Hornsby. The recruiters then go back to that “pool” of people from the ES and search for a home that matches the requirements – the home is then contacted and invited to join the panel.

So, how do we know that they aren’t in the garden etc?

There are a sophisticated set of edit rules that inspect both the TV tuning as well as the individual viewing every day. This is inspected both as daily data and as trended data. For example, if we notice a TV set on for (say) 10 straight hours without any channel change it is a pretty safe bet that it was left on inadvertently with no-one watching – so that home is removed for that day. The people are asked to push a button on the PeopleMeter remote whenever they are in the room and the TV is on – so we can inspect that data stream as well. So, if we saw a person logged in for 10 straight hours they would be removed as well. We also notice things like people being inadvertently logged in on 2 TVs at the same time – in this case we would “log out” the original TV when we noticed the second log in.

But how do we know that they ARE pushing the buttons? When you inspect the data across time, we look for people that suddenly stopped watching TV. This could indicate that they have got bored of button pushing, but in the majority of circumstances it normally expained by them being on vacation, having left the home, or having acquired a new TV that they do their viewing on. These scenarios are checked out by telephoning the home and where necessary installing meters on the new TV.

Further, when a home is contacted we ask about who was watching when the call was made – which people and which TVs, and which programme were they watching. This can then be checked against the meter records the following day. These checks are rolled up into one big study twice a year – and pleasingly 90+% of the time the buttons records accord with the telephone records. Of the remaining few percentage points that are different, it is most often because the ‘channel attribution’ is incorrect. The person on the ‘phone may say that they were watching Today Tonight on Channel 9 (even though it is on Seven). They may say they are watching Grand Designs on ABC when they are watching it on the Lifestyle Channel etc.

I know I have banged on a lot – but have merely scraped the surface.

The message is that a LOT of time, care and resources are put behind designing the sample, recruiting the panel, monitoring it daily, and then monitoring the data-streams. You could say that the system is even skeptical of itself with all the checks done!

If you have any further questions I will happily take them by email – jgrono@bigpond.net.au.

The Australian population is approaching 22 million. How many people and how are they qualified to be on the panel? If this has been covered before, please point me to the article.

I want to get my ahead around this. I am a big believer in accurate measurement, however will admit that I am skeptical of the stats that are provided about TV viewings.

How do you know if the TV is on but every member of the household is in the garden, or down the beach..?

regards

The phasing issue you raise is very true – we do have a step change as OzTAM were very clear in pointing out during all their numerous educational workshops. The ideal situation would have been to introduce them earlier when penetration was lower – but the rate of increase in PVR penetration was faster than any pundits predictions that I saw. The option of ‘phasing-in’ is a tricky one. What it means is that during the phase-in period the panel is no longer “live-only” and neither would it have represented the PVR universe – it would have been somewhere in between … neither one thing nor the other, which is far from ideal.

And yes, different meters can and do produce different raw data. In my experience the ‘change-line’ data differences are generally small and generally relate to how quickly the meter recognises a channel change. That is, if a person is surfing and flicking through a channel every second or so, do we “credit” that 1 second of viewing, or do we use a ‘rule of dominance’ (the most seconds in the minute wins), a ‘rule of the median’ (use the middle second of the minute), a ‘rule of persistence’ (credit the channel after, say. 15 seconds of consecutive viewing).

Parallel runs on the test-panel were done by OzTAM prior to the TSV change-over. Of course, no-one expected “the same ratings numbers” because the whole point of the (very expensive) exercise was to include the PVR homes which have different viewing habits. The parallel run data at the macro level met with broad expectations.

As Researcher says, you don’t just change a quarter of the panel overnight (with the valid objective of more closely reflecting the TV viewing population) and not expect change from what we were seeing the same time lst year.

Yes and a very good point too. Love your work – it makes sense now

I assume the old peoplemeters were not capable of measuring PVRs. I also assume Oztam have just stumped up the cash to get with the times and bought some of the new Nielsen Unitam meters.

The problem is, by not phasing the meters in as the proportion of households owning PVRs grew, you are left with a large step change in the data when you change 25% of the panel in one foul swoop. Even if the panel changes were like for like you would see data changes when you remove and add that many new panel mebers. It is obvious that PVR ownership is a predicator for differing viewer behaviour to “live only” households.

The other thing is and this is one that people are less comfortable with, but is true. I don’t care if Mr Grono confirms or not. If you have the same panel measured at the same time using the two different peoplemeters (even for “live only”) they will produce different results. For whatever reason the meters will produce different data when measuring the same viewing. It is odd but true.

So in this case you have 25% change in panel membership, 25% of panel using new meters and 25% of the panel using PVRs. Unfortunately it is impossible to say which is driving the change. Some parallel testing would have helped explain the differences. Assuming the research supplier has done this to the level required is often remiss.

Unfortunately for our friends hawking online advertising this has nothing to do with that. Nice try though, as John Grono alludes to, the issues with online measurement make these problems look insignificant.

Blimey who put you in charge..?

1. Tom – 19 Mar 10 – 2:20 pm. You say that the system “won’t take into account when the TV is off but the PVR, DVRs, set-top boxes are left on.”. It’s been long time since I have heard a comment on TV ratings that was more incorrect, so congratulations. If the TV is off, then no ratings are attributed irrespective of the status of any STBs etc. If multiple tuners, STBs etc are on (i.e. receiving a signal) only the one that is active on the screen is attributed. End of story.

Of course this contrasts with online measurement where as just one example page impressions served are counted whether (i) the page is rendered completely (ii) whether the page is rendered to a PC and not some bot, spider, or crawler (iii) whether the page is rendered to a browser that is both still active and with the focus (iv) whether the page is rendered to a tab in the browser that has the focus (v) whether there is a human – or maybe even more than one – at that computer. The magnitude of ‘sources’ on a PC far outweight those on a television. It is relatively simple on a TV to sort it out – on a PC it is difficult and needs a panel of people in order to model the actual usage. So if you are one of those people who nay-say TV ratings have another think about the far more fragmented usage on a PC and the implications for measurement.

2. iusedtosellshit – 22 Mar 10 – 1:32 pm. You say “i mean the length of time people have been pressing buttons every day… some 4-6 years for a good part of it.” There is a scintilla of accuracy in this. In Metro TV the rule is that a panelist can only be on the panel for a maximum of 4 years. This is constantly monitored and reported on by OzTAM. Generally, there are a handful of people who sneak into the “48+ months” group and they are quickly removed from the panel. This is despite there NOT being any compelling evidence that ‘button-pushing fatigue” increases with duration on the panel. If anything, the issue is when they first go on the TV ratings panel and some get carried away pushing buttons – which is why all new panelists are ‘withheld’ until they are ‘run in’ and patterns stabilise.

Some other facts to consider that relate to the television market in 2010 compared to 2010:
* The 2009 panel represented “live-only” homes – there were no TSV homes in the panel in 2009. In 2010 some one-quarter of the panel are TSV homes – reflecting the population more accurately. Yes, TSV homes use TV differently. Work the numbers to see how differently for your demo / client / campaign now that you have them.
* The 2010 market is different to the 2009 market. Each of the FTAs now have fully-blown second and third channels. You only need to look at the OzTAM share reports to see how much share they have captured and how much of that share has come from their primary channels – do the maths.
* Where the FTAs had second and third channels in 2009 they weren’t immediately “broken out” (i.e. they were included with the primary channel) until they had acquired some usage and mass – but by the end of 2010 each had been broken out and reported separately.

And then using that new result as currency to buy against makes your currency as valid as the Zimbabwean dollar! Good luck guys you are going to need it. Yours Truly R .MUGOBWE