Testing: audience knows best
Predicting the future of a film or television program and avoiding a financial disaster doesn’t necessarily require clairvoyant powers; testing your project with an audience at different stages of the production process can make a huge difference. Aravind Balasubramaniam reports.
“Nobody wants others to tell them their baby is ugly,” says the founder and CEO of Audience Development Australia (ADA) David Castran. “But it’s very important that producers listen with two ears and try to challenge their strongly-held views. Testing empowers producers by giving them information about the audience’s relationship with the pilot or program. I know at times that can be controversial.”
Drama has the highest production cost per hour of all TV programming, so well-executed testing can inform producers and broadcasters about the potential response to their shows.
Castran’s company was founded in the early 90s to provide comprehensive audience study services to the industry. Traditionally, groups of approximately 40 people in Melbourne and Sydney were shown the program and then handed a questionnaire to complete, followed by a moderated session, but ADA’s concept and pilot testing, as well as program evaluation, have mostly moved online – using an interface similar to catch up TV services – allowing the company to work with a bigger sample. “The laggers were the 50+ audience who were slower to get online, but they came around and now you can’t keep them out of the surveys; they absolutely love doing research online,” said Castran.
“Instead of having a sample size of 80, we are now doing 300 and the costs are the same. Plus, our clients are no longer subjected to the plight of watching judgment being passed on their projects.”
Gathering information is far easier than analysing it; things have to be put into context for the client, using normative data. Castran believes that, in the end, it’s all about the ‘actionability’ of the data – in other words, what can you do with the research material?
“We ask open-ended questions, resulting in people typing in quite a bit, a whole battery of things, which in turn calls for a filtering process to highlight general patterns. It’s really about guiding the client; when a
viewer or respondent says something, it’s often code for something else, and we can identify that because we’ve heard similar replies in the past.
“The normative database contains results of up to 400-500 tests conducted in the last 15 years, so we can make comparisons on, for example, how a typical comedy compares to others. Of course, we don’t show clients each other’s scores, instead we aggregate them into benchmarks, categories and genres. We try to answer questions such as ‘Are these results bad or good, high or low?’ or ‘Is that something I should be crying or be happy about?’”
Test results can also be a complete surprise.
“Sometimes executives will hate a show, but the audience will love it, and sometimes it can be the opposite.”
In Australia, broadcasters often commission a series without a pilot, and by the time they test one of the episodes it might be too late to rectify. Unlike the United States where testing is standard practice for all projects, Castran believes the process – including both pilot production and testing – hasn’t been sufficiently appropriated into the budgets of many local shows.
“The common excuses are ‘It’s already been shot’, or ‘It’s too late, what can we do with this information now?’, revealing a great difference between the US and Australian industries; we haven’t built testing into the production process; only a few shows go through it,” added Castran. “In the US you shoot a pilot and they make a decision on the show, but in Australia nobody wants to shoot a pilot. Broadcasters want to commission the series, and the production company wants to sell 13 or 22 episodes, not just one.
“Some shows come to us and they’re already shot and pretty much ready to go; I don’t know how many shows I have tested that have been 13-part series and the first episode is a shocker, and the network just sits there, thinking about how they are going to explain it to their managing director, why they have wasted $12 million. But other projects are indeed pilots, and the testing results can then influence the final product. For example, [Seven’s script executive] Bevan Lee does the research and then starts making decisions after coming out of that process. With Winners & Losers, he has used the results actively to make changes, and when we tested City Homicide, they looked at the research and ended up recasting three characters. The resource is there, don’t let anyone fool you, and even if a show is in production you can still make changes.”
TALKING TO YOUR AUDIENCE
As far as film testing goes in Australia, Bergent Research founder John Berenyi is a pioneer. Testing as far as the late 70s on films like Mad Max, Berenyi believes research has not been exploited enough by producers and the key reason lies in the structure and characteristics of the local screen industry.
“Clients tell us that we have contributed information that has helped them identify marketing hooks to get better box office results, and even opportunities to improve the film in terms of its playability.
“However, the nature of Australian film scene is more artisan; there are a lot of people who are making just one film or their first film, and look upon it as more of an art than a commercial production, and so their interest in testing might not be as high as it could be,” he said.
At its core, film market research is similar to research conducted for any other product; the questions remain the same – you seek out what people like about the product and what they don’t, and what motivates them. But film is not a physical object; therefore it is more difficult to market and research. “It’s harder to get 400 people to turn up to a cinema than answer a phone survey or an online survey and most importantly, everyone has their own opinion on what a good movie is. It becomes hard to get some kind of commonality and to get to the real motivators,” explained Berenyi. “For example, teenage boys might be reluctant to tell you that they liked it because they thought that one of the actresses was sexy, so they’d talk about other things. If you don’t know to probe through the real motivators, you’ll find that you missed an opportunity to attract a certain type of audience. The techniques are similar, but the analysis is very different.
Unlike Castran’s ADA TV, Bergent Research still maintains all its testing off the internet. “A film has to be tested in a cinema because the playability is different, particularly for a comedy; something might make people smile at home, but it would make them laugh with in the presence of an audience. Behaviour differs in a cinema, compared to what people do at home when they watch a movie; Tomorrow When the War Began, with its tension and special effects, also works better in the presence of an audience. If we look at some of the recent films that we have tested like Mao’s Last Dancer, you’ve got all that dancing, and it needs to be seen on a big screen. Bran Nue Dae and its big production numbers also plays better on a big screen.
“Our method starts by selecting people who are predisposed to the genre of the film. That’s very important because if you were testing Saw, you wouldn’t want an audience who like Mary Poppins, because it wouldn’t work. Over the years we’ve developed the right ratio of participants, so we then end up with an audience that is representative of the general market for that genre,” he said.
Some have debated whether testing represents a threat to artistic integrity. Berenyi believes this can be the case depending on how the research is conducted and delivered to the clients.
“Any research that tells the filmmaker or distributor how to fix the film is bad research. If I knew how to fix a film, by recommending cuts and shots, then I’d be the one making films. What the research should do is explain to the filmmaker how people are responding to the film, what parts of the story they’re involved with and what parts bore them and why.”
Screen testing is slowly becoming a priority in the budget plans of new projects. It’s a much needed practice, especially in an industry where productions are seen as lacking commercial ambitions. Producers need to both identify and understand the needs of their audience.
Robyn Kershaw, producer of box office hits Bran Nue Dae and Looking for Alibrandi, says testing played significant role in guiding the two projects to commercial success.
“The contributions of testing to the final product are huge. It’s constant evolution; it’s like the development of a screenplay, across 50 drafts,” said Kershaw.
According to Kershaw, there are different stages when you can test the material in order to support the vision for the film. “Early on in the post-production phase, what you are doing is testing the strength of the
story, whether it is delivering to the core audience, and making sure that it makes sense. Towards the end of the cut, where you are moving into a lock off situation, you’re working on how you’ll talk to the audience once the film is released into the world.” For Looking for Alibrandi, Kershaw worked closely with writer Melina Marchetta to identify the demographic required for their independently-run focus groups. Testing started early on, around the sixth week of post-production: “That was to help us understand whether we were on track, in terms of the story we were crafting. There was some great feedback, which informed the cut over the next two weeks,” recalled Kershaw.
Throughout the whole post of the film, the producer maintained a dialogue with the film’s core audience, and invited them to every screening – even the ones for investors: “They always need the assurance that the material is working for the core audience”, said Kershaw.
In the case of Bran Nue Dae, testing affirmed a position the filmmakers already believed in; they felt strongly that Ernie Dingo’s character (Uncle Tadpole) had a significant part to play in terms of the heart of the film. “The audience testing absolutely played that out,” she explained.
For that project, even the final artwork for the theatrical poster was tested via Facebook which, according to Kershaw, allowed audiences to take ownership of the film.
The producer’s interest in testing her screen projects is the result of her background in theatre, where work is constantly evolving. “It was just an instinct because in theatre you’re testing your work in front of an audience every night, and it changes according to how the audience responds. When you’re in the rehearsal room, you may think certain things are effective, but it’s not until you put it in a live situation that you can actually tell.
“However, once you have locked off a film, it is locked off forever. We were incredibly fortunate with Looking for Alibrandi and Bran Nue Dae because no re-shoots were necessary; all the raw material was there. We had the complete story captured on film, so it was a matter of making sure the right emphasis was happening in the right ways throughout the journey.”
Kershaw admits testing can be “an incredibly expensive exercise” because you can’t just rely on one test screening with 400 people to actually deliver results, but it is ultimately a good investment.
“Filmmakers need to think about how they are going to strengthen that relationship with their audience by way of testing. Sometimes it’s not just about ‘Is this working?’, but about finding the best ways to talk to that audience once the film is out there.
GOING BACK TO THE BEGINNING
Testing doesn’t necessarily have to take place at the end of the production process; even an unproduced script can be tested in front of an audience.
The Dungog Film Festival has recently expanded its script reading initiative In the Raw to a bi-monthly event in partnership with Sydney Theatre, which will allow more projects to benefit from the live reading
experience. Former In The Raw projects include Julia Leigh’s upcoming Sleeping Beauty, which participated in 2008, and Pauline Chan’s Mei Mei, currently in postproduction.
According to festival director Allanah Zitserman, testing at script stage can be hugely beneficial. “Get your script right and a significant part of the filmmaking job is done,” she said. “It’s also a great way to create momentum for a project on its way to production, or re-ignite one that may have hit some stumbling blocks.
“On a very practical level, reading can help identify flaws in structure, plot, pacing, and character, as well as highlight story holes. For the filmmakers, sitting back, watching and hearing their words read to an audience provides a whole new dimension and perspective. The key for filmmakers should be for stories and films to be written and made for audiences, and this program helps them understand if their script is doing that, by connecting with its intended audience. If it’s not, the process may help diagnose problems and find solutions.”
Testing is a creative’s friend and ally. Each producer will decide when and what – script, rough cut, marketing materials, etc. – to test, but what really matters is that they remember to be in touch with the audience they’re ultimately working for.
“Every time I do work with the audience, I just listen to them and think, ‘Look at how smart they are’. It’s not just IQ smart, but gut smart,” said David Castran.
And in the end, isn’t that the kind of smart everyone in the industry should aspire to be?
ABC Comedy Department and other networks take note…
“Sometimes executives will hate a show, but the audience will love it, and sometimes it can be the opposite.”
In Australia, broadcasters often commission a series without a pilot, and by the time they test one of the episodes it might be too late to rectify. Unlike the United States where testing is standard practice for all projects, Castran believes the process – including both pilot production and testing – hasn’t been sufficiently appropriated into the budgets of many local shows.
I mean isn’t this the problem with Australian TV right here…shouldn’t this kind of approach be factored into the pre-production process? An equilibrium must be found..because the current system, the one in which the entire population of our nation must be utterly aligned with the tastes and sense of humor of a handful of executives, producers and a few department directors has to change…if not we’ll just see more of “I Rocked” and “Laid” and “the Librarians”. Comedy is a tough gig and very hard to write effectively and I’m sick of Australian of all ages coming up to me and telling me how crap Australian TV is, a very wide demographic of complaint…it doesn’t need to be this way…the talent is there and for some utterly defunct reason it just isn’t being given a shot. Its about time some executives and directors of certain TV stations not only started having some pride in the job that they do but also the intelligence of the nation that they live in.
The stories we tell each other and the entertainment we create needs to be diverse and well crafted..if not, we’re just a nation sniggering at fart jokes or fat people trying to lose weight or people who can burn water while trying to compete in the thrilling arena of cooking!!!
…start casting the talent net wider. We need more diversity, because when we do get in right, its dramatically brilliant, thrilling and funny
User ID not verified.