Behind the smoke and mirrors of market research
In this guest post, an anonymous researcher questions the motives of the market research industry.
In the surprisingly litigious wake of the McCrindle debacle, I feel it necessary as a long-time researcher to voice my anonymous opinion regarding the legitimacy of market research.
To preface this article with a story: I once saw the MD of a research agency get up in a strategy meeting and doodle out a mathematical formula on the whiteboard, all of it weird Greek letters and unfathomable symbols.
He explained something about an analysis of a variance statistical model and strangely, I didn’t get any of it.
Why? Because he didn’t either; he was making it all up on the spot.
I’m pretty sure the clients didn’t understand it either, but unfailingly nodded with cow-blank expressions and busily took notes – and lo, the not-insubstantial contract was won.
This is particular case is extreme and an obvious outlier, however, having been around for a while, and having witnessed varying degrees of swindle, this type of thing goes on all the time. To be clear, I should stress that the issue raised by Media Watch about small sample sizes in the McCrindle Research in question, is not something I would describe as one of those swindles, although the publicity has been unhelpful for perceptions of the industry.
The odd thing is that except for a few times, such as when Media Watch happens to look up the research industry’s skirt – no-one cares.
And this is my probably wholly unwelcome defence of market research; that there is an unspoken B2B contract between the agencies and their clients.
Clients know surveys and focus groups aren’t hard science. If they were, the agencies would issue white lab-coats and clipboards as a standard uniform. You know who does wear white lab-coats? Actual researchers, do, while carrying out actual research, in an actual lab. They’re looking for things like, the cure for cancer, or less prestigiously, a better way for aging conservative males to maintain an erection, there are probably more than a few on the books of BP looking for a more efficient way to murder doe-eyed baby seals.
They’re not asking a group of fat middle aged women what they’re looking for in a chocolate bar, discussing which ridiculous car ad, most ‘empowered the consumer’, and they’re certainly not concerned with what Generation Y thinks of Kyle or Jackie O’s latest carefully structured PR ‘disaster’.
No, because all that is ‘marketing measurement’ maybe ‘consumer insights’ even, but not research. Yes, actual research does measure things, and will sometimes employ surveys, but to elevate market measurement to the status of research for this reason would be akin to giving a monkey a driving licence simply because it could beep the horn.
I’m pretty certain most (if not all) the clients out there understand that generally, few market ‘researchers’ have PhDs, or even care to understand statistics beyond the basics. I do distinctly remember a client who didn’t understand this, someone in charge of thousands of jobs and literally hundreds of millions of dollars, once briefing me, and I quote:
I read about choice modelling. What I need is for you to do that and just make a thing that will predict market movements next month, quarter and year, but it has to be correct within 1% of what actually happens – your company will be liable for any costs incurred by the failure of the model.
Now if I could do that; predict the future with a mathematical model, I’m fairly certain I wouldn’t be sitting in a briefing meeting with a man in a pink shirt and an intimidating tie. I’m pretty sure I’d be applying it to the stock market, living on my yacht and spending the rest of my brief, but thoroughly enjoyable life, up to my armpits in tequila and Tahitian princesses.
And yet, often, ‘researchers’ are trotted out, time and time again (usually when there’s budget to be spent or something has gone very, very wrong) and lauded as ‘insights gurus’ or messianic business-prophets. Regarding McCrindle, I would be very surprised if the actual count of respondents would have mattered at all to the public’s perception of the inferences made from the data.
But what it did do is attack this counterfeit-scientific legitimacy that agencies are desperate to maintain – which is why, I suspect AMSRS were investigating Mr. McCrindle, and he in turn looked to the courts.
Clients get that agencies probably can’t predict the future, but that’s not what they’re paying them to do. They’re paying them to be generally better with numbers and PowerPoint, and lets face it, carrying out surveys or focus groups is a massive pain in the arse.
The sooner agencies drop the charade of imagining they’re scientists or psychics, the sooner everyone can get on with their jobs and stop taking each other to court on ridiculous pretences.
- The person who wrote this still works in research
Agree with 99% of the above, but would like to point out there are market researchers with PhD’s using science-based methodologies, all published and peer reviewed. Proper neuroscience, not some dial-dialling-rubbish, when applied to market research, is capable of measuring consumer response and predicting consumer behaviour. In fact, it’s already happening.
User ID not verified.
What a load of nonsense.
There are good researchers and bad researchers. Ones that are passionate about delivering insights/ strategy borne out of rigorous research and ones that are either unable or too cynical too.
PhD’s are irrelevant, try looking at the some of the sample sizes the medical profession used to make sweeping health claims before you hold up scientists as the bastion of methodological excellence. In 18 years in market research I have not seen one ‘swindle’. Maybe the author needs to choose better agencies or maybe itheir agency home ts just natural selection at play. Time they had a career change.
User ID not verified.
Don’t acturies predict the future with a mathematical model?
User ID not verified.
Not sure I’m getting your point Anonymous, not sure why you are even in this industry. Most decent marketers and market researchers know that there is no such thing as the truth or a 99.9% predictive model. However, that doesn’t mean we just make up the numbers or indeed the numbers or make up of our sample. It’s one thing to say that the majority of Australians prefer to do a particular thing and another to base this on say young females only. Legitimate and professional market research (both quant or qual) helps 1,000s of companies every day in their decisions making. But agree it’s up to the market researchers and marketers to understand the limitations of any market research they do – that’s why it’s called market or social research and not scientific research.
User ID not verified.
The brave anonymous commentator is perhaps making up a straw man to knock down.
Most researchers I know and most clients who use marketing research, use it to understand who, what and why in relation to their specific activities.
For commercial activity, they want to know who has bought what, who intends to buy and why did they buy.
In government and social research, the need to know is much the same, just the payoff is generally not a purchase but a behavioural understanding and subsequent action to encourage the sought change in specific populations.
The aim is to help predict, to enable the clients to make more informed and so hopefully better decisions, as market research is just one input in organisational decision-making.
The methods used seek to provide information that will inform and assist the clients by giving better decision-making information than gut feel.
Most researchers do not pretend to be psychics or scientists, but do seek to provide accurate and reliable information that can be used as one input.
As an example, in survey research, properly constructed samples can reflect accurately responses to questions, and can often very reliably predict the future.
Look merely at the various political poll companies’ accurate prediction of the voting intentions as measured by the actual on-the-day votes.
Pretty good results and helpful to political parties and similarly accurate results are helpful to clients.
Your correspondent may enjoy a cynical and jaded pose, but the reality is good market research is an aide to better business and better practices and its use should be encouraged.
User ID not verified.
What on Earth would we do without Media Watch? Everyone will benefit from greater scrutiny of market research in Australia.
User ID not verified.
Marketing research, as it’s commonly applied to advertising, is nearly always completely useless. The reason? It’s almost always based on self-reported data (ie you ask someone whether they’d do a thing or not, or have done a thing or not). And as anyone who has ever been within cooee of a science text book will happily tell you, self reported data is erroneous. People always under or over estimate what they would or would not, or have or have not, done. People even ‘remember’ doing things they’ve never done before when asked. The only data worth its salt is actual objective observation. And no researcher ever does that. Focus groups and, worse, email surveys are a total waste of time. If they were otherwise, every client would insist upon them for everything to ensure maximum efficacy. But they don’t. Because the truth is research rarely makes a difference to the end result of a campaign, in my experience.
And don’t even get me started on corellation versus cause.
Snake oil, all of it.
User ID not verified.
Firstly, some advice to anonymous researcher: Find a better company to work for. I’ve got a fair bit of research industry experience and if you’re saying acts like the ‘outlier’ above are (oxymoronically) common place you really need to get out of where ever it is you are. There are plenty of companies that will respect you, your clients and the work they provide.
Secondly, every industry could find an “anonymous insider” willing to blow the lid off some sort of apparent controversy. I’m not particularly surprised a disgruntled arts grad thrust into the role of charting 100 crosstabs took a chance for some fleeting faux-notoriety.
My advice to research buyers: Be aware that companies/people like those described probably do exist (though from personal experience, I’ve not met one) and know how to see through their jazz hands and spirit fingers. Ask for detailed methodologies (not just buzz words), check data, samples, case studies, talk to other people they’ve worked with. Etc. Etc. Honestly, it’s not rocket science…that’s for people in white coats.
User ID not verified.
Research is great. When I was younger I filled out my entire family’s radio ratings diaries for my favourite rock station. And it was a big family. No-one else was interested.
User ID not verified.
Smoke and mirrors can make us look devastatingly attractive.
User ID not verified.
Market Researchers are fine, they take great pride in the rigour of their work in my experience. Sure it’s not rocket science, more stats meets sociology, but I have not met a shyster yet.
Client side? Another story. Ignoring huge findings because it doesn’t fit their agenda. Over simplifying a complex story. Adding 2 numbers together because the real one from the researcher they didn’t like.
Don’t get me started.
User ID not verified.
Gosh I love anonymous posts.
Being a researcher I thought I’d do some research. On the AMSRS website the membership list has 39 pages with about 55 members per page. That means over 2,000 active members. There is also an 8 page list of QPMRs – so there are around 440 of us.
In my 30+ years of research experience this is the first instance I am aware of the situation we have seen where an investigation was held, legal action was initiated, followed by the resignation of a member.
1 in 2,000 is not a bad strike rate.
Maybe I have been fortunate, but the gloomy picture painted in this article is foreign to what I have seen in my three decades in this industry.
And just some comments re ‘Cirlcing Sharks’ (sic). Fortunately you are a sample of n=1. In my experience people who use phrases like “People always under or over estimate…” or “The only data worth its salt is actual objective observation. And no researcher ever does that.” are prone to aggrandisement and hyperbole, rather than rational and reasoned observation and conclusion.
User ID not verified.
What a load of rubbish and from an anonymous source, give me a break!
There is a mountain of evidence that disproves this view and it’s an insult to thousands of researchers doing work that provides evidence for decision making every day.
The Research Industry Council Of Australia just hosted the 2012 Research Effectiveness Awards where a great many case studies were again entered and evaluted and all would prove Mr or Ms. Anonymous wrong.
If the anonymous researcher would like to discuss, I
am happy to publically or privately discuss why i think their “experience” is the exception rather than the way it really is.
User ID not verified.
And Peter, I would be glad to join in on that discussion if you would like.
User ID not verified.
I agree with Peter: Anonymous=Gutless
Market Research deserves a better caliber of debate from Anonymous than such gems as: “there are probably more than a few on the books of BP looking for a more efficient way to murder doe-eyed baby seals”……what!?
And from Circling Sharks: “Snake oil, all of it”…….seriously!? ALL of it?!
A few anecdotes by an “anonymous” source cannot be generalised to a total population…good job Market Researchers don’t take n=1 seriously!
Get real people!!!!!!
User ID not verified.
Robin makes an interesting, though ultimately irrelevant, point to the nature of media.
User ID not verified.
It gets down to what a client expects from the research and whether they have the skills to pick the right researcher for the job.
Selecting a market researcher is a lot like selecting a car mechanic. You have to know enough about the car to judge whether the suggested repairs are appropriate and let you get where you’re going.
I’ve seen market research misunderstood and misused – not because people are ‘bad’, but because they don’t have the education to understand what they actually bought.
It is often like a Douglas Adams novel. Clients understand the answer was 42, but don’t know what the question was…
User ID not verified.
Yes Robin, all of it. Because, as I mentioned, it’s underpinned by self reported data. And most researchers don’t even realise why such a source of information is so very unreliable. Marketing research is mostly a lot of people pretending their methods are as robust as actual science when the truth is anything but. And neither they, nor most clients, really understand what the difference is. Study real science for a bit, as I have before I got into this game, and you know the difference. Finish a BSc and then come talk to me.
Snake farking oil.
User ID not verified.
While I could certainly nitpick some of the content, the big picture issue that this article correctly points to is really this: Market Researchers do the most damage to perceptions of the market research industry by making patently false/weakly supported conclusions, overselling “advanced” analysis methods that are fundamentally flawed, and being unable to speak clearly to clients. I believe it is a famous Einstein quote? If you can’t explain it simply, then you don’t understand it well enough. @ResearchRocks
User ID not verified.
I worked at one of the larger agencies around town when I was a wee young suit. The kind where the creatives snarl at you “I didn’t come here to do fucking retail” when you try to give them a brief…
I sat in on a meeting where the GAD and the CEO of the agency decided to make up the research results for their last 6 months of ads for a particular client and plucked a number from the air as being the increase in unaided awareness and aided awareness to show what a good job the agency was doing.
They then chuckled as they decided they’d charge the client $40K for the made up and non-existing research.
They turned to me and winked, saying “what the client doesn’t know won’t hurt them and will help us” and laughed again. The client loves us they said and will believe us without any question.
That was my introduction to the world of multinational agencies and research
User ID not verified.
Understand the actual problem and you’ll be a long way ahead.
Research is like a lamp-post.
The idiots use it, like a drunk, for support.
The wise use it for illumination as they move on.
User ID not verified.
How representative of Market Research Companies and professionals is the anonymous writer of this article, it would be hard to know. Many are too quick to dismiss his comments out of hand. I think there is clearly an issue amongst some of the practitioners and companies operating in the market.
I actually laughed out loud about the first comment around Neuroscience. I almost cried, now that in it’s current form ,at least, is snake oil. All the white coats and PHDs won’t change that for a while.
The industry has been dumbing down for a number of years in my opinion, much of this stems from the people running many of the Market Research companies and also the universal adoption of access panel based research.
I see Market Research as a professional services business. Can you imagine having an accountancy firm run by someone who can’t really count or a legal firm being managed by someone who didn’t really have a grip on the law. Unfortunately this is exactly what has happened in the MR business, we could have been selling taps, for all many of our business leaders knew. The bottom line is king everything else is a secondary consideration including method, rigour and quality.
We are in the estimates business, we don’t deal in truth (leave that to the philosophers) or facts (the police I hope), these are samples, not a census. We are in a position to help steer our clients who are looking for information and the answers to questions so they can make better choices and decisions for their businesses.
This is probably all sounding a little bit Jerry Maguire, for that I apologise.
I work in MR and would also rather stay anonymous. I like many who have posted who work in the industry do not reckonise many of the practices outlined in the article however I am slightly less surprised that they take place.
User ID not verified.
Fortunately you people who are n=1 samples your opinion means as little in a sample as it does here.
User ID not verified.
Sorry Agency Junior, but that was your introduction to the world of Advertising Agencies, with the people involved using the term ‘market research’ to justify their extremely poor and unethical business practices.
User ID not verified.
Agency Junior – that sounds like the former boss of an agency I had the displeasure of working at – did he ever get busted?
User ID not verified.
@Researcher – thank you for being the first to spell phonetically viz:
reckonise
farking brilliant mistake – great mangle
User ID not verified.
n=1 samples aren’t irrelevant. They’re “depths” surely?
User ID not verified.
Circling Sharks: I have an MSc from UNSW in Applied Science, plus a stint as a research Scientist with CSIRO, Happy to share my Masters Thesis and peer reviewed published papers with you any time 🙂
Look: the limitations of self reporting are well known and obvious…..most people who conduct, plan and buy research understand this.
User ID not verified.
If I were to be pedantic circling sharks, if “People always under or over estimate what they would or would not, or have or have not, done”, report the average and you’ll be spot on.
(note to those whose internet sarcasm is toggled, this is sarcasm)
Anyway, I can’t wait for the next anonymous opinion piece on Mumbrella. Hope to see a guess who don’t sue column soon, maybe have a few paparazzo on the books…oops, forgot to toggle.
User ID not verified.
phds do challenge each others findings and assumptions, not accepting results on blind faith alone. (they have egos too).
When being presented to by an agency researcher, have an in-house researcher with you or someone competent enough to challenge the numbers.
To use a law analogy, can you imagine a criminal court proceeding with defense only and no prosecution?
User ID not verified.
Research can be useful if you understand the beast. I spent years in big agencies (with big research budgets) being told by researchers that certain TV ideas will work and others will fail. The fact is, almost every time a researcher was convinced we had it cracked -for all kinds of reasons on so many levels, the ad turned out to be ineffectual wallpaper. Lesson: if it comes out brilliantly in research, don’t go near it.
User ID not verified.
From the first ancient king who went to consult some senile folks pushing around their dead animals entrails – to the naked emporer asking how his clothes looked – to the modern habit of overstating the value of their research……a fool is till easily parted from their money.
User ID not verified.
@John Grono – #24
fair point – there were no actual research companies involved – just an ad agency and some unethical, but highly successful senior execs showing the new boy the ropes. I have never actually worked at any agency that did not bilk the client in some way. It was actually not until I went client side that I came across companies that did the right thing.
@Louie the Fly #25
No, as far as I know the CEO never got busted nor the GAD. This was a big firm and they were raking in the dough I believe.
User ID not verified.
You know you have written a great piece that cuts close to the bone when it’s attacked so heavily.
Seems like a lot of people want to protect their sham.
User ID not verified.
I think Anonymous is a fake with no understanding of the Market and Social Research profession.
I’m with Peter Harris – whoever you are, ‘Anonymous’ you clearly have no real idea of how serious market research works. If you do work in a research firm, I doubt that you have a position of any seniority or influence given your incomplete and one dimensional description of the topic.
We are all out here working our arses off solving problems, upholding good practice and taking on the McCrindles of this world. I suggest you learn some manners and stop bagging a profession you clearly have no serious understanding of. And give your real name if you really can back up a shred of what you say.
My real name is…
Martin O’Shannessy
CEO Newspoll
User ID not verified.
Go Martin!
As I said last week: anonymous=gutless
User ID not verified.
I think a CEO embarrassing himself publicly with a bog-standard internet teenager ad-hominem attack rather than a well reasoned argument speaks volumes about the respectability of the organisation he represents.
User ID not verified.
And I think “Interesting” has not added anything to the discussion. And frankly, until you have the guts to use your real name and put up some kind of argument you have no credibility. You don’t even say what your job is. Maybe when you do, other people will be able to anonymously post a bunch of critical and unsubstantiated bull about your industry. Until then you might consider the old adage ‘put up or shut up’.
Martin O’Shannessy.
User ID not verified.
You get what you pay for. I’ve been doing this almost 30 years and have helped many business keep customers, get new ones, focus improvement spending and test new products. There is a lot more subtlety to it that Circling Sharks seems to appreciate. Most of the research he describes which I have observed is from clients trying DIY or asking a junior staff member to design a survey. Yes, garbage in, garbage out. But with clever design, rigorous sampling and astute analysis, research can still be a great investment. I know – my clients keep coming back.
User ID not verified.