Behind the smoke and mirrors of market research

Agree with 99% of the above, but would like to point out there are market researchers with PhD’s using science-based methodologies, all published and peer reviewed. Proper neuroscience, not some dial-dialling-rubbish, when applied to market research, is capable of measuring consumer response and predicting consumer behaviour. In fact, it’s already happening.

What a load of nonsense.

There are good researchers and bad researchers. Ones that are passionate about delivering insights/ strategy borne out of rigorous research and ones that are either unable or too cynical too.

PhD’s are irrelevant, try looking at the some of the sample sizes the medical profession used to make sweeping health claims before you hold up scientists as the bastion of methodological excellence. In 18 years in market research I have not seen one ‘swindle’. Maybe the author needs to choose better agencies or maybe itheir agency home ts just natural selection at play. Time they had a career change.

Don’t acturies predict the future with a mathematical model?

Not sure I’m getting your point Anonymous, not sure why you are even in this industry. Most decent marketers and market researchers know that there is no such thing as the truth or a 99.9% predictive model. However, that doesn’t mean we just make up the numbers or indeed the numbers or make up of our sample. It’s one thing to say that the majority of Australians prefer to do a particular thing and another to base this on say young females only. Legitimate and professional market research (both quant or qual) helps 1,000s of companies every day in their decisions making. But agree it’s up to the market researchers and marketers to understand the limitations of any market research they do – that’s why it’s called market or social research and not scientific research.

The brave anonymous commentator is perhaps making up a straw man to knock down.

Most researchers I know and most clients who use marketing research, use it to understand who, what and why in relation to their specific activities.

For commercial activity, they want to know who has bought what, who intends to buy and why did they buy.

In government and social research, the need to know is much the same, just the payoff is generally not a purchase but a behavioural understanding and subsequent action to encourage the sought change in specific populations.

The aim is to help predict, to enable the clients to make more informed and so hopefully better decisions, as market research is just one input in organisational decision-making.

The methods used seek to provide information that will inform and assist the clients by giving better decision-making information than gut feel.

Most researchers do not pretend to be psychics or scientists, but do seek to provide accurate and reliable information that can be used as one input.

As an example, in survey research, properly constructed samples can reflect accurately responses to questions, and can often very reliably predict the future.

Look merely at the various political poll companies’ accurate prediction of the voting intentions as measured by the actual on-the-day votes.
Pretty good results and helpful to political parties and similarly accurate results are helpful to clients.

Your correspondent may enjoy a cynical and jaded pose, but the reality is good market research is an aide to better business and better practices and its use should be encouraged.

What on Earth would we do without Media Watch? Everyone will benefit from greater scrutiny of market research in Australia.

Marketing research, as it’s commonly applied to advertising, is nearly always completely useless. The reason? It’s almost always based on self-reported data (ie you ask someone whether they’d do a thing or not, or have done a thing or not). And as anyone who has ever been within cooee of a science text book will happily tell you, self reported data is erroneous. People always under or over estimate what they would or would not, or have or have not, done. People even ‘remember’ doing things they’ve never done before when asked. The only data worth its salt is actual objective observation. And no researcher ever does that. Focus groups and, worse, email surveys are a total waste of time. If they were otherwise, every client would insist upon them for everything to ensure maximum efficacy. But they don’t. Because the truth is research rarely makes a difference to the end result of a campaign, in my experience.

And don’t even get me started on corellation versus cause.

Snake oil, all of it.

Firstly, some advice to anonymous researcher: Find a better company to work for. I’ve got a fair bit of research industry experience and if you’re saying acts like the ‘outlier’ above are (oxymoronically) common place you really need to get out of where ever it is you are. There are plenty of companies that will respect you, your clients and the work they provide.

Secondly, every industry could find an “anonymous insider” willing to blow the lid off some sort of apparent controversy. I’m not particularly surprised a disgruntled arts grad thrust into the role of charting 100 crosstabs took a chance for some fleeting faux-notoriety.

My advice to research buyers: Be aware that companies/people like those described probably do exist (though from personal experience, I’ve not met one) and know how to see through their jazz hands and spirit fingers. Ask for detailed methodologies (not just buzz words), check data, samples, case studies, talk to other people they’ve worked with. Etc. Etc. Honestly, it’s not rocket science…that’s for people in white coats.

Research is great. When I was younger I filled out my entire family’s radio ratings diaries for my favourite rock station. And it was a big family. No-one else was interested.

Smoke and mirrors can make us look devastatingly attractive.

Market Researchers are fine, they take great pride in the rigour of their work in my experience. Sure it’s not rocket science, more stats meets sociology, but I have not met a shyster yet.

Client side? Another story. Ignoring huge findings because it doesn’t fit their agenda. Over simplifying a complex story. Adding 2 numbers together because the real one from the researcher they didn’t like.

Don’t get me started.

What a load of rubbish and from an anonymous source, give me a break!

There is a mountain of evidence that disproves this view and it’s an insult to thousands of researchers doing work that provides evidence for decision making every day.

The Research Industry Council Of Australia just hosted the 2012 Research Effectiveness Awards where a great many case studies were again entered and evaluted and all would prove Mr or Ms. Anonymous wrong.

If the anonymous researcher would like to discuss, I
am happy to publically or privately discuss why i think their “experience” is the exception rather than the way it really is.

I agree with Peter: Anonymous=Gutless

Market Research deserves a better caliber of debate from Anonymous than such gems as: “there are probably more than a few on the books of BP looking for a more efficient way to murder doe-eyed baby seals”……what!?

And from Circling Sharks: “Snake oil, all of it”…….seriously!? ALL of it?!

A few anecdotes by an “anonymous” source cannot be generalised to a total population…good job Market Researchers don’t take n=1 seriously!

Get real people!!!!!!

Robin makes an interesting, though ultimately irrelevant, point to the nature of media.

It gets down to what a client expects from the research and whether they have the skills to pick the right researcher for the job.

Selecting a market researcher is a lot like selecting a car mechanic. You have to know enough about the car to judge whether the suggested repairs are appropriate and let you get where you’re going.

I’ve seen market research misunderstood and misused – not because people are ‘bad’, but because they don’t have the education to understand what they actually bought.

It is often like a Douglas Adams novel. Clients understand the answer was 42, but don’t know what the question was…

Yes Robin, all of it. Because, as I mentioned, it’s underpinned by self reported data. And most researchers don’t even realise why such a source of information is so very unreliable. Marketing research is mostly a lot of people pretending their methods are as robust as actual science when the truth is anything but. And neither they, nor most clients, really understand what the difference is. Study real science for a bit, as I have before I got into this game, and you know the difference. Finish a BSc and then come talk to me.

Snake farking oil.

While I could certainly nitpick some of the content, the big picture issue that this article correctly points to is really this: Market Researchers do the most damage to perceptions of the market research industry by making patently false/weakly supported conclusions, overselling “advanced” analysis methods that are fundamentally flawed, and being unable to speak clearly to clients. I believe it is a famous Einstein quote? If you can’t explain it simply, then you don’t understand it well enough. @ResearchRocks

I worked at one of the larger agencies around town when I was a wee young suit. The kind where the creatives snarl at you “I didn’t come here to do fucking retail” when you try to give them a brief…

I sat in on a meeting where the GAD and the CEO of the agency decided to make up the research results for their last 6 months of ads for a particular client and plucked a number from the air as being the increase in unaided awareness and aided awareness to show what a good job the agency was doing.

They then chuckled as they decided they’d charge the client $40K for the made up and non-existing research.

They turned to me and winked, saying “what the client doesn’t know won’t hurt them and will help us” and laughed again. The client loves us they said and will believe us without any question.

That was my introduction to the world of multinational agencies and research

Understand the actual problem and you’ll be a long way ahead.

Research is like a lamp-post.

The idiots use it, like a drunk, for support.
The wise use it for illumination as they move on.

How representative of Market Research Companies and professionals is the anonymous writer of this article, it would be hard to know. Many are too quick to dismiss his comments out of hand. I think there is clearly an issue amongst some of the practitioners and companies operating in the market.

I actually laughed out loud about the first comment around Neuroscience. I almost cried, now that in it’s current form ,at least, is snake oil. All the white coats and PHDs won’t change that for a while.

The industry has been dumbing down for a number of years in my opinion, much of this stems from the people running many of the Market Research companies and also the universal adoption of access panel based research.

I see Market Research as a professional services business. Can you imagine having an accountancy firm run by someone who can’t really count or a legal firm being managed by someone who didn’t really have a grip on the law. Unfortunately this is exactly what has happened in the MR business, we could have been selling taps, for all many of our business leaders knew. The bottom line is king everything else is a secondary consideration including method, rigour and quality.

We are in the estimates business, we don’t deal in truth (leave that to the philosophers) or facts (the police I hope), these are samples, not a census. We are in a position to help steer our clients who are looking for information and the answers to questions so they can make better choices and decisions for their businesses.

This is probably all sounding a little bit Jerry Maguire, for that I apologise.

I work in MR and would also rather stay anonymous. I like many who have posted who work in the industry do not reckonise many of the practices outlined in the article however I am slightly less surprised that they take place.

Fortunately you people who are n=1 samples your opinion means as little in a sample as it does here.

Agency Junior – that sounds like the former boss of an agency I had the displeasure of working at – did he ever get busted?

@Researcher – thank you for being the first to spell phonetically viz:

reckonise

farking brilliant mistake – great mangle

n=1 samples aren’t irrelevant. They’re “depths” surely?

Circling Sharks: I have an MSc from UNSW in Applied Science, plus a stint as a research Scientist with CSIRO, Happy to share my Masters Thesis and peer reviewed published papers with you any time 🙂

Look: the limitations of self reporting are well known and obvious…..most people who conduct, plan and buy research understand this.

If I were to be pedantic circling sharks, if “People always under or over estimate what they would or would not, or have or have not, done”, report the average and you’ll be spot on.

(note to those whose internet sarcasm is toggled, this is sarcasm)

Anyway, I can’t wait for the next anonymous opinion piece on Mumbrella. Hope to see a guess who don’t sue column soon, maybe have a few paparazzo on the books…oops, forgot to toggle.

phds do challenge each others findings and assumptions, not accepting results on blind faith alone. (they have egos too).

When being presented to by an agency researcher, have an in-house researcher with you or someone competent enough to challenge the numbers.

To use a law analogy, can you imagine a criminal court proceeding with defense only and no prosecution?

Research can be useful if you understand the beast. I spent years in big agencies (with big research budgets) being told by researchers that certain TV ideas will work and others will fail. The fact is, almost every time a researcher was convinced we had it cracked -for all kinds of reasons on so many levels, the ad turned out to be ineffectual wallpaper. Lesson: if it comes out brilliantly in research, don’t go near it.

From the first ancient king who went to consult some senile folks pushing around their dead animals entrails – to the naked emporer asking how his clothes looked – to the modern habit of overstating the value of their research……a fool is till easily parted from their money.

@John Grono – #24

fair point – there were no actual research companies involved – just an ad agency and some unethical, but highly successful senior execs showing the new boy the ropes. I have never actually worked at any agency that did not bilk the client in some way. It was actually not until I went client side that I came across companies that did the right thing.

@Louie the Fly #25

No, as far as I know the CEO never got busted nor the GAD. This was a big firm and they were raking in the dough I believe.

You know you have written a great piece that cuts close to the bone when it’s attacked so heavily.

Seems like a lot of people want to protect their sham.

I think Anonymous is a fake with no understanding of the Market and Social Research profession.
I’m with Peter Harris – whoever you are, ‘Anonymous’ you clearly have no real idea of how serious market research works. If you do work in a research firm, I doubt that you have a position of any seniority or influence given your incomplete and one dimensional description of the topic.
We are all out here working our arses off solving problems, upholding good practice and taking on the McCrindles of this world. I suggest you learn some manners and stop bagging a profession you clearly have no serious understanding of. And give your real name if you really can back up a shred of what you say.
My real name is…
Martin O’Shannessy
CEO Newspoll

Go Martin!

As I said last week: anonymous=gutless

I think a CEO embarrassing himself publicly with a bog-standard internet teenager ad-hominem attack rather than a well reasoned argument speaks volumes about the respectability of the organisation he represents.

And I think “Interesting” has not added anything to the discussion. And frankly, until you have the guts to use your real name and put up some kind of argument you have no credibility. You don’t even say what your job is. Maybe when you do, other people will be able to anonymously post a bunch of critical and unsubstantiated bull about your industry. Until then you might consider the old adage ‘put up or shut up’.
Martin O’Shannessy.

You get what you pay for. I’ve been doing this almost 30 years and have helped many business keep customers, get new ones, focus improvement spending and test new products. There is a lot more subtlety to it that Circling Sharks seems to appreciate. Most of the research he describes which I have observed is from clients trying DIY or asking a junior staff member to design a survey. Yes, garbage in, garbage out. But with clever design, rigorous sampling and astute analysis, research can still be a great investment. I know – my clients keep coming back.