93% of Surveys are Meaningless

Over at Bad Science, Ben Goldacre decries an article about a spurious "study", lifted straight from a corporate press-release, in his own newspaper The Guardian:

On Monday we printed a news article about a “report” “published” by Nuffield Health, headlined “No sex please, we’re British and we’re lazier than ever”. “This is the damning conclusion of a major new report published today,” says the press release from Nuffield ... I asked Nuffield’s press office for a copy of the new report, but they refused, and explained that the material is all secret. The Guardian journalist can’t have read it either. I don’t really see how this “report” has been “published”, and in all honesty, I wonder whether it even exists, in any meaningful sense, outside of a press release.

Nuffield Health are the people who run private hospitals and clinics which you can’t afford....the Guardian gave free advertising to Nuffield, for their unpublished published “report”, which nobody even read, in exchange for 370 words of content. This is endemic, and it creeps me out.

The Telegraph also reprinted the press release; sorry, wrote an article drawing on the press-release amongst many other carefully-research sources. The other papers probably did too; I'm too lazy to check.

For you see, the alleged study found that British people are monumentally slothful: 73% of couples said that they are "regularly" too tired to have sex while 64% of parents say that they are always "too tired" to play with their children, and so on.

Yes, according to Nuffield Health, only one in three British parents ever play with their own children. The rest are always too exhausted. It's a wonder they found 2,000 people who were awake enough to answer their survey - although, as Goldacre says, maybe they didn't.

This "study" is, obviously, bollocks. It serves only as advertising for Nuffield Health's network of fitness centers, the benefits of which are helpfully listed at the end of the press release. That newspapers regularly reproduce press releases because they can't afford to pay journalists to fill the space any other way is well known nowadays. This is thanks mostly to Nick Davies and his outstanding book Flat Earth News which revealed, in great detail, just how bad things have got.

But the fact that this advert was published in the Health section of The Guardian, is more than just a symptom of the decline of British journalism. It also reflects the peculiarly British obsession with "surveys".

Even if the Nuffield data was fully published in a proper journal, and even it had been a survey of 200,000 people, it would still be meaningless. Asking people whether they are lazy is not a good way of finding out whether they are, in fact, lazy. All it can tell you is whether people think of themselves as lazy, which is very different. If you wanted to prove that British people really were lazy and getting lazier, you would have to look at actual indicators of activity like, say, gym membership rates, or amateur sports team participation, or swimming pool use, or condom sales if you really think people are too tired have sex, etc.

Yet surveys like this seem to be almost mandatory if you want to draw attention to your cause in Britain at present. You have to do one, and you have to massively over-interpret the results. The gay rights group Stonewall this week accused British football of being institutionally homophobic. Their basis for this claim was a survey of - guess - 2,000 football fans, finding, amongst other things, that

Only one in six fans said their club was working to tackle anti-gay abuse and 54% believed the Football Association, Premier League and Football League were not doing enough to tackle the issue.

This survey demonstrates, at best, that many football fans think British football is institutionally homophobic. It does not "Sadly demonstrate that football is institutionally homophobic", as a Stonewall spokesman said, unless you think that British football fans are infallible godlike beings.

I have nothing but sympathy for Stonewall, and they may well be right about homophobia in football. But their survey is meaningless. It's advertising, just like Nuffield Health's survey. Attentive Neuroskeptic readers will remember the case of "In The Face of Fear", yet another survey of about 2,000 people, claiming that Britain is in the grip of an epidemic of anxiety disorders (it's not) and serving as advertising for another well-meaning group, the Mental Health Foundation.

A large and growing proportion of British newspaper articles are essentially promotional material for some kind of company, charity, or activist organization. Honestly, newspapers should just go the whole hog and replace half their pages with paid adverts, and use the money earned to pay their journalists to actually do some journalism. There would only be half as much news, but it would at least be news.

[BPSDB]