Thursday 29 November 2012

How to rig a (Leveson) survey

The Leveson report is released today, but I'm not sure why they're bothering as everybody's already decided  on their favoured solution. The rhetoric has been ramping up for weeks, as newspapers try to apply pressure for light touch (or ideally, no) regulation and those wronged by newspapers in the past, campaign for a tough new regulator.

Guess which side the Daily Mail is on?

I normally try to avoid the Daily Mail website (thanks Kittenblock), but via Twitter today an article caught my eye. Hugh Grant has been doing the rounds with a statistic that 79% of the UK public wants an independent press regulator, with a statutory underpinning.

The Mail says this statistic is "suspect". (turn Kittenblock off for that link...)

I do like a suspect statistic, so let's have a look.



The Media Standards Trust ran a survey that asked:

"There should be an independent body, established by law, which deals with complaints and decides what sanctions there should be if journalists break agreed codes of conduct."
79% agree with that one.

And for The Sun:

"Who would you most like to see regulate newspapers and the press?: A regulatory body set up through law by Parliament, with rules agreed by MPs"
Now 24% agree.

Both polls were conducted by YouGov. What the hell?


The Daily Mail says that:

The key to the MST [Media Standards Trust; the 79% one] poll result is the use of the word 'independent' in the first question but not in the second.


True. The Mail's spun that statement heavily though, because although the second question doesn't say "Independent", it does say, "with rules agreed by MPs".

Both sides have rigged the question in their favour, by using wording which emphasises the positive or negative aspects of a statutory regulator, depending on their own point of view. The Media Standards Trust plays up "independent", while The Sun draws attention to potential political interference with a free press.

As much as I hate to say it - because personally, I do think we need a statutory underpinning - the Media Standards Trust question is probably worse than The Sun's poll, because it suffers from acquiescence bias. This is the tendency for survey respondents to answer positively to a question that asks for agreement.

"There should be", invites a strong, positive response. Disagreeing requires more thought and a stronger opinion than saying, "yes, ok, that's probably true."

The huge difference between responses to these two questions illustrates the enormous difficulties present in polling on emotive issues and the ease with which a survey can be rigged to produce the answer you want. Vic Reeves famously said, "88.2% of statistics are made up on the spot". He was wrong though - you don't need to make them up. Ask the right question and people will give you the answer you wanted.

Here's how it's done by a pro. I know I've linked this before, but it's brilliant.