How to rig a (Leveson) survey

The Leveson report is released today, but I'm not sure why they're bothering as everybody's already decided  on their favoured solution. The rhetoric has been ramping up for weeks, as newspapers try to apply pressure for light touch (or ideally, no) regulation and those wronged by newspapers in the past, campaign for a tough new regulator.

Guess which side the Daily Mail is on?

I normally try to avoid the Daily Mail website (thanks Kittenblock), but via Twitter today an article caught my eye. Hugh Grant has been doing the rounds with a statistic that 79% of the UK public wants an independent press regulator, with a statutory underpinning.

The Mail says this statistic is "suspect". (turn Kittenblock off for that link...)

I do like a suspect statistic, so let's have a look.

The Media Standards Trust ran a survey that asked:

"There should be an independent body, established by law, which deals with complaints and decides what sanctions there should be if journalists break agreed codes of conduct."
79% agree with that one.

And for The Sun:

"Who would you most like to see regulate newspapers and the press?: A regulatory body set up through law by Parliament, with rules agreed by MPs"
Now 24% agree.

Both polls were conducted by YouGov. What the hell?


The Daily Mail says that:

The key to the MST [Media Standards Trust; the 79% one] poll result is the use of the word 'independent' in the first question but not in the second.

True. The Mail's spun that statement heavily though, because although the second question doesn't say "Independent", it does say, "with rules agreed by MPs".

Both sides have rigged the question in their favour, by using wording which emphasises the positive or negative aspects of a statutory regulator, depending on their own point of view. The Media Standards Trust plays up "independent", while The Sun draws attention to potential political interference with a free press.

As much as I hate to say it - because personally, I do think we need a statutory underpinning - the Media Standards Trust question is probably worse than The Sun's poll, because it suffers from acquiescence bias. This is the tendency for survey respondents to answer positively to a question that asks for agreement.

"There should be", invites a strong, positive response. Disagreeing requires more thought and a stronger opinion than saying, "yes, ok, that's probably true."

The huge difference between responses to these two questions illustrates the enormous difficulties present in polling on emotive issues and the ease with which a survey can be rigged to produce the answer you want. Vic Reeves famously said, "88.2% of statistics are made up on the spot". He was wrong though - you don't need to make them up. Ask the right question and people will give you the answer you wanted.

Here's how it's done by a pro. I know I've linked this before, but it's brilliant.

Rigging the Scottish Independence vote

Yesterday, Alex Salmond released his preferred wording for the question that will decide whether Scotland should remain as part of the UK.

"Do you agree that Scotland should be an independent country?"

And immediately, a cross-section of the research community cried foul.

Anyone who works in marketing research will be pretty familiar with the best ways to rig a survey question; it's how PR companies get exciting sounding press releases to plant their client's name in the newspapers. In a previous job, we "proved" that British women would swap a shopping spree for a night of passion and bagged the Daily Star front page under the headline, "Sex? I'd rather go shopping." Was it true? Frankly, who cares? Probably not, but it was a PR fluff piece that got us loads of free publicity and the survey was designed to produce exactly those kind of answers.

One of the more subtle ways to rig a survey is to ask people to agree with something. When in doubt, respondents have a tendency to agree with a statement, particularly if it's a complicated concept that they don't understand, or if they don't really care either way (which is handy if you're trying to rig a PR survey.) It's called Acquiescence bias and Wikipedia explains the issue well, with a few examples.

The Guardian today gives an example that acquiescence bias could easily create a 9% swing in the response to a positively worded question. That's a lot and could easily decide a tight vote.

The first thing I'd want to do with that question above, is to get rid of the word "agree", which loads it towards a "yes" response.

"Should Scotland should be an independent country?"

Yes, or no? That's much better.

Better still, would be not to demand a yes / no response at all. There's still an issue with "Should Scotland..." because you could ask:

"Should Scotland be an independent country?"

Or... 

"Should Scotland remain part of the UK?"

You're not asking for agreement, but there's still an element of potential bias. Actually, this time the option to "remain" is likely unfair as it invites respondents stick with the status quo, which they will have a tendency to do, when in doubt.

You could argue that this is pedantry (fun though, isn't it? And if nothing else, you know how to rig a survey now) but for me, it's very important. The one thing that you don't want from a referendum is the possibility that the answer is ambiguous and can be challenged. Alex Salmond's preferred question undoubtedly can be. 

For the same reason, politicians shouldn't be allowed to avoid questions about what they will do, for example if the vote is very close, by saying "it's hypothetical". Yes it is, but it's very, very important and we need to know up front what we'll do in that situation, not to argue the meaning of the result afterwards.

The best way to ask about independence is an option that won't sit well with politicians at all, because it isn't a yes or no question.

Which of these would you prefer for Scotland?

1. To be a country within the UK

2. To be an independent country that is not part of the UK

And if you're going to be really thorough, rotate the answers so that remaining in the UK only appears in the top slot half of the time.


Don't get me started on the suggestion of third options and "Devo-max". What are you going to do if they come out with 33% of the vote each? Have a bloody great row, that's what. Which is exactly where we're headed.

And finally... if you really want to know how to rig a survey, ask the experts.