Rigging the Scottish Independence vote

Yesterday, Alex Salmond released his preferred wording for the question that will decide whether Scotland should remain as part of the UK.

"Do you agree that Scotland should be an independent country?"

And immediately, a cross-section of the research community cried foul.

Anyone who works in marketing research will be pretty familiar with the best ways to rig a survey question; it's how PR companies get exciting sounding press releases to plant their client's name in the newspapers. In a previous job, we "proved" that British women would swap a shopping spree for a night of passion and bagged the Daily Star front page under the headline, "Sex? I'd rather go shopping." Was it true? Frankly, who cares? Probably not, but it was a PR fluff piece that got us loads of free publicity and the survey was designed to produce exactly those kind of answers.

One of the more subtle ways to rig a survey is to ask people to agree with something. When in doubt, respondents have a tendency to agree with a statement, particularly if it's a complicated concept that they don't understand, or if they don't really care either way (which is handy if you're trying to rig a PR survey.) It's called Acquiescence bias and Wikipedia explains the issue well, with a few examples.

The Guardian today gives an example that acquiescence bias could easily create a 9% swing in the response to a positively worded question. That's a lot and could easily decide a tight vote.

The first thing I'd want to do with that question above, is to get rid of the word "agree", which loads it towards a "yes" response.

"Should Scotland should be an independent country?"

Yes, or no? That's much better.

Better still, would be not to demand a yes / no response at all. There's still an issue with "Should Scotland..." because you could ask:

"Should Scotland be an independent country?"

Or... 

"Should Scotland remain part of the UK?"

You're not asking for agreement, but there's still an element of potential bias. Actually, this time the option to "remain" is likely unfair as it invites respondents stick with the status quo, which they will have a tendency to do, when in doubt.

You could argue that this is pedantry (fun though, isn't it? And if nothing else, you know how to rig a survey now) but for me, it's very important. The one thing that you don't want from a referendum is the possibility that the answer is ambiguous and can be challenged. Alex Salmond's preferred question undoubtedly can be. 

For the same reason, politicians shouldn't be allowed to avoid questions about what they will do, for example if the vote is very close, by saying "it's hypothetical". Yes it is, but it's very, very important and we need to know up front what we'll do in that situation, not to argue the meaning of the result afterwards.

The best way to ask about independence is an option that won't sit well with politicians at all, because it isn't a yes or no question.

Which of these would you prefer for Scotland?

1. To be a country within the UK

2. To be an independent country that is not part of the UK

And if you're going to be really thorough, rotate the answers so that remaining in the UK only appears in the top slot half of the time.


Don't get me started on the suggestion of third options and "Devo-max". What are you going to do if they come out with 33% of the vote each? Have a bloody great row, that's what. Which is exactly where we're headed.

And finally... if you really want to know how to rig a survey, ask the experts.