In Defence of Liberty

Driven by data; ridden with liberty.

Where We Start

There is a noticeable and substantial disparity between polls for the upcoming referendum on Britain’s membership of the European Union, depending on whether those polls are conducted via the telephone or through an online panel. This disparity has been studied in Polls Apart [1], which was conducted by Matt Singh of Number Cruncher Politics and James Kanagasooriam of Populus [2].


Most of the disparity comes from the lack of Don’t Know prompting in telephone polls and sampling differences in terms of social liberalism. (Source: Populus)

Their study found that there were two major causes for the polling disparities: the suppression of ‘Don’t Know’ responses in phone polling, and of sampling differences in social attitudes between telephony respondents and online panels. Of the 16 percentage point gap between the Populus polls, these two causes account for 13 points. A three point difference still remains, which the authors suggest is due to the lack of “respondents who know exactly what their views are and who back these up with internally consistent opinions when asked for them” in online samples.

“Almost monolithic support”

An editorial post over at The Commentator asserts that — in the article entitled ‘Why EU Remain needs an 8-10 point poll lead to win — “Remain has more cause to be nervous than Brexit, despite a slight lead in the poll of polls”, because [3]:

1) Everyone agrees that, as things stand, turnout is going to be higher among Brexiters. No real surprise there. But what has not been properly considered by most commentators is the following;

2) With almost monolithic support for Remain among (domestic and foreign) establishment organisations — the PM, the leader of the opposition, the unions, the establishment think tanks such as Chatham House, the CBI, the IMF, President Obama and so on and so forth — there is likely to be a significant number of respondents to opinion polls who simply won’t be forthcoming about their real itnentions if they intend to vote for Brexit.

After stating it is “extremely difficult to quantify” this supposed effect of deliberate misreporting, the article continues:

This is from the Guardian today:

“A new Guardian/ICM telephone poll, conducted over the weekend, puts the remain on 54% and leave on 46%, while a second poll – conducted online – suggests a dead heat”.

Interesting. So an eight point lead evaporates into nothing when the methodology changes in favour of the more anonymous — and therefore less threatening — approach.

It’s back of the envelope stuff, of course, but when you take into account both turnout and the greater reluctance of Brexiters to pollsters the truth, we reckon that Remain can’t be sure of winning unless it goes into the referendum with an 8-10 point lead in the combined opinion polls.

The Commentator’s author is asserting that the more equal share between Remain and Leave in online polls is due to deliberate misreporting in telephony polls, thanks to the privacy of the former method.

This is simply incorrect, based upon the Polls Apart study, and ignores other differences between online and telephone polls.


Polling averages show a close race. (Source: Guardian)

When I highlighted this on the social media website Twitter — as well as the fact they did not cite the Populus study — The Commentator’s account responded:

No-one knows what is “correct”. The best available analysis, cited in the piece, needs a good counter argument. Got one?

It is strange to describe the Guardian article as an “analysis” of the polling disparity, let alone the “best available” such analysis. The only consideration of polls was in a single paragraph [4]:

The intensity of the debate is likely to increase as opinion polls point to an uncomfortably tight referendum for the chancellor and prime minister. A new Guardian/ICM telephone poll, conducted over the weekend, puts the remain on 54% and leave on 46%, while a second poll – conducted online – suggests a dead heat.

This is not an analysis of the polling disparity, but merely a statement of it.

Late swings and sampling problems in 1992

I then cited the Polls Apart study, with a link, which garnered the following response:

Psychological suppression differences are hard to measure. But, it’s a reason why polling gets it wrong. Major 1992…

Again, this is strange. If “psychological suppression differences are hard to measure”, then why assert that it is the primary factor in the differences between online and telephony polling for the EU referendum?

Privacy is not the sole — or even main — difference between these two types of polling. The effect of misreporting on the 1992 election is also commonly overstated.

The National Centre for Research Methods conducted their analysis of the pre-election polling, concluding that the three main causes were a late swing, inadequacies in quota sampling, and greater item refusal (either refusing to answer the voting intention question and saying ‘Don’t Know’) by Conservative supporters [5].


There were three main causes for the polling miss in 1992. (Source: NCRM)

They also concluded:

We do not believe that deliberate lying to pollsters occurred to any significant extent.

Political comment is often swamped with assertions, at divergence with the available data.


[1] Masters, A., 2016. Polls Apart. In Defence of Liberty. Available from: [Accessed: 22nd April 2016]

[2] Singh, M., and Kanagasooriam, J., 2016. Polls apart. Populus. Available from: [Accessed: 20th April 2016]

[3] The Commentator, 2016. Why EU Remain needs an 8-10 point poll lead to win. Available from: [Accessed: 20th April 2016]

[4] Asthana, A., Clark, T., and Watt, N., 2016. EU referendum: Remain camp treat public like ‘children’, says Gove. Guardian. Available from: [Accessed: 20th April 2016]

[5] NCRM, 1994. The Opinion Polls and the 1992 General Election. Available from: [Accessed: 20th April 2016]



This entry was posted on April 25, 2016 by in European Politics and tagged , .
%d bloggers like this: