In Defence of Liberty

Driven by data; ridden with liberty.

The British Polling Council Inquiry Report

The British Polling Council and the Market Research Society established an independent inquiry [1], led by Professor Patrick Sturgis, the Director of the National Centre for Research Methods at the University of Southampton [2].

(Video: BBC News)

Using raw data provided by the polling companies themselves, the report considers eight hypotheses for the polling error: postal votes, overseas voters, voter registration, question wording and framing, a late swing, deliberate misreporting, turnout weighting and unrepresentative samples. The full report expanded upon the preliminary report, with more detail [3].

Historical precedent

It should be noted that the polling error observed in 2015 is not wholly unprecedented: it is the largest minimum mean absolute error since the 1992 General Election, but the average mean absolute error was only slightly higher than in 1997. The polls were not entirely wrong either: suggesting accurate percentages for the Liberal Democrats, UKIP and the Green Party.

bpc-report-1.JPG

Polls have been this wrong before. (Source: NCRM)

Polling in Great Britain, between 1945 and 2015, tends to overestimate left-wing parties by 1.5 percentage points and underestimate right-wing parties by 1.3 percentage points. This can be observed in other countries, but the effect is generally smaller. The polling error in 2015 was not good, but also not particularly bad: “underestimating the size of a landslide is considerably less problematic than getting the result of an election wrong”.

Postal votes, overseas voters, voter registration, question wording

The polls included the same percentage of postal votes in their samples to the actual election. In 2015, postal votes constituted 20.9% of all votes cast, and postal votes comprised an average of 21% of the weighted samples. Overseas voters were 0.2% of the eligible electorate, so it could not make a discernible difference to the polling miss. Functionally, voter registration error — if people told polling companies how they intended to vote and then found they were not registered — operates the same as other forms of voter turnout misreporting.

In the British Election Study online panel, the Conservative vote proportion was unrelated to the placement of the question in the survey. Asking a constituency-specific question rather than the general voting intention question does not have a uniform effect, working better in some constituencies and not others. The report concludes that question wording and order did not contribute to the polling error.

Late swing

Another putative cause is the late swing. Since some voters do not say how they are going to vote, or may change their mind between the last poll and the polling booth, this could be a source of error. By looking at re-contact surveys, there is a weak tendency for post-election estimates to show a larger lead for the Conservatives, but this is inconsistent across polls.

The effect of allocating people who gave non-substantive responses (don’t know, or refused to answer) was negligible on estimates of the Conservative lead: the largest change was one percentage point.

Deliberate misreporting: ‘Shy Tories’

A commonly-cited reason for the polling miss is deliberate misreporting. This effect is associated with ethnicity in the United States, with some white voters telling polling companies they intend to vote for black candidates in their preferred party, despite having no real desire to vote for them, to avoid appearing prejudiced.

Crudely, the idea behind ‘shy Tories’ is that voting Conservative is seen as selfish or otherwise tainted, and so some people may not wish to tell pollsters of their true intentions. In the British context, this reasoning should not be wholly limited to the Conservatives: deliberate misreporting against the Conservatives would have to outweigh any deliberate misreporting against other parties to explain the polling miss.

The shift observed between pre-election and post-election estimates largely came from respondents who did not disclose a particular party choice. It does not explain how the British Election Study and British Social Attitudes survey managed to estimate the Conservative vote share as slightly above the actual result.

bpc-report-4.JPG

The Conservative vote share in BES and BSA was higher than the actual result. (Source: NCRM)

The Conservative lead has also been slightly higher in telephone polls since 2011, which involve more human contact than selecting a radio button on a website. Whilst it is very difficult to completely rule out sustained misreporting by Conservative voters, it is inconsistent with the results of re-contact surveys and the random probability sampling surveys undertaken after the election. The report concludes: “We reject deliberate misreporting as a contributory factor in the polling miss.”

Turnout weighting

The turnout weighting is a claimed clause. Polling companies relied heavily on a likelihood-to-vote question. Responses to “how likely is it that you will vote in the general election on 7th May?” were typically placed on a scale from 1 to 10, with 1 means certain not to vote and 10 means certain to vote. The relationship between the turnout weights and the turnout probabilities in the poll sample showed that it is not a good approximation.

bpc-report-2.JPG

This is the reported turnout probability in re-contact surveys. (Source: NCRM)

However, changing the turnout model has very little effect on the published estimate of the Conservative lead over Labour. It is suggested that voter intention might be affecting turnout: the ‘Lazy Labour’ hypothesis. The report states: “However, the re-contact surveys showed no evidence of differential turnout misreporting.”

bpc-report-3.JPG

Alternative turnout probabilities are considered. (Source: NCRM)

If vote intention were added to a turnout probability model, it is only statistically significant for one pollster, YouGov, and in the opposite direction to explain the polling miss. Given their answer to the likelihood-to-vote question, Labour voters were more likely to vote than suggested by their response.

Unrepresentative samples

Given the other seven causes have been eliminated, what remains — unrepresentative samples — is the primary cause. The report offers direct evidence in favour of this hypothesis. The British Social Attitudes survey and the British Election Study managed to get much closer to the actual result in their post-election surveys, based on a multi-stage, stratified probability sample of addresses drawn from the Post Office Address File. Multiple calls are made to the chosen person in each selected address.

Sampling procedures differ in most polls: quotas of specified demographics of people are made, and then the online database or telephony bank are run through until those quotas are hit.

bpc-report-5.JPG

Polls underestimated the Conservative lead in every age segment. (Source: NCRM)

The election polls were also quite different in the separate segments, whether comparing age versus the BES-BSA, or by region against the actual vote counts.

bpc-report-6

Polls underestimate Conservative leads in strongly Conservative places, such as the South East. (Source: NCRM)

Improving polls

There is no simple, single way to improve polling. The inquiry’s report listed twelve recommendations. The first three concern some methodological inadequacies: asking a question in the short campaign about whether someone had voted by post, reviewing turnout probability models, and reviewing current allocation methods for non-substantive responses.

The fourth is not prescriptive, but urges pollsters to “take measures to obtain more representative samples within the weighting cells they employ”, such as longer fieldwork periods, and more call-backs. The fifth recommendation wants an investigation into new quota and weighting variables, though recognising this may not be fruitful.

The sixth recommendation wants funding from the Economic and Social Research to fund a pre-election random probability survey as part of the British Eleciton Study in the 2020 election campaign. The next three recommendations are about transparency: explicit statements about how the data was weighted, what statistical adjustments are made, and a commitment to release anonymised poll micro-data at the British Polling Council’s request. To ameliorate herding, the tenth recommendation is for the pre-registration of vote intention polls.

The final two recommendations are about communication: the provision of confidence (or credible) intervals for each party in the headline share of the vote, and statistical tests for changes in vote share for all listed parties to the previous published poll.

Understanding of polls must evolve beyond the belief they are always correct against they are always wrong [4]. There are limitations to polling, but it is the best estimate we have of public opinion.

References

[1] British Polling Council, 2016. British Inquiry Report. Available from: http://www.britishpollingcouncil.org/bpc-inquiry-report/ [Accessed: 31st March 2016]

[2] Sturgis, P., Baker, N., Callegaro, M., Fisher, S., Green, J., Jennings, W., Kuha, J., Lauderdale, B., and Smith, P., 2016. Report of the Inquiry into the 2015 British general election opinion polls. London: Market Research Society and British Polling Council. Available from: http://eprints.ncrm.ac.uk/3789/1/Report_final_revised.pdf [Accessed: 31st March 2016]

[3] Masters, A., 2016. Statistics and Lampposts XXIV: Differential Non-Response. In Defence of Liberty. Available from: https://anthonymasters.wordpress.com/2016/01/18/statistics-and-lampposts-xxiv-differential-non-response/ [Accessed: 31st March 2016]

[4] Wells, A., 2016. What the BPC inquiry’s final report says. UK Polling Report. Available from: http://ukpollingreport.co.uk/blog/archives/9662 [Accessed: 31st March 2016]

Advertisements

One comment on “The British Polling Council Inquiry Report

  1. Pingback: Polls Apart | In Defence of Liberty

Comments are closed.

Information

This entry was posted on April 4, 2016 by in National Politics and tagged , , , .
%d bloggers like this: