Driven by data; ridden with liberty.
American sports-caster Vin Scully famously said: “Statistics are used much like a drunk uses a lamppost: for support, not illumination.” Given the prevalence of statistics, particularly in economics and politics, statistical literacy is increasingly important.
Political polls of public support for parties and policies are regularly undertaken, with YouGov performing five polls on such issues every week. The true percentage of the public who back or reject particular policies is unknown, as it is unfeasible to survey every adult. Instead, we survey small subsets of that population, aiming to discover the public’s political opinions. Central estimates are published by the polling companies, but what it is often unreported is each poll’s error margins.
For political polling, the margin of error is usually calculated at the 95% confidence level. Each poll will produce a central estimate with a margin of error. Given 20 polls, we expect the true percentage to fall in the range of values defined by the 20 central estimates plus or minus their margins of error 19 times – and to fall outside it once. These intervals are called confidence intervals.
With a sample size of about 2,000 people, the margin of error is about 2 percentage points. As an example, if the true support for a political party was 30%, 95% of polls would centrally estimate that party’s support between 28% and 32%. If a 95% confidence interval is already calculated, it is incorrect to say that there is a 95% chance that the true value lays within that interval. The true value is a fixed number, not a random variable, so it either lies in this interval or it does not. To repeat, what these 95% confidence intervals mean is that, in the long run, the true value will lay in these confidence intervals 95% of the time.
These error margins blur the clarity given in much news and commentary on opinion polls. A poll that shows a particularly low lead or a bubble in a small party’s support is often discussed – usually as the harbinger for a new political period. It is natural for news outlets to focus on the unusual and extraordinary. However, it is implicit that some polls will produce estimates that wildly differ from true public opinion, and consequently from most other polls. It is possible that a shock poll might be the beginning of a new trend, but that is not always the case. Differences in advocacy between two parties are especially erratic, as estimates for both parties are subject to errors – even if no true change has occurred. Reading deeply into a single poll can be foolish, as that poll could be an outlier.
This is why poll aggregators, such as the UK Polling Report – which collate all current polls – are becoming popular and respected. Nate Silver’s elections model is a type of poll aggregator, which weighs more accurate polls more heavily than inaccurate ones. Regular polls are a great insight, but be aware of the margins of error.