In Defence of Liberty

Driven by data; ridden with liberty.

On Anecdotal Evidence

It is the tale we tell ourselves. (Edited: physiognomist)

It is the tale we tell ourselves. (Edited: physiognomist)

In The Problem of Induction, philosopher Karl Popper wrote:

In the empirical sciences, which alone can furnish us with information about the world we live in, proofs do not occur, if we mean by ‘proof’ an argument which establishes once and for ever the truth of a theory.

When discussing empirical matters, it is common to observe the citation of personal experience, or the single experiences of others, as supportive evidence for generalised claims. We live without proof, and the power of stories to influence and persuade may help explain why anecdotal evidence persists in our reasoning.

Single anecdotes, or a small collection thereof, are an unreliable form of substantiation. When a person recounts their anecdote, it is not known whether such stories are part of the majority or the minority, part of the test or the control, part of the typical experience or extraordinary.

Cognitive Biases

Human memory is fallible and alterable. Tidbits become tales, tales become sagas, and sagas become legends. Confirmation bias means we are prone to seek and interpret information in a manner conducive and consistent with pre-existing beliefs. Subjective validation means people are more likely to accept a statement as being true if it is directly addressed to them, or of personal significance. The availability heuristic is a mental shortcut, where people form judgments based on what is more immediately brought to their mind.

From this small sub-sample, we may then claim to estimate characteristics and outcomes of the studied population. This bias is usually called the law of small numbers. There is also reporting bias: the unusual, the strange and the exceptional are more likely to highlighted, whilst those are unable or unwilling to report cannot be counted. For instance, a survey of Russian Roulette players would not imply the game is safe, despite every respondent surviving the game.

These are cognitive biases, overlaid as lenses above a mosaic of truth. These limitations in reasoning hold regardless of the veracity of the anecdote itself.

This is not to say that anecdotes are without worth: anecdotes can be illuminating — a guiding light towards new hypotheses. A company may consider the testimonials — both positive and negative — of customers about its products, fostering further research. Doctors (and other trained observers) can publish their anecdotes as case studies in medical journals, where the written statements are subjected to peer reviews.

Anecdotes cannot be used as a robust ballast for beliefs.



This entry was posted on March 8, 2015 by in Law and Philosophy and tagged .
%d bloggers like this: