No Evidence of Effect Is Not Evidence of No Effect

The points below about the asymmetry between absent and negative evidence are ordered by how much difference they make, not by how often they get repeated.

The distinction stated plainly

No evidence of effect means nobody has demonstrated one, which can result from nobody having looked properly. Evidence of no effect means someone looked with adequate methods and found the effect is absent or too small to matter.

The first is a statement about the state of research and the second is a statement about the world. Conflating them allows a seller to dismiss criticism and allows a critic to overstate a refutation. Keeping them separate is the single most useful habit for reading arguments about health claims.

Why small studies produce empty results

Statistical power is the probability that a study would detect an effect of a given size if it exists. Small samples have low power, so a genuine moderate effect can easily fail to reach significance in them. A study of twenty people reporting no significant difference has usually shown that twenty people was not enough.

Reporting such a result as proof that something does not work misrepresents what the study could have found. Good papers state the effect size they were powered to detect, which tells you what a null result rules out.

Reading the interval instead of the verdict

A confidence interval gives a range of effect sizes compatible with the data, which is far more informative than significant or not. A null result with a narrow interval close to zero genuinely constrains how large any effect could be. A null result with an interval spanning large benefit and large harm has established almost nothing.

Against the trial data, both are reported as no significant difference, which is why the phrase alone should never end an argument. Looking at the interval takes a few seconds and changes the interpretation of a great many papers.

When absence does become informative

If a claim has been investigated repeatedly with adequate methods and nothing consistent has emerged, that pattern is meaningful. A claim old enough and lucrative enough to have attracted serious investigation, with no result, has failed a real test. The strength of the inference depends on how hard and how competently anyone looked.

Once you look at who funded it, for a mechanism that would produce large obvious effects, the absence of any detection is more telling still. This is how established scepticism about several long-marketed products was reached, and it was not reached quickly.

How both sides misuse it

Sellers say the absence of evidence merely means science has not caught up, which is occasionally true and usually convenient. Critics say a null trial proves a product does nothing, which overstates what a single underpowered study establishes. The honest positions are that it has not been shown to work and that we do not yet know, and they are different positions.

Being able to say we do not know without embarrassment is what separates scepticism from contrarianism. A site that claimed certainty in both directions would be as unreliable as the marketing it criticises.

Unproven is not the same as harmful, and neither is the same as useless — the three get collapsed constantly.

What follows practically

Not proven does not mean forbidden, and people can reasonably try low-risk unproven things with their eyes open. It does mean the cost, the risk and the opportunity cost fall on the buyer rather than on any evidence.

For anything with meaningful risk, or replacing an established treatment, unproven should mean not yet. Never delay diagnosis or established treatment for something in the not-yet-shown category, whatever its promise. The asymmetry is the point: absence of proof licenses curiosity and never licenses a health claim.

Everything above, in order of what to do first

1. The distinction stated plainly. No evidence of effect means nobody has demonstrated one, which can result from nobody having looked properly.
2. Why small studies produce empty results. Statistical power is the probability that a study would detect an effect of a given size if it exists.
3. Reading the interval instead of the verdict. A confidence interval gives a range of effect sizes compatible with the data, which is far more informative than significant or not.
4. When absence does become informative. If a claim has been investigated repeatedly with adequate methods and nothing consistent has emerged, that pattern is meaningful.
5. How both sides misuse it. Sellers say the absence of evidence merely means science has not caught up, which is occasionally true and usually convenient.
6. What follows practically. Not proven does not mean forbidden, and people can reasonably try low-risk unproven things with their eyes open.

The takeaway

Nobody has shown it and it has been shown not to work are different sentences. Check the interval before you believe either one.

The body already has organs for this, and none of them are sold in a box.