What a Study of Twelve People Can and Cannot Show

Treat the sections below as a sequence. With sample size and what it determines, getting the early decisions right makes the later ones much easier.

What small studies are legitimately for

Early studies check whether a protocol is workable, whether people will complete it and whether obvious safety problems appear. They can establish that a compound is absorbed, that a measurement is feasible, and roughly what direction an effect might take. That is genuinely useful work and it is how larger studies get designed sensibly rather than expensively.

The researchers running them usually describe them accurately as pilot or preliminary work in the paper itself. The distortion generally occurs afterwards, in the press release and the product page rather than in the research.

Why the estimate is unstable

With few participants, one unusual individual can shift the average result substantially in either direction. People differ enormously in baseline health, diet, genetics and behaviour, and small groups do not average that out.

Follow the mechanism and the confidence interval around a small study effect is correspondingly wide, often spanning benefit and harm together. Repeating the same small study would frequently produce a noticeably different answer through chance alone. That instability is a property of the design rather than a flaw in the execution.

Chance findings and why they get published

A study measuring several outcomes in a small group has a good probability of finding something impressive by accident. Impressive accidental findings are more likely to be written up, submitted, accepted and publicised than null results.

On the label, the result is a literature in which small striking findings are overrepresented relative to how often they are true. When larger studies follow, effects frequently shrink or disappear, which is a well-documented pattern across fields. A dramatic result from a small study is therefore weak evidence rather than strong evidence, which inverts the intuition.

Designs that make small studies stronger

Crossover designs, in which each participant receives both treatment and comparison, use each person as their own comparison. That removes between-person variation and can extract more information from a small group. Proper randomisation and blinding matter more in small studies, not less, because a single unbalanced factor distorts more.

Against the trial data, preregistering the primary outcome prevents the multiple-comparison problem that small studies are most vulnerable to.

A well-designed small study is worth considerably more than a large but poorly controlled one.

Why marketing prefers them

They are affordable, they finish quickly, and they can be commissioned repeatedly until one produces a usable result. A cited study on a product page is rarely accompanied by any mention of how many people took part. The word clinical does a great deal of work here, since a clinical study can involve a dozen volunteers for four weeks.

Once you look at who funded it, the word trial similarly implies rigour that the design may not have included at all. Looking up the participant count is the fastest way to assess a citation you have been shown.

Never stop or alter a treatment a clinician has prescribed on the strength of something you have read.

What to ask about any cited trial

How many participants, for how long, and compared against what. Were participants randomly assigned, and did anyone know which group they were in.

Follow the mechanism and was the reported outcome declared in advance or selected from among several measured. Has anyone independent reproduced it, and if not, how many years have passed since publication. These five questions can be answered from an abstract in a few minutes and settle most citations.

The takeaway

Small studies start conversations rather than settling them. Ask how many people, for how long, and against what comparison.

The body already has organs for this, and none of them are sold in a box.