Publication Bias and the Trials You Never Get to Read

This is less a set of instructions about unpublished studies and the published record than an argument, and it is worth saying so at the start.

The studies that never appear

A researcher who finds nothing has a manuscript that is harder to place, less likely to be cited and less rewarding to write. Journals have historically preferred novel positive findings, because those attract readers, citations and attention. The result is a drawer full of completed studies that produced null results and were never submitted anywhere.

That is not fraud, it is a set of individually reasonable decisions that produce a collectively distorted record. The distortion is systematic and predictable, which means it can be estimated and, to some extent, corrected for.

What it does to the picture

Anyone reading the published literature sees a sample that has been filtered towards positive results. A compound with several unpublished null studies and one published positive one appears effective to a reader.

Systematic reviews inherit this problem, because they can only include what was published in the first place. This is why a review can reach a confident conclusion that later collapses when unpublished data emerges. The strength of a review depends heavily on whether the underlying literature was complete, not just on its own method.

How researchers detect it

A funnel plot displays each study effect against its precision, and a complete literature produces a roughly symmetrical shape. Missing small negative studies leave a visible gap on one side of the funnel, which is a recognised warning sign.

Statistical tests exist to assess that asymmetry, though they are imperfect and can be affected by other factors. Reviews increasingly report these assessments, and their presence indicates a review team taking the problem seriously. The methods are indirect because the missing studies cannot be examined, only inferred from the shape of what remains.

The structural fix

Trial registration requires researchers to record a study before it begins, including its primary outcome and planned analysis. That creates a public record of studies that exist, so those that never report can at least be identified.

Follow the mechanism and many jurisdictions now also require results to be posted within a set period after a trial finishes. Compliance with those requirements has been variable, and organisations exist that track and publish compliance rates.

Registration also prevents outcome switching, which is the related practice of reporting whichever measurement turned out well.

Where this hits wellness claims hardest

Supplement research is less often registered than pharmaceutical research, so the missing-study problem is harder to assess. A company running several small studies has no obligation in many places to publish the ones that disappointed.

Marketing then cites the surviving study, which looks like evidence and is actually a selected result. Asking whether a trial was registered, and comparing the registry entry with the publication, is a genuinely powerful check. Registries are free, public and searchable, which makes this one of the more accessible things a reader can do.

What it means when reading anything

A single positive study is weaker evidence than it appears, because you cannot see what else was attempted. Replication by independent groups matters more than the impressiveness of any individual result. Effects that shrink as more studies accumulate are the normal pattern rather than a sign of something going wrong.

A claim that has stayed at one striking study for many years has usually not survived quiet attempts to repeat it. Assume the record you can see is the flattering portion of the record that exists.

The takeaway

What you can read is the flattering slice of what was done. Weight replication heavily and treat single striking studies as provisional.

When the marketing is more precise than the study, believe the study.