Twenty Outcomes and the One That Reached Significance

The options around multiple outcomes and outcome switching are set out side by side below, with the conditions that genuinely favour one over the other.

The arithmetic of looking twice

Conventional statistical thresholds accept a certain chance of a false positive on any single comparison. Running twenty independent comparisons makes it likely that at least one will cross the threshold by chance. Nothing dishonest need occur; the researcher simply reports the result that emerged from the data.

This is why the number of comparisons performed is as relevant as the result of any one of them. A paper reporting one striking finding from a study that measured thirty things has told you less than it appears to.

Subgroups multiply the problem

Splitting participants by sex, age band, baseline severity and other characteristics creates many more comparisons. Subgroups are also smaller, so their results are less stable and more prone to extreme values.

At the dose actually studied, a finding that a treatment worked in older women with high baseline scores is a hypothesis, not a result. Pre-specified subgroups analysed with the appropriate statistical adjustment are legitimate and are usually few in number. Subgroup findings presented as the headline when the overall result was null deserve particular suspicion.

What the primary outcome is for

A well-designed trial names one primary outcome before recruitment, and the study is sized to answer that question. Secondary outcomes are collected and are explicitly exploratory, generating questions for future trials.

This structure exists precisely to prevent the results from choosing themselves after the data arrive. Statistical corrections for multiple comparisons exist and make it harder for any single result to reach significance. When a paper is vague about which outcome was primary, that vagueness is itself informative.

Outcome switching and how to catch it

Trials in most countries are registered publicly before they begin, with their planned outcomes recorded. Comparing the registry entry against the published paper reveals whether the primary outcome changed after the fact. Studies auditing this have found substantial rates of discrepancy across the medical literature, with switches usually favouring positive results.

The registry is public, and checking one takes a few minutes, which makes this the most practical technique in this article.

A registered trial whose published primary outcome matches its registration has passed a test most readers never apply.

Where it shows up in wellness research

Supplement trials frequently measure long panels of blood markers, questionnaires and physical measures. With enough measures, a paper can report an improvement in something regardless of whether the product does anything. Marketing then quotes that one outcome without mentioning the twenty-nine that did not move.

A study described as showing improvements in markers of wellbeing is usually describing this situation. Asking what the primary outcome was, and what happened to it, cuts through most of these claims immediately.

What to do with a single striking result

Treat it as a reason to run a confirmatory study designed around that specific question with a declared primary outcome. Findings that survive that second test are worth acting on and findings that do not are the ordinary fate of exploratory results.

Replication is the mechanism that distinguishes the two, and it is slower and less publishable than discovery. The pattern of an exciting first result followed by smaller or absent confirmations is normal rather than scandalous. Knowing that pattern makes it much easier to hold an early finding lightly without dismissing it.

Side by side

The takeaway

Ask what the primary outcome was and whether it moved. The registry entry is public, and comparing it against the paper takes five minutes.

When the marketing is more precise than the study, believe the study.