How to Read a Study Someone Has Waved at You

What follows is an argument about checking a cited study yourself, and about where the received version of it stops being true.

Find the actual paper first

Search the title or the authors rather than reading the article about the study, since coverage adds interpretation at every stage. Abstracts are freely available through public databases for the overwhelming majority of published biomedical research. The abstract alone contains the design, the number of participants, the duration and the primary result in most cases.

If a claim cites no traceable paper at all, that is a complete answer and the exercise is finished. Product pages citing research that cannot be located are more common than most readers expect.

Read the methods before the conclusions

Identify whether the subjects were cells, animals or people, since that single fact reorders everything else. Note the number of participants, the duration and what the comparison group received.

Follow the mechanism and check whether assignment was random and whether participants or assessors knew who received what. Establish whether the study measured an outcome people care about or a marker standing in for one. Conclusions written by authors are usually careful, and conclusions written by publicists usually are not.

Check what was actually measured

Look for the primary outcome and compare it with the result being quoted at you. If the quoted result was one of many measured, ask whether it was declared in advance in a registry. Look for the effect size in real units and for the interval around it rather than for a significance verdict.

Check whether the effect is large enough to matter to a person rather than merely detectable by an instrument. A great many citations collapse at this step alone, without any need to assess anything more technical.

Check the context around the paper

Funding and conflict of interest statements are printed in almost every paper and take seconds to read. Note the journal and whether it is indexed in major databases, which filters out the pay-to-publish category.

Look for whether other groups have reproduced the finding, which matters far more than where it appeared. Check the publication date, since a striking result from many years ago with no follow-up is informative.

None of these require expertise, and together they place the paper in its actual context.

Trace the chain to the claim

Compare what the paper concluded with what the press release said and with what the product page claims. Each link in that chain typically strengthens the claim slightly, and the cumulative distance is often enormous.

Watch for a marker becoming a function, a function becoming a benefit, and a benefit becoming a promise. Watch also for a finding in a specific population being generalised to everybody who might buy the product. Documenting that chain once, for any product you use, is a genuinely instructive exercise.

A short list to keep

Who was studied, how many, for how long, and compared against what. What was measured, was it declared in advance, and how large was the effect. Who paid for it, where was it published, and has anyone independent repeated it.

Against the trial data, does the claim being made match what the paper actually reported. For anything involving your own treatment, the last step is always a clinician rather than a conclusion of your own.

The takeaway

Ten minutes with the abstract beats an hour of argument. Read the methods first and compare the claim with what was actually measured.

The body already has organs for this, and none of them are sold in a box.