A Meta-Analysis Is Only as Good as What Went Into It

The points below about systematic reviews and pooled analyses are ordered by how much difference they make, not by how often they get repeated.

What the method is meant to do

A systematic review defines a question, searches for all studies addressing it by stated rules, and assesses each one for quality. A meta-analysis then combines their results statistically, weighting larger and more precise studies more heavily. The gain is precision, since combining participants narrows the range of plausible values for an effect.

It also reduces the temptation to cherry-pick, because the search and inclusion criteria are declared in advance. Done properly it is the most useful single document in evidence-based medicine, which is exactly why the label is worth checking.

The input problem

Statistical pooling does not correct bias in the original studies, it averages it and reports it with tighter confidence. Twelve small unblinded industry trials pooled together produce a confident estimate of a biased result. Good reviews assess risk of bias in each included study and report how the conclusion changes when weak studies are excluded.

A review that lists included studies without assessing their quality has skipped the step that mattered most. The phrase supported by a meta-analysis therefore means very little until you know what was pooled.

Comparing things that are not alike

Heterogeneity describes how much the included studies disagree with each other beyond what chance would explain. High heterogeneity means the studies are answering different questions, using different doses, populations, durations or outcome measures. Averaging across such studies produces a number that describes none of them, which is the apples-and-oranges objection stated formally.

Against the trial data, reviews report heterogeneity statistics, and a high value alongside a confident conclusion is a contradiction worth noticing. This is particularly acute in supplement and nutrition research, where formulations and doses vary enormously between trials.

What is missing from the pool

A review can only include studies that exist publicly, so unpublished negative trials distort the result before anyone starts. Reviews test for this using funnel plots and related methods, which look for the asymmetry that missing small negative studies produce. Restricting the search to one language or one database excludes work systematically rather than randomly.

Against the trial data, including the same participants twice, through duplicate publication of one dataset, inflates the apparent evidence. Good reviews describe their search in enough detail that someone else could repeat it, and that description is the honest part.

Reviews that are not systematic

A narrative review selects studies at the author discretion and can support almost any conclusion by choosing what to discuss. These are often useful as introductions and they should not be cited as though they weighed the evidence.

Titles frequently blur the distinction, and the methods section is where the difference becomes visible. Registering a review protocol in advance is now standard practice for the better ones and is easy to check. Reviews from organisations with a declared methodology and independence from product manufacturers are worth more than reviews without either.

Reading one in five minutes

Read how many studies were included and how many participants in total, since a review of four tiny trials is still four tiny trials. Read the risk of bias summary, which is usually a table or a coloured chart near the results. Read the heterogeneity figure and the authors own comment on whether pooling was appropriate.

Against the trial data, read the funding statement and the conflicts of interest, for the review and where possible for the included trials. Read the conclusion last, because reading it first makes everything above look like support for it.

Everything above, in order of what to do first

1. What the method is meant to do. A systematic review defines a question, searches for all studies addressing it by stated rules, and assesses each one for quality.
2. The input problem. Statistical pooling does not correct bias in the original studies, it averages it and reports it with tighter confidence.
3. Comparing things that are not alike. Heterogeneity describes how much the included studies disagree with each other beyond what chance would explain.
4. What is missing from the pool. A review can only include studies that exist publicly, so unpublished negative trials distort the result before anyone starts.
5. Reviews that are not systematic. A narrative review selects studies at the author discretion and can support almost any conclusion by choosing what to discuss.
6. Reading one in five minutes. Read how many studies were included and how many participants in total, since a review of four tiny trials is still four tiny trials.

The takeaway

Pooling makes the answer more precise, not more correct. Read what went in before believing what came out.

The body already has organs for this, and none of them are sold in a box.