YOUR STUDY LIBRARY
Topic index ↗
DETAILED NOTE / 16.05

Evidence appraisal

BDS corePublic health & evidence

Translate results into absolute patient benefit, then assess whether the study can support the claim. A large relative effect or a small p-value is not sufficient by itself.

3 min study read·Practise recall
START WITH THE ESSENTIALS

Quick revision summary

Precision and significance

A confidence interval describes uncertainty and precision under the statistical model. For a ratio measure, the null is 1; for a risk difference, it is 0.

Read effect size in the patient’s context

Relative measures describe proportional change; absolute measures depend on baseline risk. If an event falls from 10% to 6%, the absolute reduction is 4 percentage points and the relative reduction is 40%.

Assess bias and applicability before adopting the conclusion

Ask whether the study design can answer the question. Randomisation can reduce confounding, but allocation, missing data, deviations from treatment and selective reporting still matter.

Common mistake

A statistically significant result may be too small, uncertain or inapplicable to justify a clinical change.

DETAILED TEACHING

Treatment effects, confidence intervals and study quality

Translate results into absolute patient benefit, then assess whether the study can support the claim. A large relative effect or a small p-value is not sufficient by itself.

Worked effect measures

MeasureExample: risk falls from 10% to 5%Interpretation
Absolute risk reduction0.10 − 0.05 = 0.05, or 5 percentage pointsFive fewer events per 100 treated over the stated period.
Relative risk0.05 / 0.10 = 0.5Treatment risk is half the comparator risk.
Relative risk reduction1 − 0.5 = 0.5, or 50%Must be interpreted with baseline risk.
NNT1 / 0.05 = 20Treat 20 people to prevent one additional event over that period.
Number needed to harm1 / absolute excess adverse-event riskA separate outcome that belongs in the benefit–harm discussion.
Odds ratioOdds of an event in one group divided by the otherNot identical to relative risk; approximation is better when events are rare.

Precision and significance

A confidence interval describes uncertainty and precision under the statistical model. For a ratio measure, the null is 1; for a risk difference, it is 0. A wide interval can include clinically important benefit, no effect and harm. Compatibility with the null is not proof that the treatments are identical.

A p-value addresses the compatibility of the data with a specified model and null hypothesis, not the probability that the treatment works. Statistical significance does not establish clinical importance, freedom from bias or applicability. NNT depends on outcome, baseline risk and follow-up period; omitting the period makes comparison misleading.

Study design and bias

Randomisation reduces confounding on average; allocation concealment prevents foreknowledge influencing recruitment; blinding limits performance or detection bias where feasible. Attrition, selective reporting, deviations and poorly defined outcomes can still weaken an RCT. Intention-to-treat preserves the assigned groups for the relevant analysis but does not magically repair missing data.

Cohort and case-control designs have important roles, especially for exposures, prognosis or rare outcomes, but association can reflect confounding or selection. A systematic review is only as informative as its included evidence, methods and heterogeneity. Ask whether participants, intervention, comparator and outcomes match the clinical decision you face.

Read effect size in the patient’s context

Relative measures describe proportional change; absolute measures depend on baseline risk. If an event falls from 10% to 6%, the absolute reduction is 4 percentage points and the relative reduction is 40%. The number needed to treat is 1 divided by 0.04, or 25, for the outcome and follow-up represented by those data. These are illustrative calculations, not treatment claims.

The same relative effect can produce a smaller absolute benefit in a lower-risk population. Explain the outcome, comparator, time horizon and uncertainty whenever quoting an NNT. Include adverse effects, treatment burden and patient preferences rather than assuming that a favourable effect estimate settles the decision.

Assess bias and applicability before adopting the conclusion

Ask whether the study design can answer the question. Randomisation can reduce confounding, but allocation, missing data, deviations from treatment and selective reporting still matter. Observational associations may reflect differences between groups rather than a causal treatment effect. Laboratory findings can explain mechanisms without directly predicting clinical survival.

Check whether the participants, intervention, setting and outcomes resemble the clinical decision. A surrogate improvement may not establish a patient-important benefit. Confidence intervals express statistical uncertainty but do not account for every source of bias. A systematic review is only as useful as its methods and the relevant evidence it includes; pooling weak or incompatible studies does not automatically create certainty.

A distinction to remember

A statistically significant result may be too small, uncertain or inapplicable to justify a clinical change.

APPLY IT TO A CASE

What extra information is needed before comparing two published NNT values?

Reveal the reasoning

The outcome, population / baseline risk, intervention and comparator, follow-up period and uncertainty. NNT values without this context may describe very different clinical questions.

Test your recall

Does statistical significance prove clinical importance?

Show answer
MODEL ANSWER

No; the size and relevance of the effect also matter.

More questions on this topic

All Public health & evidence topics

Find your next topic

Search topics, diagrams, schools and revision guides.

Reference image