P Value, Confidence Interval and Statistical Significance Explained
What a p-value really means, how to read a 95% confidence interval, and why statistical significance is not the same as clinical importance.
"Is it significant?" is the most common question asked about any result. It is also one of the most misunderstood. This article explains p-values and confidence intervals in plain language, so you can report and defend your results correctly.
What a p-value is
A p-value is the probability of getting results at least as extreme as those you observed, assuming the null hypothesis is true (that there is really no difference or no association).
If p = 0.03, data like yours would be fairly unusual if there were truly no difference. That is evidence against the null hypothesis.
What a p-value is not
The American Statistical Association's 2016 statement on p-values made several points worth remembering:
- A p-value does not give the probability that the null hypothesis is true.
- It does not measure the size or importance of an effect.
- Scientific conclusions should not be based only on whether a p-value passes a threshold such as 0.05.
- Proper interpretation needs full reporting and transparency.
Confidence intervals
A 95% confidence interval (CI) gives a range of values for the true effect that are reasonably compatible with your data. Example:
Mean reduction in systolic blood pressure was 6 mmHg greater with drug A than drug B (95% CI 2 to 10 mmHg; p = 0.003).
This tells the reader the size of the difference (6 mmHg), the precision (2 to 10 mmHg) and the evidence against no difference (the interval does not include 0, and p is small).
For ratios such as odds ratios and relative risks, "no effect" is 1, not 0. An odds ratio of 2.1 (95% CI 1.3 to 3.4) does not include 1.
Statistical vs clinical significance
A large study can find a tiny difference with a very small p-value. A blood pressure difference of 1 mmHg might be "statistically significant" yet clinically unimportant. A small study can miss an important difference. Always ask: is the size of the effect large enough to matter to patients?
When the result is "not significant"
A p-value above 0.05 does not prove there is no difference. As Altman and Bland put it, absence of evidence is not evidence of absence. Look at the confidence interval. If it is wide and includes clinically important values, the study may simply have been too small.
Write: "We did not find clear evidence of a difference (mean difference 3 mmHg, 95% CI −1 to 7)" rather than "there was no difference".
How to report your results
| Instead of | Write |
|---|---|
| p < 0.05 | The exact p-value, such as p = 0.03 (or p < 0.001 for very small values) |
| "Significant" alone | The effect size and its 95% CI, then the p-value |
| "No difference" | "No clear evidence of a difference", with the CI |
| p = 0.000 | p < 0.001 |
Common mistakes
- Saying a p-value of 0.04 means "96% probability the drug works".
- Treating p = 0.049 and p = 0.051 as completely different results.
- Reporting only p-values without effect sizes and confidence intervals.
- Running many tests and reporting only the significant ones.
Key takeaways
- A p-value measures how compatible the data are with "no effect"; it is not the probability that the hypothesis is true.
- Confidence intervals show the size and precision of an effect; always report them.
- Statistical significance is not the same as clinical importance.
- A non-significant result is not proof of no effect.
References
- Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–133. doi:10.1080/00031305.2016.1154108
- Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–350. doi:10.1007/s10654-016-0149-3
- Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485.
This article is for education. Always follow the requirements of your university, guide, ethics committee and target journal.