statistics · mlops

These two terms appear in every statistics textbook, every ML course, and almost every data science interview. They are almost universally confused — including by people who have been doing this for years.

The one-sentence version

A confidence interval quantifies uncertainty about a parameter — a population mean, a regression coefficient, a model's expected performance.

A prediction interval quantifies uncertainty about a single future observation — the actual value the next data point will take.

A concrete example

You're predicting wheat yield for a farm in Kansas. You fit a linear regression and get a prediction of 4.2 tonnes/ha.

The confidence interval on that prediction might be [4.0, 4.4] — tight, because you have lots of training data and the regression line is stable.

The prediction interval for the actual next observation might be [3.1, 5.4] — wide, because individual farms vary enormously around the mean even after accounting for their covariates.

If you use the confidence interval to make decisions about a specific farm, you will be systematically wrong. The right interval for individual predictions is always the prediction interval.

Why this matters in production

In production ML systems, you are almost always making individual predictions, not statements about population means. Yet most uncertainty quantification tutorials teach confidence intervals first and many practitioners never fully internalise the distinction.

The prediction interval is always wider. Always. It has to be — it's answering a harder question.

The diagnostic test

If you're unsure which interval you have, ask: as my training set size grows to infinity, does the interval shrink to zero?

Yes → confidence interval. No → prediction interval.

Conformal prediction and quantile regression produce proper prediction intervals. Bootstrap intervals and cross-validation error bars are closer to confidence intervals and will be systematically overconfident when used as prediction intervals.