Prediction intervals are more useful than point estimates — here's how to add them to XGBoost
Your XGBoost model predicts house prices. It says £342,000. That number is almost certainly wrong — not because the model is bad, but because a single number implies a precision that no model actually has.
Three approaches
XGBoost doesn't output uncertainty natively. Here are three ways to add it.
1. Quantile regression
Train two separate models — one predicting the 5th percentile, one predicting the 95th. The interval between them is your 90% prediction interval. Use the objective reg:quantileerror with quantile_alpha set to your desired quantile.
2. Conformal prediction
Hold out a calibration set. Measure the absolute residuals on it. The (1-alpha) quantile of those residuals is your margin. Add and subtract it from every new prediction. Coverage is guaranteed by construction.
3. Bootstrapping
Train 100 models on random subsamples of the training data. Use the spread of their predictions as your uncertainty estimate. Computationally expensive and tends to underestimate uncertainty, but useful for understanding model variance.
Which to use
For most production use cases, conformal prediction is the right default. The coverage guarantee is mathematically provable and the implementation is simple — you only need to hold out a calibration set.