Building a self-healing ML pipeline: from drift detection to automated retraining
Most ML projects die the same quiet death. You train a model, it performs well, you deploy it, and then you move on. Six months later someone notices the predictions are off.
The problem: models go stale
When I ran the drift detector on the Grapes model in Cropcast, here's what came back:
Data drift — detected: True | drifted cols: 22/23 | share: 95.7% Model drift — ref MAE: 0.036 | cur MAE: 0.515 | degradation: 1334.2%
22 out of 23 features had shifted distribution. The model's error had degraded by 1334%. The solution isn't to retrain once and hope. It's to build a system that catches drift and heals itself.
The four stages
1. Drift detection — KS test and PSI to detect distribution shifts in incoming data.
2. Retraining — XGBoost with Optuna hyperparameter tuning and walk-forward cross-validation.
3. Evaluation — compare new model MAE against production. New model must beat it by 2% to promote.
4. Promotion — MLflow Model Registry manages staging → production → archived lifecycle.
Orchestration
The full pipeline runs every Monday at 6am UTC via GitHub Actions. Tests run first — if they fail, retraining never starts. If retraining fails, deployment never starts. Production only ever sees models that passed every gate.