Drift Detection

Drift detection is the practice of watching for statistical change between the data a model was trained on and the data it now sees in production. Data drift means the input distribution has shifted — new user segments, seasonal behavior, a changed upstream field. Concept drift means the relationship the model learned no longer holds, so yesterday's correct answer is today's mistake. Either way the model silently degrades while its code stays untouched. Because models are frozen but the world keeps moving, drift is the leading cause of quietly rotting predictions. Teams baseline the training distribution and then compare live traffic using measures like population stability index or KL divergence, firing an alert when a feature crosses a threshold. For LLM and RAG products the analog is tracking output quality, shifting user prompts, and embedding drift over time. Practical note: pair automatic drift alarms with a scheduled evaluation run so a spike triggers a real quality check, not just a dashboard blip.