1. what if we just label all the data except the last week as 0, the last week as 1 and build a classifier? In case there is no difference between them ROC AUC will be near 0.5. Then repeat this procedure once per week, until the metric is statistically different from 0.5 -- a signal to retrain the model?
2. In many cases, when applicable, just retrain the model once per week without even checking for covariance shift. Just keep track on CV score sending an alert if the new quality is lower than expected
1. what if we just label all the data except the last week as 0, the last week as 1 and build a classifier? In case there is no difference between them ROC AUC will be near 0.5. Then repeat this procedure once per week, until the metric is statistically different from 0.5 -- a signal to retrain the model?
2. In many cases, when applicable, just retrain the model once per week without even checking for covariance shift. Just keep track on CV score sending an alert if the new quality is lower than expected