Design Patterns for Resilient Serving - Continued Model Evaluation

This design pattern helps detect and take action when a deployed model is no longer fit-for-purpose.

Reasons for model degradation

1. Concept drift
2. Data drift

Identifying model deterioration

1. Continuous monitoring of the model's predictive performance over time
2. Assess this performance with the same evaluation metrics used during development

Continuous model evaluation provides a framework to evaluate a deployed mode's performance exclusively on new data. This means we can detect a model's staleness as early as possible. This information helps to do either of the following:

1. Retrain the model
2. Replace the existing model with a new version entirely.

This is done by capturing the following:

1. ground truth
2. prediction inputs and outputs for comparing with ground truth values
3. model versions
4. timestamp of prediction requests

Triggers for retraining

Whether to retrain or not based on an evaluation report depends on what amount of deterioration of performance is acceptable in relation to the cost of retraining.

Setting a higher threshold for model performance ensures a higher-quality model in production but will require more frequent retraining jobs which is costly.

Having a lower threshold on the other hand, would mean more cost effective but there is a chance of having a stale model in production.

Scheduled retraining