Data Quality and Model Performance Monitoring

The data scientist’s job isn’t over when the trained model is shipped to production. In fact, the production phase of the process is the most important of all. This is when the end users actually use the trained model, so any problem that crops up here has business implications. And this is where observability comes in. Observability is the ability of data scientists to monitor the status of the model in production. It allows the data scientist to:

• Flag issues early before they escalate: Without observability, data scientists only know of a problem in predictions when business stakeholders mention it. A system that monitors the input data and model predictions in production can be set up to alert the development team in a timely manner.

• Troubleshoot production bugs: Logging and monitoring issues in production make troubleshooting easier.

• Investigate and address questions on model predictions from business stakeholders: End users may have questions such as why the model is predicting a particular value. With proper observability, data scientists can investigate these questions and provide some level of explanation to stakeholders.