Understand model techniques along with the best practices in applying ML models in production at scale.

Deep Dive into System Design Interview

Machine learning system design is an important component of any machine learning interview. The ability to address problems, identify requirements, and discuss trade-offs can help us stand out among hundreds of other candidates. This module will discuss model techniques, along with best practices in applying scalable machine learning models in production. Once we're done with this module, we’ll be able to apply and leverage knowledge from top researchers at tech companies. We’ll have an up-to-date knowledge of model techniques from hundreds of the latest research and industry papers. We might even be able to impress interviewers once we’ve gone through the lessons of this module.

Machine Learning System Design

# Metrics evaluation

In practice, it’s common that the model performs well during offline evaluation but does not perform well when in production. Therefore, it is important to measure model performance in both on and offline environments.

## Offline metrics
- During offline training and evaluating, we use metrics like logloss, #key# MAE: Mean Absolute Error #key#, and #key# R2: Coefficient of determination #key# to measure the goodness of fit. Once the model shows improvement, the next step would be to move to the staging/sandbox environment to test for a small percentage of real traffic. 

## Online metrics

- During the staging phase, we measure certain metrics, such as Lift in revenue or click through rate, to evaluate how well the model recommends relevant content to users. Consequently, we evaluate the impact on business metrics. If the observed revenue-related metrics show consistent improvement, then it is safe to gradually expose the model to a larger percentage of real traffic. Finally, when we have enough evidence that new models have improved revenue metrics, we can replace the current production models with new models. For further reading, explore how [Sage Maker enables A/B testing](https://aws.amazon.com/blogs/machine-learning/a-b-testing-ml-models-in-production-using-amazon-sagemaker/)
or [LinkedIn A/B testing](https://engineering.linkedin.com/blog/2015/11/top-five-lessons-from-running-a-b-tests-on-the-world-s-largest-p). 

- This diagram shows one way to allocate traffic to different models in production. In reality, there will be few a dozen models, each getting real traffic to serve online requests. This is one way to verify whether or not a model actually generates lift in the production environment. 



Learn different metrics for model evaluations. 

Metrics Evaluation

Learn different metrics for model evaluations.