Understand model techniques along with the best practices in applying ML models in production at scale.

Deep Dive into System Design Interview

Machine learning system design is an important component of any machine learning interview. The ability to address problems, identify requirements, and discuss trade-offs can help us stand out among hundreds of other candidates. This module will discuss model techniques, along with best practices in applying scalable machine learning models in production. Once we're done with this module, we’ll be able to apply and leverage knowledge from top researchers at tech companies. We’ll have an up-to-date knowledge of model techniques from hundreds of the latest research and industry papers. We might even be able to impress interviewers once we’ve gone through the lessons of this module.

Machine Learning System Design

# Training pipeline

- A training pipeline needs to handle a large volume of data with low costs.  One common solution is to store data in a column-oriented format like Parquet or ORC. These data formats enable high throughput for ML and analytics use cases. In other use cases, the #key# tfrecord: TensorFlow format for storing a sequence of binary records #key# data format is widely used in the TensorFlow ecosystem. 

## Data partitioning
- Parquet and ORC files usually get partitioned by time for efficiency as we can avoid scanning through the whole dataset. In this example, we partition data by year then by month. In practice, most common services on AWS, #key# RedShift: Amazon fully managed petabyte-scale data warehouse service in the cloud. #key#, and #key# Athena: Interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. #key# support Parquet and ORC. In comparison to other formats like csv, Parquet can speed up the query times to be 30x faster, save 99% of the cost, and reduce the data that is scanned by 99%.








## Handle imbalance class distribution
In ML use cases like Fraud Detection, Click Prediction, or Spam Detection, it’s common to have imbalance labels. There are few strategies to handle them, i.e, you can use any of these strategies depend on your use case. 

- Use **class weights in loss function**: For example, in a spam detection problem where non-spam data has 95% data compare to other spam data which has only 5%. We want to penalize more in the spam class. In this case, we can modify the entropy loss function using weight. 

  ```
  //w0 is weight for class 0 (non-spam), w1 is weight for class 1 (spam)
  loss_function = -w0 * ylog(p) - w1*(1-y)*log(1-p)
  ```

- Use **naive resampling**: Resample the non-spam class at a certain rate to reduce the imbalance in the training set. It’s important to have validation data and test data intact (no resampling). 

- Use **synthetic resampling**: The Synthetic Minority Oversampling Technique (SMOTE)
 consists of synthesizing elements for the minority class, based on those that already exist. It works by randomly picking a point from the minority class and computing the k-nearest neighbors for that point. The synthetic points are added between the chosen point and its neighbors. For practical reasons, #key# SMOTE: Synthetic Minority Oversampling Technique #key# is not as widely used as other methods. 




## Choose the right loss function
- It depends on the use case when deciding which loss function to use. For binary classification, the most popular is `cross-entropy`. In the Click Through Rate **(CTR)** prediction, Facebook uses Normalized Cross Entropy loss (a.k.a. `logloss`) to make the loss less sensitive to the background conversion rate. 
![](/api/collection/5184083498893312/5582183480688640/page/4695696996302848/image/4781220364812288)

> From an information theory perspective, the cross-entropy formula typically uses logarithm base-2 (reflecting bits of information). However, in most machine learning libraries and implementations such as TensorFlow, PyTorch, and scikit-learn, the natural logarithm (base-e) is used instead when computing cross-entropy loss.

- In a forecast problem, the most common metrics are the Mean Absolute Percentage Error (MAPE) and the Symmetric Absolute Percentage Error (SMAPE). For MAPE, you need to pay attention to whether or not your target value is skew, i.e., either too big or too small. On the other hand, SMAPE is not symmetric, as it treats under-forecast and over-forecast differently. 

  **Mean Absolute Percentage Error**


   $M = \frac{1}{n} \sum_{t=1}^n [\frac{A_{t}-F_{t}}{A_{t}}] \times 100$

  **Symmetric Absolute Percentage Error**

   $SMAPE = \frac{100}{n} \sum_{t=1}^n \frac{[F_{t}-A_{t}]}{([A_{t}] + [F_{t}])/2}$

  $M$ = mean absolute percentage error

  $n$ = number of data

  $A_{t}$ = actual value

  $F_{t}$ = forecast value


- Other companies also use Machine Learning and Deep Learning for forecast problems. For example, Uber uses many different algorithms like Recurrent Neural Networks(RNNs), Gradient Boosting Trees, and Support Vector Regressors for various problems. 

Some of the problems include Marketplace forecasting, Hardware capacity planning, and Marketing. 

- For the regression problem, DoorDash used Quantile Loss to forecast Food Delivery demand.  

- The Quantile Loss is given by:

<center>

   $L(\hat{y}, y) = max(\alpha(\hat{y}-y), (1-\alpha)(y-\hat{y}))$
 
</center>



## Retraining requirements
- Retraining  is a requirement in many tech companies. In practice, the data distribution is a non-stationary process, so the model does not perform well without retraining.
- In AdTech and recommendation/personalization use cases, it’s important to retrain models to capture changes in user’s behavior and trending topics. So, machine learning engineers need to make the training pipeline run fast and scale well with big data. When you design such a system, you need to balance between model complexity and training time. 
- A common design pattern is to use a **scheduler** to retrain models on a regular basis, usually many times per day.


Learn common requirements and patterns in building training pipelines. 

Training Pipeline

Learn common requirements and patterns in building training pipelines.