Understand model techniques along with the best practices in applying ML models in production at scale.

Deep Dive into System Design Interview

Machine learning system design is an important component of any machine learning interview. The ability to address problems, identify requirements, and discuss trade-offs can help us stand out among hundreds of other candidates. This module will discuss model techniques, along with best practices in applying scalable machine learning models in production. Once we're done with this module, we’ll be able to apply and leverage knowledge from top researchers at tech companies. We’ll have an up-to-date knowledge of model techniques from hundreds of the latest research and industry papers. We might even be able to impress interviewers once we’ve gone through the lessons of this module.

Machine Learning System Design

# Estimate Delivery Time
## 1. Problem statement
Build a model to estimate the total delivery time given order details (e.g. order size, restaurant location, customer location), market conditions (e.g. weather, events, holidays), and real-time traffic status (e.g. road closures, congestion levels).

> To keep it simple, we do not consider batching (group multiple orders at restaurants) in this exercise. 

<center>

$Delivery Time = Pickup Time + Point\_to\_Point Time + Drop\_off\_Time$

</center>

## 2. Metrics design and requirements

### Metrics
* Offline metrics: Use Root Mean Squared Error (RMSE) to measure the difference between the predicted delivery time and the actual delivery time. The target RMSE should be less than 10-15 minutes to ensure accurate estimates.



   $\sqrt{\sum_{k=1}^n\frac{(predict-y)^2}{n}}$



  where, 

  $n$ is the total number of samples, 
  
  $predict$ is Estimated wait time, 

  $y$ is the actual wait time.


- Online metrics: Conduct A/B testing to monitor #key# RMSE: Root Mean Squared Error #key#, as well as customer engagement (order completion rate) and retention (repeat orders) to ensure the model is providing reliable and useful delivery time estimates.

### Requirements
#### Training
- During training, we need to handle a large amount of data. For this, the training pipeline should have a high throughput. To achieve this purpose, data can be organized in [Parquet files](https://www.educative.io/courses/machine-learning-system-design/training-pipeline)

- The model should undergo retraining every few hours. Delivery operations are under a dynamic environment with a lot of external factors: traffic, weather conditions, etc. So, it is important for the model to learn and adapt to the new environment. For example, on game day, traffic conditions can get worse in certain areas. Without a retraining model, the current model will consistently underestimate delivery time. Schedulers are responsible for retraining models many times throughout the day.

- Balance between overestimation and under-estimation. To help with this, retrain multiple times per day to adapt to market dynamic and traffic conditions.


#### Inference
- For every delivery, the system needs to make real-time estimations as frequently as possible. For simplicity, we can assume we need to make 30 predictions per delivery. 
- Near real-time update, any changes on status need to go through model scoring as fast as possible, i.e., the restaurant starts preparing meals, the driver starts driving to customers. 
- Whenever there are changes in delivery, the model runs a new estimate and sends an update to the customer. 
-  Capture near real-time aggregated statistics, i.e., feature pipeline aggregates data from multiple sources (Kafka, database) to reduce latency.
- Latency from 100ms to 200ms


### Summary

|**Type**|**Desired goals**|
|:-: |:-: |
|Metrics|Optimized for low RMSE. Estimation should be less than 10-15 minutes. If we overestimate, customers are less likely to make orders. Underestimation can cause customers upset.|
|Training|High throughput with the ability to retrain many times per day|
|Inference|Latency from 100ms to 200ms|
















Let’s dive deeper into the problem statement and metrics required for the food delivery system.

Problem Statement and Metrics

Estimate Delivery Time

1. Problem statement