Learn how to approach machine learning system design. We’ll learn how to approach machine learning system design from a top-down view.

Machine Learning with System Design

Learn how to approach machine learning system design. Most major companies, such as Facebook, Google, Amazon, etc., expect their ML developers to have a solid engineering foundation and know about the standard development cycle of ML. This module will help us achieve this goal. We’ll learn how to approach machine learning system design from a top-down view. In addition, we'll learn how to approach different problems, such as how to make a video recommendation algorithm like the one used on Youtube, a feed ranking system like the one used on Linkedin, an ad click prediction system, and a food delivery system.

Machine Learning System Design

# Airbnb rental search ranking
 
## 1. Problem statement
Airbnb users often search for homes in a specific location. The platform must return relevant stays—but more than that, it must return homes users are likely to book. So the goal of the ranking system is simple:
> Rank the homes such that those most likely to be booked appear higher in the search results.

A naive method might use keyword matching or hand-crafted scoring—like sorting based on similarity between the query and listing descriptions. But this fails in practice. Text similarity might show results that “sound good,” but don’t necessarily lead to bookings.

Instead, we want a data-driven approach. If we could estimate the likelihood of a user booking a given listing, we could rank by that likelihood. That brings us to the core idea:

Train a supervised machine learning model that learns from historical user sessions and predicts whether a listing will be booked.
This becomes a binary classification task: booked vs. not booked.
> Why binary classification?
>* Our outcome is binary (booked or not).
>* It allows flexibility in evaluating ranking, user behavior, and optimizing for downstream metrics like revenue.
>* Alternative methods like regression (predicting booking probability directly) could work, but classification gives more control when balancing false positives vs. negatives—important in high-stakes ranking.




## 2. Metrics design and requirements
### Metrics
Designing the right metrics is just as important as choosing the algorithm. The wrong metric can optimize the wrong behavior.

We break down metrics into two buckets: offline metrics (evaluated during training) and online metrics (measured in production).
#### Offline metrics
* **Normalized discounted Cumulative Gain:** nDCG is a standard metric in ranking problems where position matters. It gives higher weight to correct predictions near the top of the list—exactly what we want in search ranking.
> Why nDCG?
> * Users rarely scroll through all results. A relevant result at position 2 is more valuable than at position 10.
> *It accounts for both relevance and position, unlike basic accuracy or AUC.
> *It reflects user satisfaction better than simple classification metrics like precision or recall.

$$DCG_{p} = \sum_{i=1}^p {rel_{i} \over log_{2}(i+1)}$$

$$ nDCG_{p} = {DCG_{p} \over IDCG_{p}}$$

Where:

* relᵢ = relevance score of the result at position i

* IDCG = ideal DCG, if all relevant results were perfectly ordered

This allows us to optimize how well the most relevant (bookable) listings are surfaced at the top.



#### Online metrics
* Conversion rate and revenue lift: This measures the number of bookings per number of search results in a user session.

  $conversion\_rate = \frac{number\_of\_bookings}{ number\_of\_search\_results}$


### Requirements
#### Training on imbalanced data

The dataset is heavily imbalanced: most listings don’t get booked. The model must be robust to this skew.
* Use stratified sampling, class weighting, or focal loss to avoid biasing toward the majority class.
* Consider downsampling non-bookings or using techniques like SMOTE to handle the imbalance.

#### Time-based train/validation split
To mimic production behavior:
* Choose a cutoff date.
* Use sessions before that date for training.
* Use sessions after that date for validation.

### Inference and serving considerations
* **Latency target:** 50–100ms. Search ranking needs to be fast to avoid degrading user experience.

* **Cold-start challenge:** New listings have no data. Use hybrid models or backfill with rule-based heuristics (e.g., prioritize listings with high-quality photos or competitive pricing).



## Summary

|**Type**|**Desired goals**|
|:-: |:-: |
|Metrics| Achieve high normalized discounted Cumulative Gain metric |
|Training|Ability to handle imbalance data|
||Split training data and validation data by time|
|Inference|Latency from 50ms to 100ms|
||Ability to avoid under-predicting for new listings|

Let’s dive into the problem statement and metrics required for the Airbnb rental search ranking application.