Learn to identify and address irregularities in data using SQL. This module covers detecting anomalies, backtesting techniques, and server log analysis to improve your database's efficiency.

Become a Database Professional with SQL

In this module, we’ll focus on detecting anomalies in data using SQL. We’ll explore how to apply mathematical concepts like mean, standard deviation, and z-scores to identify anomalies. Additionally, we’ll delve into backtesting techniques to investigate past data and refine our detection models. We’ll also cover server log analysis to improve and optimize the overall efficiency and scalability of databases.

Anomaly Detection

In the last lesson, we detected a large number of "anomalies" with just one entry. This is very common in errors that do not happen very often. In our case, every once in a while, we get a `400` status code. However, the standard deviation is very low because it does not happen very often so even a single error can be considered way above the acceptable value.

We do not really want to receive an alert in the middle of the night just because of one `400` status code. We cannot have every curious developer fiddling with the dev tools in his browser wake us up in the middle of the night.

To eliminate rows with only a few entries, we set a threshold:

Learn the intuition behind, and the application of, thresholds in finding anomalies.

Detecting Anomalies

Analyzing A Server Log

Backtesting

Improving Accuracy

Conclusion

Adding Thresholds