Anomaly Detection with H2O

What is anomaly detection?

Anomaly detection, also known as outlier detection, is a technique used in data analysis and machine learning to identify data points or instances that deviate significantly from the expected behavior of the majority of the data. These anomalous data points are referred to as anomalies or outliers.

The goal of anomaly detection is to distinguish between normal and abnormal patterns in the data. Anomaly detection is crucial for detecting rare events or abnormal behaviors that may have important implications for decision-making, risk assessment, and quality control. It’s commonly used in domains such as finance, cybersecurity, manufacturing, and healthcare to identify unusual or suspicious events or patterns that may indicate errors, fraud, faults, or abnormal behavior.

Anomaly detection approaches

Anomaly detection is a versatile technique that can be employed using different approaches—supervised, semi-supervised, and unsupervised.

• In the supervised approach, we have labeled data that indicates whether each observation is anomalous or genuine. During training, the model uses this labeled data to learn the patterns of anomalies and genuine observations. However, obtaining labels for all observations might be challenging or impractical in many real-world scenarios.

• The semi-supervised approach takes a more practical stance. Here, we only have labels for the genuine, nonanomalous observations and no information about the anomalous ones. During training, the model learns from the genuine data to identify normal patterns. Subsequently, during prediction, it assesses the similarity of new observations to the training data and how well they fit the established model.

• On the other hand, the unsupervised approach works with an unlabeled dataset that includes both genuine and anomalous observations. The model then learns to identify patterns that are different from the majority of the data, effectively detecting anomalies without the need for explicit labels.

Each approach has its merits and limitations, and the choice of the method depends on the availability of labeled data and the nature of the anomaly detection task.

Key principles of the isolation forest algorithm

Isolation forest shares principles with random forest, leveraging the robust foundation of decision trees. The core principle behind isolation forest revolves around the ease of segregating anomalies from the rest of the dataset compared to regular data points. Its functioning can be summarized in the following broad steps:

• Forest of decision trees: Isolation forest employs a multitude of decision trees to break down the data. This is a pivotal step for effective anomaly detection.

• Efficient anomaly separation: The algorithm capitalizes on the fact that anomalies are more effortlessly isolated from the dataset compared to typical data points.

• Random partitioning: The data gets partitioned through a collection of decision trees within the isolation forest. These partitions are made randomly.

• Anomaly identification: The number of random splits necessary to isolate a record determines its anomaly status. A lower number of splits indicates an anomaly.

• Collective anomaly indication: When a group of random trees yields shorter path lengths for specific samples, those samples are highly likely to be anomalies.