Dockerfile.tar.gz

Numerical Variables

Default Job

In this course, you will learn how to apply basic and more advanced feature engineering to tabular data with python, during this course,  we will see a range of different techniques and methods to handle many common cases within the data set, so in result, you will create great features so that your machine learning models can predict good results.

Feature Engineering with Python

## Definition
Outliers are data points that are far from other data points. In other words, they are exceptional values in a dataset, an outlier may be due to variability in the measurement, or it may indicate an experimental error.

The latter can seriously affect our predictions and model's performance; therefore, they are sometimes excluded from the data set, but in some other cases, they are given special attention depending on the context. 

Linear models tend to be the algorithms that are most sensitive to outliers.

## Detecting outliers
Visualization is one of the best and easiest ways to infer the overall data and the outliers. Box plot is one of the most preferred visualization tools to detect outliers.

Another way of detecting outliers is to use extreme value analysis with a normal distribution:

- About 99% of the observations of a normally distributed variable lie within the mean ±3 × standard deviations.

- Outliers tend to be outside mean ±3 × standard deviations.

The plot below is a box plot taken from a real dataset called ```total_bill```, and The black dots are considered outliers using the extreme value analysis method.

# Definition
Outliers are data points that are far from other data points. In other words, they are exceptional values in a dataset, an outlier may be due to variability in the measurement, or it may indicate an experimental error.

The latter can seriously affect our predictions and model's performance; therefore, they are sometimes excluded from the data set, but in some other cases, they are given special attention depending on the context. 

Linear models tend to be the algorithms that are most sensitive to outliers.

# Detecting outliers
Visualization is one of the best and easiest ways to infer the overall data and the outliers. Box plot is one of the most preferred visualization tools to detect outliers.

Another way of detecting outliers is to use extreme value analysis with a normal distribution:

- About 99% of the observations of a normally distributed variable lie within the mean ±3 × standard deviations.

- Outliers tend to be outside mean ±3 × standard deviations.

The plot below is a box plot taken from a real dataset called ```total_bill```, and The black dots are considered outliers using the extreme value analysis method.

In this lesson, we will talk about a common problem that we find in datasets called outliers, how to detect them in different ways, and how to handle this kind of data.


Introduction

Variable Types

Common Concerns in Datasets

Handling & Imputing Missing Values

Encoding Categorical Variables

Transforming Variables

Variable Discretization

Handling Outliers

Feature Scaling

Engineering Geospatial Data

Handling Date-Time and Mixed Variables

Resampling Imbalanced Data

Advanced Feature Engineering Techniques

Conclusion

Outliers

Definition