Rules of Probability

At the simplest level, a model, be it for machine learning or a more classical method such as linear regression, is a mathematical description of how various kinds of data relate to one another.

In the task of modeling, we usually think about separating the variables of our dataset into two broad classes:

1. Independent data: It primarily means inputs to a model are denoted by $X$. These could be categorical features (such as a $0$ or $1$ in six columns indicating which of six schools a student attends), continuous (such as the heights or test scores of the same students), or ordinal (the rank of a student in the class).

2. Dependent data: It refers to the outputs of our models and are denoted by $Y$. As with the independent variables, these can be continuous, categorical, or ordinal, and they can be an individual element or multidimensional matrix (tensor) for each element of the dataset.

So, how can we describe the data in our model using statistics? In other words, how can we quantitatively describe what values we are likely to see, how frequently, and which values are more likely to appear together? One way is by asking the likelihood of observing a particular value in the data or the probability of that value. For example, if we were to ask what the probability is of observing a roll of $4$ on a six-sided die, the answer is that, on average, we would observe a 4 once every six rolls. We write this as follows:

where $P$ denotes the probability of.

What defines the allowed probability values for a particular dataset? If we imagine the set of all possible values of a dataset, such as all values of a die, then a probability maps each value to a number between $0$ and $1$. The minimum is $0$ because we can’t have a negative chance of seeing a result; the most unlikely result is that we would never see a particular value, or 0% probability, such as rolling a 7 on a six-sided die. Similarly, we can’t have greater than 100% probability of observing a result, represented by the value $1$; an outcome with probability $1$ is absolutely certain.

This set of probability values associated with a dataset belongs to discrete classes (such as the faces of a die) or an infinite set of potential values (such as variations in height or weight). In either case, however, these values have to follow certain rules, the probability axiomsUniversity of York. 2019. “University of York.” York.ac.uk. April 26, 2019. https://www.york.ac.uk/.‌:

1. The probability of an observation (a die roll, a particular height, and so on) is a non-negative, finite number between $0$ and $1$.

2. The probability of at least one of the observations in the space of all possible observations occurring is $1$.

3. The joint ...