Introduction to Regression Methods

Learn the basics of linear, regularization, polynomial, and logistic regression.

Regression refers to the task of predicting a continuous class variable, given some input features. We’ll discuss linear regression as the main method frequently used for regression, and some other more advanced techniques in this chapter.

For the sake of consistency, we are going to use the following system of naming:

  • X=(X1,X2,X3,...,Xn)X=(X_1,X_2,X_3,...,X_n) are the input variables.
  • x=(x1,x2,x3,...,xn)x = (x_1, x_2, x_3, . . . , x_n) is an input instance, in which input variable XiX_i takes on value xix_i. To denote different input instances, we use x(i)x^{(i)}, x(j)x^{(j)}.
  • Here, YY is the target/output/class variable and yy is a value YY can take.
  • In case there are many class values to be discussed, we’ll use subscripts to differentiate them (e.g., yjy_j or yky_k). Similar to the input, we use y(i)y^{(i)}, y(j)y^{(j)} to denote corresponding target values of inputs x(i)x^{(i)}, x(j)x^{(j)}.

Linear regression

Linear regression is a regression method that models the relationship between a continuous class/target variable YY, called the dependent variable, and a set of input features X=(X1,X2,...,Xn)X = (X_1, X_2, . . . , X_n), called independent variables. A dependent variable is what we are measuring (or predicting in this case) and an independent variable is what we are controlling or manipulating (or have as an input in this case). Note that we also call the input features, XX, feature vector, since it comprises multiple variables, each corresponding to one dimension in the vector space formed by the features. As such, we are going to use data points and data vectors interchangeably in this chapter.

Back to the method at hand, linear regression assumes a linear relationship between the variables. The dependent variable YY can be expressed as a linear combination of independent variables X1,X2,,XnX_1, X_2,\cdots, X_n, as illustrated in the below figure.

Get hands-on with 1200+ tech skills courses.