How to implement linear regression from scratch

Linear regression serves as a foundational statistical and machine learning method employed to establish a connection between one or more independent variables (features) and a dependent variable (target). This technique operates under the assumption of a linear relationship, signifying that alterations in the dependent variable exhibit proportionality to changes in the independent variables.

Mathematical representation

For a simple linear regression with one independent variable:

Where,

  • yy is the dependent variable (target).

  • xx is the independent variable (feature).

  • mm is the slope or coefficient.

  • bb is the intercept.

Linear regression parameters

Let’s take a look at the parameters required for the linear regression function.

Parameter

Description

X

Input features

y

Actual output values

weights

Coefficients for each feature in the input data

bias

Intercept term in the linear equation

learning_rate

Step size to adjust weights and bias in each iteration

epochs

Number of iterations the model goes through the dataset

cost

Cost or loss function, measures the difference between predicted and actual values

m

Number of samples in the training dataset

n

Number of features in the input data

Implementation in Python

Let’s break down the steps to implement linear regression from scratch:

Step 1: Initialize parameters

Initialize the weights[object Object] (mm) and bias[object Object] (bb) with random values.

weights = np.random.randn(X.shape[1], 1)
bias = np.random.randn()

Step 2: Make predictions

Use the current weights and bias to make predictions. 

y_pred = np.dot(X, weights) + bias

Step 3: Calculate loss

Calculate the error or any suitable loss function to measure the difference between predictions and actual values.

loss = y_pred - y

Step 4: Compute gradients

Calculate the gradients of the loss with respect to the weights and bias.

dw = 2 / len(X) * np.dot(X.T, loss)
db = 2 / len(X) * np.sum(loss)

Step 5: Update parameters

Update the weights and bias using the gradients and a learning rate.

weights = weights - learning_rate * dw
bias = bias - learning_rate * db

Step 6: Repeat

Repeat steps 2–5 for a predefined number of epochs.

import numpy as np
def linear_regression(X, y, learning_rate=0.01, epochs=1000):
    n_samples, n_features = X.shape
    weights = np.zeros(n_features)
    bias = 0
    for epoch in range(epochs):
        y_pred = np.dot(X, weights) + bias
        loss = y_pred - y
        dw = 2 / len(X) * np.dot(X.T, loss)
        db = 2 / len(X) * np.sum(loss)
        weights = weights - learning_rate * dw
        bias = bias - learning_rate * db
    return weights, bias

Code example

Here’s an example of how to implement linear regression in Python from scratch.

import numpy as np
def linear_regression(X, y, learning_rate=0.01, epochs=1000):
    # Initialize weights and bias
    weights = np.random.randn(X.shape[1], 1)
    bias = np.random.randn()
    # Perform gradient descent
    for epoch in range(epochs):
        # Calculate predictions
        y_pred = np.dot(X, weights) + bias
        # Calculate errors
        loss = y_pred - y
        # Calculate gradients
        dw = 2 / len(X) * np.dot(X.T, loss)
        db = 2 / len(X) * np.sum(loss)
        # Update weights and bias
        weights = weights - learning_rate * dw
        bias = bias - learning_rate * db
    return weights, bias
#X_bias = np.c_[np.ones((X.shape[0], 1)), X]
np.random.seed(42)
X_train = 2 * np.random.rand(100, 1)
y_train = 4 + 3 * X_train + np.random.randn(100, 1)
# Add a column of ones to X_train for the bias term
X_train_bias = np.c_[np.ones((100, 1)), X_train]
# Train the model
trained_weights, trained_bias = linear_regression(X_train_bias, y_train)
# Now, let's perform predictions using the trained model
def predict(X, weights, bias):
    """
    Predict output values using the trained linear regression model.
    Parameters:
    - X: Input features (matrix)
    - weights: Trained model coefficients
    - bias: Trained model intercept
    Returns:
    - y_pred: Predicted output values
    """
    return np.dot(X, weights) + bias
# Generate test data for prediction
X_test = np.array([[1.5], [2.5]])
# Add a column of ones to X_test for the bias term
X_test_bias = np.c_[np.ones((len(X_test), 1)), X_test]
# Perform predictions
predictions = predict(X_test_bias, trained_weights, trained_bias)
# Display the predictions
print("Predictions:", predictions)

Code explanation

  • Line 1: Import the necessary library, numpy.

  • Line 3: Define the linear_regression function with parameters X (input features), y (output), learning_rate, and epochs.

  • Lines 5–6: Initialize weights and bias with random values.

  • Line 9: Perform gradient descent for the specified number of epochs.

  • Lines 11–18: Calculate predictions, errors, and gradients.

  • Lines 21–22: Update weights and bias using the learning rate.

  • Line 27: Set a random seed for reproducibility.

  • Lines 28–29: Generate synthetic training data (X_train, y_train).

  • Line 32: Add a column of ones to X_train for the bias term.

  • Line 35: Train the model using the linear_regression function.

  • Lines 38–50: Define a function (predict) for making predictions using the trained model.

  • Line 53: Generate test data (X_test) for prediction.

  • Line 56: Add a column of ones to X_test for the bias term.

  • Line 59: Perform predictions using the trained model.

  • Line 62: Display the predictions.

Here’s a quiz to test your knowledge.

1

In the given linear regression code, what does the variable dw represent?

A)

Model weights

B)

Learning rate

C)

Gradients of weights

D)

Bias term

Question 1 of 20 attempted

Free Resources

Copyright ©2026 Educative, Inc. All rights reserved