Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Cost Functions and Gradient Descent

Explore how cost functions such as mean squared error quantify model prediction errors and why squaring residuals is important. Understand how gradient descent iteratively updates model coefficients to minimize errors by adjusting intercept and slope parameters. Learn the impact of learning rates on convergence and how these concepts fit into building and evaluating linear regression models for practical delivery time predictions.

We'll cover the following...

For delivery distances of 1, 2, and 3 kilometers, suppose the actual times are 8, 11, and 16 minutes. One candidate line predicts 7, 10, and 13. Another predicts 9, 12, and 15. Before calculating anything, decide which line seems better. A visual judgment can help, but it becomes unreliable when predictions differ across many rows or when errors cancel each other out.

A cost function turns all model errors into one number that we can compare. For linear regression, mean squared error (MSE) is the average of the squared residuals. Squaring makes every error positive and gives larger misses more weight than smaller misses.

For residuals 2, -1, 3, and -2, the squared residuals are 4, 1, 9, and 4. Their average is 4.5. The negative residual no longer cancels a positive one because both contribute positive squared error. The calculation below makes each step visible.

Python
def main() -> None:
    residuals = [2, -1, 3, -2]
    squared_residuals = [r ** 2 for r in residuals]
    mse = sum(squared_residuals) / len(squared_residuals)
    print(f"Residuals: {residuals}")
    print(f"Squared residuals: {squared_residuals}")
    print(f"Sum of squared residuals: {sum(squared_residuals)}")
    print(f"Mean squared error (MSE): {mse}")
if __name__ == "__main__":
    main()

  • Lines 4–5: The squared residuals [4, 1, 9, 4] are computed first, then MSE is their average: (4 + 1 + 9 + 4) / 4 = 4.5, matching the lesson’s step-by-step calculation.

  • Lines 7–10: Printing the original residuals, squared residuals, their sum (18), and the final MSE (4.5) makes the “squaring prevents cancellation” point visible in the output.

This code shows that MSE summarizes both the direction and size of errors only after squaring removes their signs. A lower MSE means the ...