How to build machine learning regression models with Python

How to evaluate regression models#

Building a regression model is only half of the machine learning process. The other half is determining whether the model actually makes accurate predictions on data it has never seen before.

This is where evaluation metrics become important. Different regression metrics measure prediction quality in different ways. Some focus on average error, others penalize large mistakes, and some evaluate how much of the underlying pattern in the data the model captures. Understanding these metrics helps you choose better models and avoid misleading conclusions.

Regression metrics at a glance#

Mean Absolute Error (MAE)#

Mean Absolute Error measures the average size of prediction errors without considering whether the model overpredicted or underpredicted.

The formula is:

[
MAE = \frac{\sum |y_{true} - y_{pred}|}{n}
]

Suppose a model predicts fish weights:

Absolute errors:

• 10g

• 10g

• 10g

MAE:

[
\frac{10 + 10 + 10}{3} = 10
]

The model is off by an average of 10 grams.

Why practitioners like MAE:

• Easy to understand

• Uses the same units as the target variable

• Less sensitive to extreme outliers

Mean Squared Error (MSE)#

Mean Squared Error squares each prediction error before averaging.

[
MSE = \frac{\sum (y_{true} - y_{pred})^2}{n}
]

Using the same errors:

• 10² = 100

• 10² = 100

• 10² = 100

MSE:

[
\frac{100 + 100 + 100}{3} = 100
]

The key difference is that large mistakes are penalized much more heavily.

For example:

• Error of 2 → 4

• Error of 10 → 100

This makes MSE useful when large prediction errors are especially costly, such as:

• Medical predictions

• Financial forecasting

• Resource planning systems

Root Mean Squared Error (RMSE)#

RMSE is simply the square root of MSE.

[
RMSE = \sqrt{MSE}
]

If MSE = 100:

[
RMSE = 10
]

RMSE combines two useful properties:

• Penalizes large errors like MSE

• Returns values in the same units as the target variable

For this reason, RMSE is one of the most commonly reported regression metrics in industry and research.

Many machine learning practitioners prefer RMSE because it is easier to interpret than MSE while still highlighting significant prediction mistakes.

R² Score#

R² Score (Coefficient of Determination) measures how much of the variation in the target variable is explained by the model.

Unlike MAE, MSE, and RMSE, R² does not measure error directly.

Instead, it answers the question:

How well does the model explain the data?

Typical interpretations:

For example:

• R² = 0.90 means the model explains most of the observed variation.

• R² = 0.50 suggests useful predictive power but significant unexplained variance remains.

• R² = 0.10 indicates the model may be missing important features or relationships.

Comparing two regression models#

Imagine two models predicting fish weight.

At first glance, Model B appears better because it has a lower MAE.

However:

• Model A has lower MSE

• Model A has lower RMSE

• Model A has slightly higher R²

This suggests that Model B performs well on average but occasionally makes very large mistakes. Model A is more consistent.

This example illustrates why relying on a single metric can be misleading.

Which metric should you use?#

Use the following decision guide:

In practice:

• MAE is often a good starting point.

• RMSE is widely used in production systems.

• R² provides useful context about overall model quality.

• MSE is valuable when large mistakes are particularly undesirable.

Common evaluation mistakes#

Evaluating only on training data#

A model may perform extremely well on training data but fail on unseen examples.

Always evaluate using a separate test set or cross-validation.

Comparing models using only one metric#

A model with the best MAE may not have the best RMSE or R².

Review multiple metrics before choosing a model.

Ignoring outliers#

A few extreme values can significantly affect MSE and RMSE.

Investigate unusual observations rather than blindly trusting metric values.

Misinterpreting R²#

A high R² does not guarantee a useful model.

A model can have a high R² while still producing prediction errors that are too large for a real-world application.

Calculating regression metrics with scikit-learn#

Sample output:

Final takeaway#

No single metric tells the whole story. MAE provides an intuitive measure of average error, MSE and RMSE highlight the impact of large mistakes, and R² shows how much of the underlying variation your model explains.

Strong regression model evaluation typically combines MAE, RMSE, and R² to build a complete picture of prediction performance. Looking at multiple metrics helps you make better decisions, compare models more effectively, and avoid being misled by any single measurement.

Deploying a regression model with FastAPI#

Training a regression model is only one part of the machine learning workflow. In real-world applications, a trained model often needs to serve predictions to websites, dashboards, mobile apps, or other services.

This is where model deployment comes in. Instead of manually running predictions inside a Jupyter notebook, you can expose your model through an API. Applications send input data to the API, and the API returns predictions in real time. FastAPI has become a popular choice for this task because it is lightweight, fast, and works seamlessly with Python machine learning libraries.

Basic deployment workflow#

A typical machine learning deployment pipeline looks like this:

Once a model has been trained and evaluated, it can be saved to disk and reused without retraining every time the application starts.

Saving a trained regression model#

After training a scikit-learn model, you can save it using joblib.

This creates a file named fish_weight_model.pkl that contains the trained model. Later, your API can load this file and use it to generate predictions.

Creating a FastAPI prediction endpoint#

The following example demonstrates a simple FastAPI service that loads the trained model, accepts fish measurements as input, and returns a predicted fish weight.

This example demonstrates several useful FastAPI features:

• Pydantic models validate incoming data automatically.

• Type hints improve readability and reduce errors.

• JSON responses make integration with other applications straightforward.

• Automatic API documentation is generated without additional work.

Example API request#

A client can send a POST request containing fish measurements:

Example API response#

The API returns the model's prediction:

The exact value will depend on the model and training data used.

Why FastAPI is useful for model serving#

FastAPI has become one of the most popular frameworks for deploying machine learning models because it offers a strong balance between simplicity and performance.

Some key advantages include:

• Lightweight and easy to learn

• Excellent performance for Python applications

• Automatic OpenAPI and Swagger documentation

• Native support for type hints and validation

• Seamless integration with scikit-learn, pandas, NumPy, and other ML libraries

• Easy deployment to cloud platforms and containerized environments

For many machine learning projects, FastAPI provides everything needed to move from experimentation to production.

Deployment best practices#

Before deploying a model, consider a few important practices:

• Validate all incoming input data.

• Keep feature ordering identical to the training process.

• Save preprocessing logic alongside the model.

• Monitor prediction quality after deployment.

• Retrain models when underlying data changes.

• Avoid exposing experimental or untested models directly to end users.

These practices help ensure that predictions remain reliable and consistent over time.

Keep preprocessing and prediction together#

For production systems, it is often best to save a complete scikit-learn pipeline instead of saving only the model.

For example:

Using a pipeline ensures that the same preprocessing steps applied during training are also applied during prediction, reducing the risk of inconsistent results.

Final takeaway#

A regression model becomes significantly more useful when it can serve predictions outside a notebook. FastAPI provides a simple and practical way to turn a trained scikit-learn model into a web API that other applications can consume.

As you continue learning machine learning, understanding how to deploy models is just as important as learning how to train them. Building APIs around your models helps bridge the gap between experimentation and real-world software systems.

Regression datasets to practice your machine learning skills#

Machine learning skills improve through experimentation. Once you understand the basics of regression, the best next step is to apply the same concepts across different datasets and compare how your models behave.

Each dataset teaches something slightly different. Some are clean and beginner-friendly, while others introduce categorical variables, missing values, nonlinear relationships, feature engineering, and overfitting risks. Practicing across multiple datasets helps you build stronger regression intuition.

Regression dataset comparison#

Beginner-friendly datasets#

California Housing#

The California Housing dataset predicts median house value using features such as income, location, population, and household information. It is available directly through scikit-learn, which makes it easy to load and use.

This dataset is beginner-friendly because it is already structured and works well for multiple linear regression. It is a good next step after simple one-feature regression because you can practice using several input variables at once.

Good techniques to practice:

• Multiple linear regression

• Train-test splitting

• MAE, MSE, RMSE, and R² evaluation

• Feature scaling

Medical Insurance Costs#

The Medical Insurance Cost dataset predicts insurance charges based on features such as age, BMI, smoking status, region, and number of children.

This dataset is useful because it introduces categorical variables. You will need to encode features like sex, smoker, and region before training a model.

Good techniques to practice:

• Multiple linear regression

• One-hot encoding

• Feature importance

• Comparing numeric and categorical predictors

Fish Market Dataset#

The Fish Market dataset predicts fish weight using measurements such as length, height, and width.

This is a great beginner dataset because the relationship between physical measurements and weight is intuitive. It also works well for comparing simple linear regression, multiple linear regression, and polynomial regression.

Good techniques to practice:

• Simple linear regression

• Feature selection

• Polynomial regression

• Visualizing predictions

Intermediate regression projects#

Bike Sharing Demand#

Bike Sharing Demand predicts the number of bike rentals based on weather, season, time, and calendar-related variables.

This dataset introduces real-world complexity because demand is affected by time, temperature, humidity, holidays, and user behavior.

You can practice:

• Time-based feature engineering

• Handling seasonality

• Comparing linear and tree-based models

• Evaluating prediction errors across different conditions

Student Performance Dataset#

The Student Performance dataset predicts student outcomes based on academic, demographic, and lifestyle-related features.

This dataset is useful for practicing categorical encoding and careful interpretation. Some features may appear correlated with performance, but correlation does not always imply causation.

You can practice:

• Categorical encoding

• Correlation analysis

• Feature selection

• Ethical interpretation of model results

Automobile Price Prediction#

The Automobile Price dataset predicts car prices using features such as engine size, horsepower, fuel type, body style, and brand.

This dataset is valuable because it often includes missing values and mixed data types. It teaches you that preprocessing is often just as important as model selection.

You can practice:

• Missing value handling

• Encoding categorical variables

• Feature selection

• Model comparison

Advanced regression projects#

Ames Housing Prices#

Ames Housing is a richer and more realistic housing dataset than many beginner examples. It includes many numeric and categorical features related to property size, location, condition, year built, and sale details.

This dataset is excellent for practicing end-to-end regression workflows.

You can practice:

• Advanced feature engineering

• Cross-validation

• Handling many categorical features

• Comparing linear models with tree-based models

• Preventing overfitting

Energy Efficiency Dataset#

The Energy Efficiency dataset predicts heating and cooling loads based on building characteristics.

This dataset is useful because relationships between features and targets may be nonlinear. It is a strong fit for polynomial regression and model comparison.

You can practice:

• Polynomial regression

• Nonlinear feature relationships

• Multi-output regression concepts

• Model evaluation across different target variables

Other multi-feature datasets#

Once you are comfortable with beginner and intermediate datasets, try regression problems with many features, missing data, or domain-specific variables.

These datasets help you practice the full machine learning workflow: cleaning data, selecting features, training models, evaluating results, and explaining trade-offs.

What to practice with each dataset#

Suggested learning progression#

Step 1: Fish Market Dataset#

Start here if you want to reinforce the basics. Use one feature first, then add more features and compare performance.

Step 2: California Housing#

Move to California Housing when you are ready for multiple linear regression. This dataset helps you practice working with several numerical features.

Step 3: Medical Insurance Costs#

Use this dataset to learn categorical encoding and feature interpretation. It is especially useful for understanding how one feature, such as smoking status, can strongly affect predictions.

Step 4: Bike Sharing Demand#

This dataset introduces time-based patterns and more realistic feature engineering. It helps you think beyond simple columns and rows.

Step 5: Ames Housing#

Use Ames Housing when you are ready for a larger, more realistic regression challenge. This dataset is ideal for cross-validation, feature engineering, and model comparison.

Common regression mistakes when practicing#

Using all features without analysis#

More features do not always mean a better model. Some features may be noisy, redundant, or irrelevant.

Ignoring train-test separation#

Always split your data before evaluating your model. Testing on the same data used for training gives overly optimistic results.

Overfitting polynomial models#

Polynomial regression can fit curves well, but higher-degree models can memorize training data instead of learning useful patterns.

Misinterpreting correlation#

A strong correlation does not prove that one feature causes the target variable to change. Be careful when explaining model results.

Using MAE alone for evaluation#

MAE is useful, but it does not tell the full story. Compare multiple metrics such as MAE, RMSE, and R² to better understand model performance.

Final recommendations#

If you are new to regression, do not jump directly into the largest dataset. Start with small, intuitive datasets where you can clearly understand the relationship between features and the target variable.

A strong practice path looks like this:

The fastest way to become comfortable with regression is to solve similar prediction problems across multiple datasets. Each new dataset introduces different challenges and helps you build stronger machine learning intuition over time.