Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Time Series Cross-Validation and Forecast Metrics

Explore how to correctly validate time series forecasting models using chronological and rolling-origin cross-validation methods. Learn to apply evaluation metrics such as RMSE and MAPE while preventing data leakage by respecting time order. Understand how these approaches help produce honest, reliable forecast assessments critical for informed decision-making.

We'll cover the following...

A demand forecast can report an RMSE of 2.4 under shuffled cross-validation, then miss future demand by an RMSE of 13.2 when we predict in date order. The smaller number looks like better forecasting, but it is not an honest forecast result. A shuffled split lets a training set contain observations that occur after some validation dates.

Leakage occurs when a model uses information unavailable at the timestamp it is meant to predict. For a forecast of April demand, May demand is future information. Shuffling dates breaks the time arrow because the model learns patterns from later observations before scoring earlier ones. A model cannot do that when making a real replenishment decision.

The following comparison uses a small upward-trending demand series and the same simple forecaster under two splitting rules. Look for the difference between the shuffled and chronological RMSE values.

Python
import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold
def generate_trending_demand(n: int = 50, seed: int = 7) -> np.ndarray:
    rng = np.random.default_rng(seed)
    t = np.arange(n)
    # Upward trend with modest noise; parameters chosen to make leakage obvious.
    demand = 100 + 1.8 * t + rng.normal(0, 2.0, size=n)
    return demand
def nearest_dates_predictor(train_idx: np.ndarray, y_train: np.ndarray, test_idx: np.ndarray) -> np.ndarray:
    # Predict each date as the average of the two training observations closest in time.
    preds = []
    for i in test_idx:
        nearest = np.argsort(np.abs(train_idx - i))[:2]
        preds.append(y_train[nearest].mean())
    return np.array(preds, dtype=float)
def rmse(y_true: np.ndarray, y_pred: np.ndarray) -> float:
    return float(np.sqrt(mean_squared_error(y_true, y_pred)))
def shuffled_kfold_rmse(y: np.ndarray, n_splits: int = 5, seed: int = 42) -> float:
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=seed)
    fold_rmses: list[float] = []
    for train_idx, test_idx in kf.split(y):
        # Leakage: the training set includes dates after some validation dates.
        y_train = y[train_idx]
        y_test = y[test_idx]
        y_pred = nearest_dates_predictor(train_idx, y_train, test_idx)
        fold_rmses.append(rmse(y_test, y_pred))
    return float(np.mean(fold_rmses))
def chronological_holdout_rmse(y: np.ndarray, train_size: float = 0.8) -> float:
    n = len(y)
    split = int(np.floor(train_size * n))
    train_idx = np.arange(split)
    test_idx = np.arange(split, n)
    y_pred = nearest_dates_predictor(train_idx, y[train_idx], test_idx)
    return rmse(y[test_idx], y_pred)
def main() -> None:
    y = generate_trending_demand(n=50, seed=7)
    shuffled_rmse = shuffled_kfold_rmse(y, n_splits=5, seed=42)
    chronological_rmse = chronological_holdout_rmse(y, train_size=0.8)
    print(f"Shuffled 5-fold CV RMSE: {shuffled_rmse:.1f}")
    print(f"Chronological holdout RMSE: {chronological_rmse:.1f}")
if __name__ == "__main__":
    main()

  • Lines 10–12: The synthetic series is constructed with a clear upward trend plus noise so that any evaluation method that accidentally uses “future” values (via shuffling) will look artificially accurate.

  • Lines 15–21: The forecaster predicts each date as the average of the two training observations closest in time, whether those observations come before or after that date.

  • Lines 28–40: ...