Understanding Deep Learning Applications in Rare Event Prediction/

...

/

Applying Convolutional Networks to Multivariate Time Series

Applying Convolutional Networks to Multivariate Time Series

Discover convolutional networks’ application in predicting rare events within multivariate time series data.

We'll cover the following...

The rare event prediction problem explored in this course is a multivariate time series. Let’s proceed with modeling it with convolutional networks.

Convolution on time series

Before modeling, let’s briefly explore the filters and convolution operation in the context of multivariate time series.

A multivariate time series structure is shown in the image below. It shows an illustrative example in which the x-, y-, and z-axis, show the time, the features, and the features’ values, respectivelymultivariate .

Multivariate time series
Multivariate time series

The time series in the illustration has three features with rectangular-, upward pulse-, and downward pulse-like movements. The features are placed along with the depth, making them the channels. A filter for such a time series is shown in the illustration below.

Multivariate time series filter
Multivariate time series filter

The convolution operation between the filter and the time series is shown in the illustration below. As time series has only one spatial axis along time, the convolution sweeps it over time. At each stride, a similarity between the filter and a section of the time series is emitted (not shown in the figure). The convolution variants, such as padding, stride (>1)(>1), dilation, and 1×11 × 1, work similarly along the time axis.

Illustration of a convolution operation on a multivariate time series
Illustration of a convolution operation on a multivariate time series

Imports and data preparation

Like always, the modeling starts with importing the required libraries, including the user-defined ones.

Python 3.8
import pandas as pd 
import numpy as np
import tensorflow as tf
from tensorflow.keras import optimizers
from tensorflow.keras.models import Model
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Conv1D
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPool1D
from tensorflow.keras.layers import AveragePooling1D
from tensorflow.keras.layers import MaxPool2D
from tensorflow.keras.layers import ReLU
from tensorflow.keras.layers import Flatten
from tensorflow.python.keras import backend as K
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from collections import Counter 
import matplotlib.pyplot as plt
import seaborn as sns
# user-defined libraries
import datapreprocessing as dp 
import performancemetrics as pm 
import simpleplots as sp
from numpy.random import seed 
seed (1)
SEED = 123 # used to help randomly select the data points
DATA_SPLIT_PCT = 0.2
from pylab import rcParams 
rcParams['figure.figsize'] = 8, 6 
plt.rcParams.update({'font.size': 22})
print( " Data split percent: ", DATA_SPLIT_PCT )
print( " Random generator seeds: ", SEED )
print( " Size of figures to be plotted later: ", rcParams['figure.figsize'] )

The above code shows data split percent, random generator seed, and size of figures as an output.

The tensor shape of a multivariate time series in a convolutional network is the same as in an LSTMLong Short-Term Memory network. The temporalization procedure discussed in LSTMs is also applied here. The data preprocessing steps for converting categorical features to one-hot encoding and curve-shifting are in the code below.

Python 3.8
df = pd.read_csv("processminer-sheet-break-rare-event-dataset.csv")
df.head(n=5) # visualize the data.
# Hot encoding
hotencoding1 = pd.get_dummies(df['Grade&Bwt'])
hotencoding1 = hotencoding1.add_prefix('grade_') 
hotencoding2 = pd.get_dummies(df['EventPress']) 
hotencoding2 = hotencoding2.add_prefix('eventpress_')
df = df.drop(['Grade&Bwt', 'EventPress'], axis=1)
df = pd.concat([df, hotencoding1 , hotencoding2], axis =1)
# Rename response column name for ease of understanding
df = df.rename(columns={'SheetBreak': 'y'})
# Shift the response column y by 2 rows to do a 4- min ahead prediction.
df = dp.curve_shift(df, shift_by=-2)
# Sort by time and drop the time column.
df['DateTime'] = pd.to_datetime(df.DateTime) 
df = df.sort_values(by='DateTime')
df = df.drop(['DateTime'], axis=1)
# Converts df to numpy array
input_X = df.loc[:, df.columns != 'y'].values 
input_y = df['y'].values
print(df)

The above shows clean data as output, displaying different dataset columns.

Baseline

As usual, modeling starts with constructing a baseline model. The baseline models the temporal dependencies up to a lookback of 20, which is the same as the LSTM models. The temporalization is done with this lookback in the code below. The resultant data in X is a (samples, timesteps, features) array.

lookback = 20
X, y = dp.temporalize(X=input_X ,
                      y=input_y , 
                      lookback=lookback)
                      
# Divide the data into train , valid , and test
X_train , X_test , y_train , y_test = train_test_split(np.array(X), 
                                                       np.array(y),
                                                       test_size=DATA_SPLIT_PCT ,
                                                       random_state=SEED) 
X_train , X_valid , y_train , y_valid = train_test_split(X_train , 
                                                         y_train ,
                                                         test_size=DATA_SPLIT_PCT , 
                                                         random_state=SEED)

# Scaler using the training data.
scaler = StandardScaler().fit(dp.flatten(X_train)) 

X_train_scaled = dp.scale(X_train , scaler)
X_valid_scaled = dp.scale(X_valid , scaler) 
X_test_scaled = dp.scale(X_test , scaler)

# Axes lengths
TIMESTEPS = X_train_scaled.shape[1] 
N_FEATURES = X_train_scaled.shape[2]

print('TIMESTEPS:', TIMESTEPS)
print('\nNumber of features:', N_FEATURES)
Data temporalization and split

The above code shows a number of lookbacks and a number of features as an output.

The baseline is a simple network constructed with the Sequential() framework in the code below.

<center>
    <img src="/score.png" height="240"/>
    <br> </br>
     <img src="/score1.png" height="240"/>
    <br> </br>
     <img src="/score2.png" height="240"/>
</center>
Baseline convolutional neural network

Note: It takes time to complete code execution, but once that’s done, please open the link in the widget to observe the output

The above code shows the training steps of the model and its accuracy and loss at the end.

The network’s components are as follows.

Input layer

The shape of an input sample is defined in the Input() layer. An input sample here is a (timesteps, n_features) tensor due to the temporalization during the data processing.

Conv layer

The network begins with a Conv1D layer. The kernel_size is set to 4 and the number of filters is 16. Therefore, there will be 16 convolutional filters each being a (4, n_features) tensor.

Here n_features=69 and, therefore, each filter will have 4×69 ...