Understanding Deep Learning Applications in Rare Event Prediction/

...

/

Enhancing Model Selection with Custom Metrics

Enhancing Model Selection with Custom Metrics

Discover how to implement custom metrics to refine model evaluation and selection.

We'll cover the following...

Metrics customization

Looking at suitably chosen metrics for a problem tremendously increases the ability to develop better models. Although a metric does not directly improve model training, it helps in a better model selection. Several metrics are available outside TensorFlow, such as in sklearn. However, they can’t be used directly during model training in TensorFlow. This is because the metrics are computed while processing batches during each training epoch.

Fortunately, TensorFlow provides the ability for this customization. The custom-defined metrics F1Score and FalsePositiveRate are provided in the user-defined performancemetrics library. Learning the programmatic context for the customization is important and, therefore, is elucidated here. The TensorFlow official guide shows the steps for writing a custom metric. It instructs to create a new subclass inheriting the Metric class and work on the following definitions for the customization:

  • __init__(): All the state variables should be created in this method by calling the self.add_weight() method, e.g.,
    self.var = self.add_weight(...)
  • update_state(): All updates to the state variables should be done as
     self.var.assign_add(...)
  • result(): The final result from the state variables is computed and returned in this function.

Using these instructions, the FalsePositiveRate() custom metric is defined in the code below. Note that the FalsePositives metric is already present in TensorFlow. However, drawing the false positive rate from it during training is not direct, so, the FalsePositiveRate() is defined.

Press + to interact
class FalsePositiveRate(tf.keras.metrics.Metric): 
  def __init__(self, name='false_positive_rate',
    **kwargs):
    super(FalsePositiveRate , self). 
    __init__(name=name, **kwargs) 
    self.negatives = self.add_weight(name='negatives', 
        initializer='zeros')
    self.false_positives = self.add_weight( 
      name='false_negatives', 
      initializer='zeros')
  def update_state(self,
    y_true , y_pred , sample_weight=None):
  ''' Arguments:
      y_true The actual y. 
             Passed by default to Metric classes.
      y_pred The predicted y. Passed
             by default to Metric classes.
      '''
    # Compute the number of negatives.
    y_true = tf.cast(y_true , tf.bool)
    negatives = tf.reduce_sum(tf.cast( 
      tf.equal(y_true , False), self.dtype))
    self.negatives.assign_add(negatives)
    
    # Compute the number of false positives.
    y_pred = tf.greater_equal( 
      y_pred , 0.5
    ) # Using default threshold of 0.5 to
      # call a prediction as positive labeled.
    false_positive_vector = 
      tf.logical_and(tf.equal(y_true , False),
    tf.equal(y_pred , True)) 
    false_positive_vector = tf.cast(false_positive_vector , 
       self.dtype)
    if sample_weight is not None: 
      sample_weight = tf.cast(sample_weight ,
        self.dtype)
      sample_weight = tf.broadcast_weights(
        sample_weight , values) 
      values = tf.multiply(false_positive_vector , 
        sample_weight)
    false_positives = tf.reduce_sum(false_positive_vector)
    self.false_positives.assign_add( false_positives)
  def result(self):
      return tf.divide(self.false_positives , 
        self.negatives)

Working of metrics computation and customization

Model training is done iteratively. The iterations happen at multiple levels. The iteration levels for Multi-layer Perceptron (MLP) training are laid out in the illustration below.

Press + to interact
Levels of iterations in MLP training
Levels of iterations in MLP training

The topmost iteration level is epochs. Within an epoch, a model is trained iteratively over randomly selected batches. The batches are the second level of iteration. Multiple samples are present within a batch. The model can be trained by processing one sample at a time. But they’re processed together as a batch for computational efficiency. Therefore, the sample is grayed in the illustration, indicating it’s a logical iteration instead of an actual one.

All the model parameters—weights and biases—are updated during batch processing. Simultaneously, the states of a metric are also updated. The estimated parameters and computed metrics are returned upon processing all the batches in an epoch.

Note: All these operations are enclosed within an epoch, and no values are communicated between two epochs.

The meaning of the metrics state, metrics computation, and the programmatic logic in the code for defining FalsePositiveRate are enumerated below.

  • The class FalsePositiveRate() is inheriting the metric class in TensorFlow. As a result, it automatically has the __init__(), update_state(), and result() definitions. These definitions will be overwritten during the customization.

  • __init__() is the entry gate to a metric class. It initializes the state variables, which are used for metric computation.

  • The false-positive rate is the ratio of false positives over the negatives. Therefore, false_positives and negatives become the state variables.

  • The state variables are prefixed with a self variable, which ...