...
/Enhancing Model Selection with Custom Metrics
Enhancing Model Selection with Custom Metrics
Discover how to implement custom metrics to refine model evaluation and selection.
We'll cover the following...
Metrics customization
Looking at suitably chosen metrics for a problem tremendously increases the ability to develop better models. Although a metric does not directly improve model training, it helps in a better model selection.
Several metrics are available outside TensorFlow, such as in sklearn
. However, they can’t be used directly during model training in TensorFlow. This is because the metrics are computed while processing batches during each training epoch.
Fortunately, TensorFlow provides the ability for this customization. The custom-defined metrics F1Score
and FalsePositiveRate
are provided in the user-defined performancemetrics
library. Learning the programmatic context for the customization is important and, therefore, is elucidated here.
The TensorFlow official guide shows the steps for writing a custom metric. It instructs to create a new subclass inheriting the Metric
class and work on the following definitions for the customization:
__init__()
: All the state variables should be created in this method by calling theself.add_weight()
method, e.g.,self.var = self.add_weight(...)
update_state()
: All updates to the state variables should be done asself.var.assign_add(...)
result()
: The final result from the state variables is computed and returned in this function.
Using these instructions, the FalsePositiveRate()
custom metric is defined in the code below. Note that the FalsePositives
metric is already present in TensorFlow. However, drawing the false positive rate from it during training is not direct, so, the FalsePositiveRate()
is defined.
class FalsePositiveRate(tf.keras.metrics.Metric):def __init__(self, name='false_positive_rate',**kwargs):super(FalsePositiveRate , self).__init__(name=name, **kwargs)self.negatives = self.add_weight(name='negatives',initializer='zeros')self.false_positives = self.add_weight(name='false_negatives',initializer='zeros')def update_state(self,y_true , y_pred , sample_weight=None):''' Arguments:y_true The actual y.Passed by default to Metric classes.y_pred The predicted y. Passedby default to Metric classes.'''# Compute the number of negatives.y_true = tf.cast(y_true , tf.bool)negatives = tf.reduce_sum(tf.cast(tf.equal(y_true , False), self.dtype))self.negatives.assign_add(negatives)# Compute the number of false positives.y_pred = tf.greater_equal(y_pred , 0.5) # Using default threshold of 0.5 to# call a prediction as positive labeled.false_positive_vector =tf.logical_and(tf.equal(y_true , False),tf.equal(y_pred , True))false_positive_vector = tf.cast(false_positive_vector ,self.dtype)if sample_weight is not None:sample_weight = tf.cast(sample_weight ,self.dtype)sample_weight = tf.broadcast_weights(sample_weight , values)values = tf.multiply(false_positive_vector ,sample_weight)false_positives = tf.reduce_sum(false_positive_vector)self.false_positives.assign_add( false_positives)def result(self):return tf.divide(self.false_positives ,self.negatives)
Working of metrics computation and customization
Model training is done iteratively. The iterations happen at multiple levels. The iteration levels for Multi-layer Perceptron (MLP) training are laid out in the illustration below.
The topmost iteration level is epochs. Within an epoch, a model is trained iteratively over randomly selected batches. The batches are the second level of iteration. Multiple samples are present within a batch. The model can be trained by processing one sample at a time. But they’re processed together as a batch for computational efficiency. Therefore, the sample is grayed in the illustration, indicating it’s a logical iteration instead of an actual one.
All the model parameters—weights and biases—are updated during batch processing. Simultaneously, the states of a metric are also updated. The estimated parameters and computed metrics are returned upon processing all the batches in an epoch.
Note: All these operations are enclosed within an epoch, and no values are communicated between two epochs.
The meaning of the metrics state, metrics computation, and the programmatic logic in the code for defining FalsePositiveRate
are enumerated below.
-
The class
FalsePositiveRate()
is inheriting the metric class in TensorFlow. As a result, it automatically has the__init__()
,update_state()
, andresult()
definitions. These definitions will be overwritten during the customization. -
__init__()
is the entry gate to a metric class. It initializes the state variables, which are used for metric computation. -
The false-positive rate is the ratio of false positives over the negatives. Therefore,
false_positives
andnegatives
become the state variables. -
The state variables are prefixed with a
self
variable, which ...