Search⌘ K
Home
Courses
Natural Language Processing with Machine Learning

Calculating Loss

Explore how to convert LSTM outputs into logits and apply sparse softmax cross entropy loss in language models. Understand using a padding mask to exclude padded time steps, ensuring accurate loss calculation. This lesson guides you through implementing loss calculation for sequence data in NLP models using TensorFlow.

We'll cover the following...

Chapter Goals:

  • Convert your LSTM model's outputs into logits

  • Use a padding mask to calculate the overall loss

A. Logits & loss

As mentioned in earlier chapters, the task for a language model is no different from regular multiclass classification. Therefore, the loss function will still be the regular softmax cross entropy loss. We use a final fully-connected layer to convert model outputs into logits for each of the possible classes (i.e. vocabulary words).

Python 3.5
import tensorflow as tf
# Output from an LSTM
# Shape: (batch_size, time_steps, cell_size)
lstm_outputs = tf.compat.v1.placeholder(tf.float32, shape=(None, 10, 7))
vocab_size = 100
#print(lstm_outputs)
logits = tf.keras.layers.Dense(units=vocab_size)(lstm_outputs)
# Target tokenized sequences
# Shape: (batch_size, time_steps)
target_sequences = tf.compat.v1.placeholder(tf.int64, shape=(None, 10))
loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
    labels=target_sequences,
    logits=logits)

The function used to calculate the softmax cross entropy loss for feed-forward neural networks is tf.nn.softmax_cross_entropy_with_logits. However, we can only use this function if the ...